National Academies Press: OpenBook

Evaluation of PEPFAR (2013)

Chapter: 11 PEPFAR's Knowledge Management

« Previous: 11 PEPFAR's Knowledge Management-Main Messages
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

11

PEPFAR’s Knowledge Management

INTRODUCTION

Knowledge is generated across all levels of PEPFAR including implementing partners, partner country mission teams, PEPFAR implementing agencies, and Office of the U.S. Global AIDS Coordinator (OGAC) headquarters. This knowledge, if appropriately synthesized, transferred, disseminated, shared, and used, has the potential not only to contribute to program improvement and the sustainability of PEPFAR’s efforts, but also to help the global community in its response to the HIV/AIDS epidemic. As the largest donor currently addressing the global HIV/AIDS epidemic, PEPFAR has both the ability and the responsibility to play a significant leadership role in this realm (IHME, 2011).

Knowledge management has been defined by Swan and colleagues as “any process or practice of creating, acquiring, capturing, sharing and using knowledge wherever it resides, to enhance learning and performance in organizations” (Swan et al., 1999, p. 669). Knowledge management is a strategy used by many organizations to harness and respond to both existing and created knowledge and has been adopted by organizations such as the World Bank and the World Health Organization (WHO) (Loermans, 2002; WHO, 2005; World Bank, 2003).

An organization that is skilled in knowledge management is able to efficiently and effectively manage knowledge that has been created (Loermans, 2002). As discussed in this chapter, the types of knowledge PEPFAR has created and utilized include developing a system for collecting extensive

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

program monitoring data, supporting epidemiologic and surveillance activities in partner countries, strengthening partner country health information systems, implementing various program evaluation approaches, supporting research, and the creation of both tacit and experiential knowledge as a result of program implementation.

Evaluating PEPFAR’s knowledge management was not an explicit part of the committee’s congressional mandate, but because availability and access to information was key to every aspect of this evaluation, the committee felt strongly that to help guide PEPFAR’s future efforts, examining and making recommendations regarding PEPFAR’s knowledge management approach was critical. The committee determined that PEPFAR has made strong efforts in generating knowledge, often at a level not seen in other development programs. Yet, as reflected in prior chapters of this report, there are key areas where the information needed to assess efforts and guide future activities is unavailable or insufficient. Significant gaps remain in PEPFAR’s knowledge management approach, especially in the realms of knowledge creation, dissemination, and utilization, and to date, OGAC has not articulated a clear and comprehensive strategy for managing knowledge to optimize PEPFAR’s performance and effectiveness.

This chapter shifts its focus away from assessing and addressing the limitations in the available information that affected the committee’s ability to respond to the specific charge mandated by Congress; these were discussed in Chapter 2. Rather, the aim of this chapter is to offer an assessment to guide PEPFAR to more strategically and efficiently meet its information needs going forward. This chapter will review and assess PEPFAR’s current approach to knowledge management, culminating with recommendations from the committee for future directions to address current gaps and to strengthen PEPFAR’s ability to generate, share, and utilize knowledge more effectively.

Strategic Information

PEPFAR articulated a goal of having evidence-based programs from the outset (OGAC, 2004). To meet this goal, the OGAC Office of Strategic Information (SI), which is responsible for using SI to guide and coordinate PEPFAR performance planning and reporting, was established (GAO, 2011a). The first Five-Year Strategy defined strategic information as “the systematic collection, analysis, and dissemination of information about reaching the Emergency Plan’s objectives, as well as the related programmatic activities funded to reach these goals” (OGAC, 2004, p. 73). Strategic information was used as an organizing concept because ‘WHO was just starting to use the term strategic information, and that resonated with us—the use of information for program improvement and operations—so, we

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

decided to use that name. Gathering of information—it had to be strategic and it had to be used(NCV-3).1,2

At the peak of funding around 2007–2008, the OGAC SI office had an annual budget of around $33 million for centrally funded SI activities, but over time funding has been reduced to less than $10 million annually (NCV-2-USG). At the partner country level, from fiscal year (FY) 2006 to FY 2011, approximately 4 to 5 percent of total funding for PEPFAR partner country activities was budgeted for SI activities, excluding staff salaries (see Figure 11-1) (OGAC, 2005a, 2006d, 2007g, 2008d, 2010d, 2011i,j). These activities have included monitoring and reporting partner results, as well as surveillance, surveys, and efforts to strengthen partner country health information systems (OGAC, 2008b, 2009d, 2010c). Reflecting an increased focus on country ownership, FY 2012 Country Operational Plan (COP) guidance advised mission teams that activities planned under the SI budget code should aim “to build individual, institutional, and organizational capacity in country” for strategic information activities (OGAC, 2011h, p. 68).

PROGRAM TARGETS AND PRIORITIES

Setting Program Targets

Setting priorities and targets is one important aspect of planning and managing programs. Subsequently monitoring and assessing progress and performance in meeting these targets is critical for program management. When PEPFAR was authorized in 2003,3 it was established with an emphasis on accountability by setting specific performance targets and with a recognition of the necessity of monitoring and evaluation to assess the performance of PEPFAR-supported programs. The initial 5-year goals for

__________________

1 Single quotations denote an interviewee’s perspective with wording extracted from transcribed notes written during the interview. Double quotations denote an exact quote from an interviewee either confirmed by listening to the audio-recording of the interview or extracted from a full transcript of the audio-recording.

2 Country Visit Exit Synthesis Key: Country # + ES

Country Visit Interview Citation Key: Country # + Interview # + Organization Type

Non-Country Visit Interview Citation Key: “NCV” + Interview # + Organization Type

Organization Types: United States: USG = U.S. Government; USNGO = U.S. Nongovernmental Organization; USPS = U.S. Private Sector; USACA = U.S. Academia; Partner Country: PCGOV = Partner Country Government; PCNGO = Partner Country NGO; PCPS = Partner Country Private Sector; PCACA = Partner Country Academia; Other: CCM = Country Coordinating Mechanism; ML = Multilateral Organization; OBL = Other (non-U.S. and non-Partner Country) Bilateral; OGOV = Other Government; ONGO = Other Country NGO.

3 United States Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 2003, P.L. 108-25, 108th Cong., 1st sess. (May 27, 2003).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

img

FIGURE 11-1 PEPFAR funding for country-level strategic information in constant 2010 dollars and as percentage of total PEPFAR funding.

NOTES: This figure represents funding for all PEPFAR countries as planned/approved through PEPFAR’s budget codes for country-level Strategic Information activities. The budget codes are the only available source of funding information disaggregated by type of activity and are therefore used in this report as the most reasonable and reliable approximation of PEPFAR investment by programmatic area. Data are presented in constant 2010 USD for comparison over time. See Chapter 4 for a more detailed discussion of PEPFAR’s budget codes and the available data for tracking PEPFAR funding.

SOURCE: OGAC, 2006d, 2007g, 2008d, 2010d, 2011i,j.

the 15 focus countries were to “provide treatment to 2 million HIV-infected people; prevent 7 million new HIV infections; and provide care to 10 million people infected and affected by HIV/AIDS, including orphans and vulnerable children” (OGAC, 2004, p. 7). The treatment and care 5-year targets were based on meeting 50 percent of the estimated need for the focus countries, using estimates made with input from economists based at the Joint United Nations Programme on HIV/AIDS (UNAIDS) and the National Institutes of Health (NIH) (NCV-2-USG) (Donnelly, 2012; IOM, 2007a). The 5-year prevention targets were based on cost estimates from UNAIDS and on approximately half of the expected new infections in the focus countries (Donnelly, 2012; IOM, 2007a). With reauthorization under the 2008 Lantos-Hyde Act and ongoing PEPFAR activities, the main cumulative targets for treatment, prevention and care have increased steadily (see Table 11-1). In December 2011, on World AIDS Day, President Obama announced an increase in PEPFAR’s target number of people on treatment from 4 million to 6 million by the end of 2013 (Obama, 2011).

To accomplish the overall PEPFAR I targets, each partner country mission team was assigned a target to achieve during the initial 5-year implementation period (OGAC, 2003). Starting in FY 2009, under PEPFAR II,

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

TABLE 11-1 Key PEPFAR Targets Under Legislation and Strategy Mandates

  Leadership Acta and First PEPFAR Five-Year Strategy Lantos-Hyde Reauthorization Actb Second PEPFAR Five-Year Strategy Presidential Declaration, World AIDS Day, 2011
Target Timeframe FY 2004–FY 2008 Through FY 2013 Through FY 2014 Through 2013
Targets Treatment for 2 million Treatment for at least 3 million Treatment for more than 4 million Treatment for 6 million
  Prevention of 7 million new infections Prevention of 12 million new infections Prevention of more than 12 million new infections
  Provision of care to 10 million, including OVC Provision of care to 12 million, including 5 million OVC Provision of care to more than 12 million, including 5 million OVC
    Training and retention of 140,000 new health care workers Training and retention of more than 140,000 new health care workers

NOTE: OVC = orphans and vulnerable children.

aUnited States Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 2003, P.L. 108-25, 108th Cong., 1st sess. (May 27, 2003).

bTom Lantos and Henry J. Hyde United States Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008, P.L. 110-293, 110th Cong., 2nd sess. (July 30, 2008).

SOURCES: Obama, 2011; OGAC, 2004, 2009f.

targets were determined at the partner country level by PEPFAR mission teams (OGAC, 2008b, 2009d, 2010c, 2011h).

To inform the targets for each upcoming fiscal year, which are determined as part of the process of developing the COP, mission teams look at programmatic results from previous years (240-33-USG; 636-1-USG). Ideally, targets should be set based on data, including estimated need, and in at least one partner country there appears to have been an evolution toward an increased use of data by the mission team to determine program targets (240-33-USG). However, the epidemiological data needed to support rational targeting are not always available, and the data that are available vary in their reliability (461-16-USG; 461-18-USG). Mission teams described working closely with implementing partners to set program targets (116-1-USG; 461-16-USG; 461-18-USG). One mission team described the target setting process in this way:

[We] work with implementing partners to set targets based on the partners’ budget, disease burden, and previous performance. [We] then aggregate implementing partner’s targets and adjust for over-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

lap to get the overall PEPFAR target. Always need to ensure that their target doesn’t exceed the national number.’ (461-16-USG)

However, the targets are not always realistic and achievable; as one implementing partner interviewee stated about the organization’s program targets, “[It] feels like being asked to make an elephant fly(166-10-USNGO).

Use of Program Targets

Interviewees described using targets for program accountability and planning. At the headquarters (HQ) level, OGAC interviewees described comparing data reported by mission teams to the targets set in the COP (NCV-2-USG; NCV-7-USG). At the partner country level, mission teams used targets for COP planning and to assess whether implementing partners met their goals (196-1-USG; 636-1-USG; 461-16-USG). Some mission teams saw program targets as having limited utility for program management (461-16-USG; 196-1-USG). Other mission teams, however, found the information useful for program planning (116-1-USG; 636-1-USG):

In particular for the PMTCT [prevention of mother-to-child transmission] and treatment indicators, the PEPFAR team has had a process to look back at programmatic results from previous years to inform the targets for the upcoming fiscal year. These programmatic results are useful when developing consensus around the targets and planning of the activities to be implemented in the next year. So, indicator data are used programmatically to inform the managers on how to implement the program especially when trying to scale up.’ (636-1-USG)

OGAC is working toward linking program monitoring targets more closely with financial information. Initially, targets were set using best-guess estimates of what the money could buy, given the costs at the time, without knowing the real costs or knowing what the partner country health system could absorb, particularly in the areas of treatment and care (NCV-11-USG). In 2012 OGAC began an expenditure analysis in 10 countries to better understand the range of unit costing for PEPFAR’s core services in order to help mission teams build budgets and more accurately estimate costs (NCV-11-USG) (Holmes et al., 2012). This type of expenditure analysis will become a routine process after this initial study (Holmes et al., 2012). The increased emphasis on tying targets to financial cost may be due to the fact that, as one interviewee put it, ‘the budget now provides constraints and [we] have to really think about how to leverage resources(NCV-2-USG). The targets are ‘more useful and more realistic now(NCV-2-USG).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Interviewees described an inherent tension between trying to meet program targets while also trying to implement interventions such as investing in quality programs, health systems strengthening efforts, building capacity, and focusing on prevention. These types of activities contribute to PEPFAR goals but could result in lower numbers reached toward the program targets as compared to investing directly in implementing service delivery (331-43-USG; 587-12-USG; 166-3-USG; 166-6-USG; 166-10-USNGO; 116-7-USG; 272-15-PCNGO). As an example, one mission team interviewee described how the focus on program targets can conflict with efforts to build capacity:

PEPFAR funds civil society to do specific projects but this doesn’t teach them how to engage the government, motivate staff, etc. PEPFAR is set up to fund organizations to achieve PEPFAR outcomes/targets. It is hard for PEPFAR to help civil society grow into these roles while also achieving PEPFAR targets.’ (166-4-USG)

A PEPFAR-funded nongovernmental organization (NGO) described trying to achieve the targets in this way:

Ultimately it becomes a number crunching exercise. We are chasing the numbers. We have to find a balance of achieving the target but also rendering a quality service to the OVC. Sometimes it is just the figures that makes a difference—if you do not achieve the target you get “rapped on the knuckles” but if you achieve the target nobody ever asks if you can ensure the quality of the services. We try and render quality services and also meet the targets.’ (272-15-PCNGO)

Use of Evidence to Prioritize Activities

PEPFAR has emphasized the use of epidemiological data and intervention effectiveness data to determine which activities and target populations should be prioritized for implementation in partner countries (NCV-13-ML; NCV-16-USG; NCV-27-ML; NCV-28-ML; NCV-29-ML) (see also the sections later in this chapter on PEPFAR support for epidemiological data and for evaluation and research). Despite this emphasis on using evidence to drive PEPFAR activities, there are examples, particularly from early in PEPFAR’s implementation, where evidence-informed strategies were not employed, such as the emphasis on the abstinence and be faithful components of the “Abstinence, Be faithful, and correct and consistent Condom use” approach (also known as “ABC”) prevention strategy and the lack of approval for needle exchange programs despite epidemiological data supporting the success of comprehensive programs that included needle exchange among people who inject drugs (IOM, 2007b; Lyerla et al., 2012).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

However, there are also clear examples where PEPFAR has functioned as a learning organization, shifting the focus of its activities in response to new evidence. One such example is in the area of using voluntary medical male circumcision (VMMC) to prevent HIV/AIDS via sexual transmission. After WHO and UNAIDS released normative guidance regarding the benefits of VMMC in 2007, PEPFAR began implementing VMMC in countries with high HIV prevalence and low male circumcision rates and it has since become the largest supporter of VMMC for HIV prevention globally (NCV-7-USG) (Goosby, 2012; WHO, 2012). Other examples of PEPFAR’s programs evolving over time to reflect the available knowledge and evidence include moving to a combination prevention approach for the prevention of sexual transmission, moving to comprehensive prevention approaches for people who inject drugs, and shifting the initiation threshold for treatment to higher CD4 counts (Lyerla et al., 2012; Needle et al., 2012; OGAC, 2010a, 2011c). (See also Chapter 5, “Prevention,” and Chapter 6, “Care and Treatment.”)

Although PEPFAR policies have changed in response to emerging scientific evidence, PEPFAR has typically not moved ahead of global standards. PEPFAR usually changes its internal policies only after normative bodies, such as WHO, release appropriate guidelines (NCV-7-USG). The U.S. government (USG) is, however, heavily involved in the process for developing these normative guidelines. For example, OGAC technical working groups (TWGs) include representatives from WHO and UNAIDS, and when these organizations develop new guidelines, they are typically cleared by members of the OGAC TWGs (NCV-7-USG).

In terms of implementing policy changes within PEPFAR, HQ-level TWGs are engaged in putting evidence together, which then goes to the Deputy Principals, followed by the ambassador (U.S. Global AIDS Coordinator), who makes the final decisions about a policy change or moving forward on new topics (NCV-7-USG). PEPFAR previously had a Scientific Steering Committee that met “regularly to ensure that PEPFAR programs [were] scientifically sound” (OGAC, 2007d, p. 168). Since 2011, the PEPFAR Scientific Advisory Board (SAB) has provided guidance to the ambassador on “scientific, implementation and policy issues” related to the HIV/AIDS response (OGAC, 2011a) (NCV-7-USG). See section titled “Implementation Science: The Way Forward” later in this chapter for additional discussion of the SAB.

Conclusion: Target setting has been used to focus PEPFAR activities and for program planning and accountability. PEPFAR has utilized epidemiologic data, normative guidelines, and intervention effectiveness data to drive program activities. Despite some exceptions, especially in the first phase of implementation, PEPFAR has

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

based its programs on available evidence and has responded to new knowledge and scientific evidence as it has emerged.

Alignment of Targets and Priorities with Partner Countries

In PEPFAR II, there has been an increased emphasis on aligning PEPFAR targets with partner country priorities. As stated in the FY 2010 COP guidance, “Annual technical area summary targets should be based on USG support and should feed into the national program five-year goals set through a strategic planning process led by the host country government and supported by key stakeholders” (OGAC, 2009c, p. 52). One mission team interviewee spoke of the need for increased coordination with partner country governments in setting PEPFAR targets:

Going forward we need to have more discussions and involvement with the Ministry of Health. If PEPFAR is going to support the national program then targets should be based on that. There needs to be more communication among [PEPFAR mission team] TWGs and with the ministry when target setting.’ (461-18-USG)

One mission team described its current alignment with the government as ‘the PEPFAR team takes the government vision and targets (from the HIV plan) and tries to align by saying, “Here’s what we can do to meet your goals”’ (240-9-USG).

One mechanism for increased partner country alignment is the PEPFAR Partnership Framework structure (OGAC, 2009b) described in more detail in Chapter 10. Partnership Frameworks are intended “to provide a 5-year joint strategic framework for cooperation between the USG, the partner government, and other partners to combat HIV/AIDS in the country through technical assistance and support for service delivery, policy reform, and coordinated financial commitments” (OGAC, 2009b, p. 3). As of July 2012, 19 partner countries and 2 regions had signed Partnership Frameworks (OGAC, 2012d). Ideally, targets and priorities that are set based on a country’s Partnership Framework would result in alignment between PEPFAR and the partner country government (OGAC, 2009b). In one country, USG interviewees described how the Partnership Framework process is a key aspect of annual COP planning and has helped to align PEPFAR and national priorities (116-1-USG; 116-4-USG), with PEPFAR saying to the partner country government, “We will put your priorities ahead of ours(116-2-USG). A partner country government interviewee in this same country commented on the USG’s efforts to align with the partner country priorities:

Must commend the efforts that PEPFAR has made in recent years. PEPFAR has tried as much as possible to harmonize and

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

align with country priorities, for example, through the Partnership Framework. This has been advocated at the highest level. This is a strong achievement.” (116-16-PCGOV)

PEPFAR’s priorities and country government priorities, however, may not always align. In one partner country, for example, government interviewees described how during the Partnership Framework process, the country’s prevention priorities did not match PEPFAR’s priorities (587-7-PCGOV; 587-8-PCGOV). Despite some exceptions, however, interviewees across partner countries felt that the Partnership Framework structure was helping to improve alignment of PEPFAR and partner country priorities (116-4-USG; 116-16-PCGOV; 166-10-USNGO; 272-5-PCGOV; 272-36-USG).

PROGRAM MONITORING DATA

PEPFAR’s largest and most sustained effort to create knowledge has been the generation of program monitoring data to track results and report on PEPFAR achievements to Congress. The following sections describe several interrelated aspects of PEPFAR’s program monitoring system: collection and reporting, indicator selection and appropriateness, alignment and harmonization with partner countries and other stakeholders, data quality, and data use.

Collection and Reporting

Program monitoring data are collected by staff at PEPFAR-supported sites such as clinics and community-based programs. Partners who implement programs with PEPFAR funds collate PEPFAR indicator data from the sites that they operate or support and report these data to their respective PEPFAR funding agency in country, e.g., the U.S. Agency for International Development (USAID), the U.S. Centers for Disease Control and Prevention (CDC), etc. (GAO, 2011a). Data from different implementing partners are aggregated by agency and then across mission team agencies before being submitted to OGAC by the in-country PEPFAR SI Liaison (GAO, 2011a). Often, implementing partners and site-level staff also carry out data collection and reporting to meet their own organizational reporting requirements, as well as the reporting requirements for partner countries. The degree to which this data reporting uses indicators and processes that overlap with PEPFAR varies; these issues are discussed in more detail later in this section of the chapter.

OGAC provides guidance defining the indicators in PEPFAR’s program monitoring system and the level at which each indicator is to be reported (see Table 11-2). PEPFAR mission teams report data for the required indi-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

cators centrally to the OGAC SI office on an annual basis, with a subset of indicators (7 indicators in PEPFAR I and 8 indicators in PEPFAR II) also reported semi-annually (NCV-2-USG) (GAO, 2011a,f). Over the course of PEPFAR, there have been a few iterations of program monitoring indicator guidance released by OGAC (OGAC, 2005c, 2007f, 2009e). As summarized in Table 11-2, the first round of indicator guidance, released in 2005, defined 65 indicators to be reported annually to OGAC. The next indicator guidance, issued in 2007, increased the number of centrally reported indicators to 76. The Next Generation Indicators (NGIs) guidance, introduced in 2009 for reporting beginning in FY 2010, reduced the number to 31 centrally, routinely reported indicators (25 programmatic indicators, 1 additional programmatic indicator if a partner country has a signed Partnership Framework, and 5 national-level indicators).

With the introduction of the NGIs, OGAC created a new category of indicators (n=31) that are essential for mission teams to collect but that do not have to be routinely reported centrally (see Table 11-2). The rationale for these indicators is to ensure that mission teams have specific data available at the partner country level to respond to ad hoc requests for information from Congress (NCV-2-USG). OGAC indicator guidance also includes definitions for additional indicators that are recommended for mission teams to use for program management, if applicable to that country’s program; however, these indicators are not reported centrally. The number of this type of indicator increased substantially with the introduction of the NGIs, from 23 indicators to 92 recommended indicators (see Table 11-2). The evolution of indicators in the new guidance is discussed in more detail later in this section of the chapter.

From FY 2006 to FY 2009, COPs and program monitoring data were submitted from PEPFAR mission teams to OGAC via an electronic, Internet-based system called the Country Operational Plan Reporting Sys-

TABLE 11-2 Number of PEPFAR Indicators by Reporting Status and Year of Indicator Guidance

  2005 2007 2009
Routinely Reported to OGAC 65 76 31
Not Routinely Reported to OGAC 23 23 123
Essential for PEPFAR mission teams 31
Recommended for PEPFAR mission teams 23 23 92
Total 88 99 154

NOTES: One indicator defined in the 2009 guidance is routinely reported only from programs that have signed a Partnership Framework with the partner country. One indicator that was previously not routinely reported was elevated to being routinely reported starting in FY 2011, increasing the total number of routinely reported indicators to 32.

SOURCES: OGAC, 2005c, 2007f, 2009e, 2012b.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

tem (COPRS) (OGAC, 2005b, 2006c, 2007e, 2008b). A second iteration of the system, COPRS II, was to be launched in July 2009 after being redesigned with input from “a series of focus group discussions with USG field teams, TWGs, PEPFAR Coordinators, and the Deputy Principals to come up with a solution and long-term vision of a unified system for foreign assistance” (OGAC, 2009d, p. 35). Although a contractor was hired to work on developing COPRS II, contractor issues led to the system not being completed and, as a stopgap measure, OGAC utilized spreadsheets for data submission (NCV-2-USG). Concurrently, the U.S. Office of Management and Budget was working on an effort to consolidate and reduce the number of data systems used within the Department of State (DoS) and requested that OGAC begin using an existing DoS system called FACTS Info (NCV-2-USG). The COPRS II development effort was canceled, and OGAC has had a process of transitioning to FACTS Info (NCV-2-USG). The PEPFAR module in FACTS Info was launched in January 2012 and includes all historical program data (OGAC, 2012c). The new system is being used to support all business cycles, including the COP, Annual Program Results, and Semi-Annual Program Results (OGAC, 2012c). Moving forward, OGAC would like for the FACTS Info system to be able to generate useful reports to the field, support internal analytics, and be used to link program monitoring data to financial data, UNAIDS data, and Global Fund databases (NCV-2-USG).

Although temporary, the lack of a reporting database system for FY 2010 and FY 2011, which coincided with the committee’s partner country visits, had negative ramifications for mission teams. In particular, using the program monitoring data for analysis was cumbersome, and not having a system to access past years of data submitted was an issue (240-8-USG; 240-33-USG; 331-1-USG; 331-48-USG). In spreadsheet form, data were “hard to manipulate,” and, you ‘couldn’t look at the data across the country level(331-1-USG). As one interviewee stated, ‘It boggles the mind that a multi-billion-dollar program is run by spreadsheets(331-1-USG).

Another consequence of shifting data systems is that OGAC’s ability to access and utilize data across the various databases and systems that have been used to collate program monitoring data is limited (NCV-2-USG). Only a small number of indicators were made available to the committee in response to requests for data from PEPFAR I. Given the importance of program monitoring data for knowledge management, it is critically important for OGAC to have all program monitoring data available in a usable format.

Conclusion: PEPFAR’s system of indicators to monitor program activities has faced technological challenges that have made it difficult for both PEPFAR and external stakeholders to utilize and access data, and it is critical for the ongoing monitoring of the program

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

that these challenges be resolved. As technology is updated and new data systems adopted, historical data also need to be maintained and kept accessible to allow for assessments of trends going back to the earliest years of implementation.

Indicator Selection and Evolution

Interview data collected by the committee during country and non-country visits suggested that what gets measured gets done (NCV-2-USG; NCV-23-USNGO; 272-36-USG; 396-1920-USG). OGAC’s selection of indicators for program monitoring therefore, clearly plays an important role in which program activities are prioritized in partner countries. Figure 11-2 shows the primary areas of programmatic activity captured by PEPFAR’s program monitoring indicators, the number of indicators that have been centrally reported in each area, and how those numbers have changed over time with the different iterations of indicator guidance.

Determining indicators for PEPFAR program monitoring has historically been a multi-stage process. The first iteration of indicators (2005) was developed through a discussion among the relevant USG agencies, including

img

FIGURE 11-2 Number of indicators routinely reported to OGAC by Next Generation Indicator (NGI) reporting category and guidance year.

a In 2005 and 2007, OGAC had a set of indicators that were defined as impact indicators that were designed to look at the broad impact of programs. These indicators cannot be classified as falling into any specific NGI reporting category because they cut across program areas.

NOTES: Indicators defined in the 2005 and 2007 indicator guidance were categorized according to 2009 NGI Guidance categories to allow for comparison over time. HSS = Health Systems Strengthening.

SOURCE: OGAC, 2005c, 2007f, 2009e.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

CDC, USAID, the Health Resources and Services Administration (HRSA), Peace Corps, and the Department of Defense (DoD), with consideration of existing indicators being used by USG agencies, as well as indicators being used globally (NCV-3-USG). The initial indicators were also vetted by partner country mission teams (NCV-3-USG). Determining the NGIs, which led to a dramatic change in the program monitoring indicators, involved convening a PEPFAR interagency TWG, which included multilateral partners such as WHO, PEPFAR-funded implementing partners, and civil society participants (OGAC, 2009e). OGAC was able to involve partners and contractors working in the field to a larger extent during the process for determining the NGIs than in previous indicator determination processes (NCV-3-USG).

Although it is appropriate that the indicators have evolved as PEPFAR has matured and its activities have changed, revising program monitoring indicators creates challenges for mission teams and implementing partners:

The transition from the old to the new indicators is a challenge; most existing registers and tools need to be adjusted and levels of disaggregation may need to be adjusted for proper reporting. There is often not enough money for advocacy, adjusting registers, and re-training workers. Quality assurance is also a challenge when old tools are not eliminated or withdrawn from the field and there is not widespread adoption of the new tools—when there are not enough tools available, people will revert to using antiquated tools.’ (331-34-USNGO)

To address these challenges, OGAC HQ and mission teams have provided technical assistance with the collection of the new indicators, but some confusion has remained about what the new indicators mean and how to collect them (331-1-USG; 587-9-USG; 636-1-USG; 396-56-USNGO).

Another challenge with revising the indicators is that the introduction of new indicators and the elimination of previous indicators, as seen with PEPFAR’s transition to the NGIs, limits the ability to look at trends in program performance over time. Only nine PEPFAR indicators can be tracked across all partner countries for the duration of PEPFAR because these indicators are both reported centrally to OGAC and their definitions that have remained consistent over time (see Table 11-3).

Despite losing the ability to follow some indicators long-term with the transition to the NGIs, the NGIs were seen by some interviewees as an improvement because they reflected feedback from the field, resolved issues with earlier indicators, increased clarity around some indicators, and decreased reporting burden (331-23-USNGO; 331-34-USNGO). However, interviewees reported that there were still issues with the NGIs, including confusion around definitions, difficulties with measurement, and the perception by

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

TABLE 11-3 PEPFAR Indicators Consistent Across the Duration of PEPFAR

NGIs Reporting Area Indicator Definition Indicator Level
Care
Number of HIV-positive patients in HIV care or treatment (pre-ART or ART) who started TB treatment PEPFAR Output
Health Systems Strengthening
Number of testing facilities (laboratories) with capacity to perform clinical laboratory tests PEPFAR Output
Prevention
Number of HIV-positive pregnant women who received antiretrovirals to reduce risk of mother-to-child-transmission PEPFAR Output
Percent of HIV-positive pregnant women who received antiretrovirals to reduce the risk of mother-to-child-transmission National Outcome
Number of individuals who received testing and counseling services for HIV and received their test results PEPFAR Output
Treatment
Number of adults and children with advanced HIV infection newly enrolled on ART PEPFAR Output
Percent of adults and children with advanced HIV infection receiving antiretroviral therapy PEPFAR Output
Percent of adults and children with advanced HIV infection receiving antiretroviral therapy National Outcome
Percent of adults and children known to be alive and on treatment 12 months after initiation of antiretroviral therapy PEPFAR Outcome

NOTE: Indicator level classified according to the 2009 NGI Guidance. ART = antiretroviral therapy; TB = tuberculosis.

SOURCES: OGAC, 2005c, 2007f, 2009e.

some interviewees that there were still too many indicators (587-9-USG; 331-1-USG; 587-9-USG; 636-1-USG; 396-56-USNGO). Additionally, the indicator changes led to a number of challenges, including the need for staff retraining and new data collection tools and the loss of the ability to examine time trends (331-34-USNGO; 331-23-USNGO; 272-27-USG).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Conclusion: PEPFAR’s program monitoring system has evolved over time to include a greater number of indicators but with fewer indicators that are centrally reported, which has resolved some challenges with clarity of definitions and reporting burden. There is a need for a PEPFAR program monitoring strategy that can respond to feedback, adapt to emerging program priorities, and accurately reflect program activities and outcomes. However, this adaptability over time needs to be balanced with the reality that changes in indicators place a burden on partner country programs and limit the comparability of PEPFAR monitoring data, hampering the ability to monitor trends.

Program Monitoring Indicator Appropriateness

Program monitoring indicators are designed to be used by implementing partners and mission teams to assess program performance; the indicators are not designed to “adequately capture every aspect of a comprehensive program” (OGAC, 2009e, p. 5). Nonetheless, many interviewees expressed frustration that the indicators did not reflect their activities and were not well aligned with what interviewees perceived as OGAC program priorities. Areas that interviewees identified as not being captured well included efforts related to tuberculosis (TB); changing social norms, including stigma reduction; policy development; and overall health system strengthening, including strategic information activities and laboratory strengthening (331-1-USG; 587-12-USG; 196-6-USG; 196-26-USG; 636-9-USACA; 935-9-USG; 542-6-ML; 396-8-PCNGO; 396-18-USG; 396-1920-USG; 166-4-USG; 461-18-USG).

In particular, PEPFAR country programs with a strong focus on capacity building and technical assistance noted that they were not able to report on these prioritized program activities because of the lack of relevant indicators (331-1-USG; 331-3-USG; 196-1-USG; 196-6-USG; 196-28-USG; 542-6-ML). Although they don’t contribute to the ‘big numbers that are meaningful to Congress(196-28-USG), there is a need to be able to track and document these types of activities in order to recognize their importance and to evaluate these types of approaches, especially as PEPFAR transitions to a more country-led program approach. PEPFAR indicators also do not capture service delivery quality well, despite a stated intention to include this component (OGAC, 2009e). When local programs emphasize program quality, which can sometimes reduce the numbers treated and numbers accessing care, these efforts are not measurable and attributable in the indicators (331-3-USG; 331-43-USG).

Another key issue cited by interviewees was that the PEPFAR indicators are focused on inputs and outputs and not on outcomes and impact, making it difficult to determine the effectiveness of PEPFAR’s efforts (587-9-USG; 587-22-USG; 166-23-USG; 461-4-USG; 461-14-USG; 396-15-USNGO). The routinely reported indica-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

tors are weighted more toward outputs than outcomes and impact (OGAC, 2009e). This limited ability to measure impact is a serious issue at both the country program level and the central OGAC level; as one interviewee said, ‘When you invest this much and spend this much time, you need to look at impact(587-22-USG).

To address indicator limitations, some mission teams and their implementing partners have developed custom indicators to allow for better monitoring of partners and initiatives (196-1-USG; 116-1-USG; 116-7-USG; 461-17-PCNGO; 934-21-USG). Examples of custom indicators include nurse exam passing rates; indicators to track reproductive health services; detailed orphans and vulnerable children (OVC) measures; measures of reduction in stigma; capacity-building measures; training indicators; quality improvement targets; health systems strengthening measures; TB-activity-related indicators; and process indicators for lab strengthening (196-USG; 116-1-USG; 116-7-USG; 461-17-PCNGO; 934-21-USG). These custom indicators have enabled partners to provide more information about their programs and to give more detail about how programs are performing and contributing to the national program (116-1-USG). These custom indicators are primarily used by mission teams and have not been adopted by OGAC. In one case, described by a mission team interviewee, ‘the country team developed indicators [to reflect efforts around technical assistance] but OGAC rejected them(196-28-USG).

Conclusion: The current PEPFAR indicators do not reflect all of PEPFAR’s stated prioritized goals and activities and are focused primarily on inputs and outputs and not on outcomes and impact. For these reasons, the program monitoring system has limited utility for determining the effectiveness of PEPFAR’s efforts.

A subsequent section of this chapter will describe PEPFAR’s evaluation and research activities, and the chapter will culminate with recommendations for how these activities can complement program monitoring data and be strategically coordinated to address the need to monitor program performance and to assess effectiveness and impact.

Alignment with Partner Country HIV/AIDS Monitoring and Evaluation Systems

To encourage a more coordinated response to the HIV/AIDS epidemic, the 2004 Consultation on the Harmonization of International AIDS Funding, which brought together representatives from governments, donors, international organizations, and civil society, set forth three key principles for donor harmonization known as the “Three Ones Principles” (UNAIDS, 2004). According to these principles, each country should have one agreed-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

upon HIV/AIDS Action Framework, that there should be one National AIDS Coordinating Authority, and the “Third One,” which called for donors to endorse “One agreed country-level monitoring and evaluation (M&E) system” (UNAIDS, 2004, p. 2). The “Third One” Principle has proven especially challenging as donors struggle to reconcile their own reporting requirements with existing country monitoring and evaluation (M&E) systems.

During PEPFAR I (2004–2008), the high level of investment, the rapid expansion of HIV services, and the pressure to gather information to report results and demonstrate feasibility and scalability of HIV/AIDS service delivery programs, led to the creation of parallel M&E systems (Porter et al., 2012). Aligning with partner country M&E systems was complicated by the fact that many countries had weak HIV/AIDS M&E systems and most of these systems were not fully functional (Porter et al., 2012).

Interviewees in partner countries described how, initially, PEPFAR emphasized developing M&E capacity to report for PEPFAR rather than improving existing partner country M&E systems (331-24-PCGOV; 587-2-USG; 587-9-USG; 636-1-USG; 636-9-USACA; 166-4-USG; 166-12-USG; 166-34-PCGOV; 272-27-USG; 461-11-PCGOV). The reasons cited for PEPFAR’s parallel M&E system included that the existing partner country systems did not capture the information needed for OGAC to report to Congress (636-9-USACA; 166-1-USG; 166-10-USNGO; 166-12-USG; 272-27-USG; 461-15-USG; 461-20-PCPS; 934-21-USG), that there were multiple systems within the government making it difficult to integrate and align with the existing systems (331-34-USNGO; 196-8-ML; 116-16-PCGOV; 396-1920-USG), and that there were issues with the quality of partner country data and questions about data ownership (461-15-USG).

At the site level, with inadequate alignment of reporting and no single country-level M&E system, implementers collect data not only for government indicators but also for PEPFAR indicators and any additional indicators required by each implementing partner (116-12-PCNGO; 166-4-USG; 461-17-PCNGO). PEPFAR often requires more information than the government (636-9-ACA; 116-12-PCNGO; 166-4-USG; 166-15-USACA; 272-27-USG), and collecting data for both partner country M&E systems and PEPFAR places a large burden on staff:

There is a long list of NGIs data, but it is a challenge to get that data. It is not routinely reported in the national system, so it requires additional data collection, which is a burden on limited facility staff.’ (NCV-6-USNGO)

In one partner country, for example, the need to report to PEPFAR was described as having a negative impact on reporting to the Ministry of Health (MOH) because the same staff collect information for both systems (461-15-USG; 461-16-USG):

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

The public-sector sites that PEPFAR supports have to report to both PEPFAR and the Ministry of Health. At these sites, PEPFAR becomes the priority because they have money and the government becomes the second priority. The public-sector staff that have to report to both PEPFAR and the MOH spend a significant amount of time on reports. Staff positions are not filled and the technical staff’s time is filled with reporting.’ (461-15-USG)

OGAC’s shift to the NGIs for program monitoring, described earlier in the chapter, was done in part with the goal of increasing alignment with country M&E systems. As described in the NGI guidance, the shift to the NGIs “attempts to minimize PEPFAR-specific reporting requirements to allow PEPFAR mission teams more flexibility to design M&E plans in line with host countries and strikes a better balance between support for USG reporting needs and national M&E systems” (OGAC, 2009e, p. 6). Although interviewees described some challenges with alignment and reporting burden as ongoing even after the introduction of the NGIs, many noted that with PEPFAR II there has indeed been much more progress aligning with national M&E systems (240-20-ML; 331-18-USNGO; 587-9-USG; 166-1-USG; 272-27-USG; 461-1-USG). Box 11-1 highlights some of these alignment efforts. An increased

BOX 11-1
Select PEPFAR Efforts to Align with
Partner Country M&E Systems

•   Revision of national data collection tools to ensure that PEPFAR indicators are incorporated into and aligned with the Ministry of Health (MOH) M&E system. (331-18-USNGO; 636-9-USACA; 636-18-ONGO; 166-12-USG; 461-18-USG)

•   Efforts to align PEPFAR indicators with MOH indicators. (636-9-USACA; 636-18-ONGO; 166-1-USG; 166-4-USG; 166-12-USG; 166-15-USACA; 272-27-USG; 461-15-USG; 461-18-USG)

•   Cooperative agreements to unify partner country health management information systems and efforts to integrate vertical systems within a particular country (331-24-PCGOV; 196-8-ML)

•   Use of indicators already being gathered by the national system as a proxy for PEPFAR indicators. (587-9-USG)

•   Health workers collect one set of data that is separated out when it is aggregated for the monthly reports. One copy goes to the PEPFAR partner, and one copy goes to the MOH system. (166-12-USG)

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

emphasis has also been placed on strengthening national systems of data collection and reporting (240-20-ML; 636-1-USG; 636-18-ONGO; 166-4-USG; 272-27-USG; 461-20-PCPS).

PEPFAR has articulated a goal of aligning its monitoring and evaluation system with partner country governments and has made some progress toward this goal; however, in general, alignment efforts have not yet fully succeeded, nor have they achieved the desired magnitude of reduction in reporting burden for partner countries and implementing partners. As national health information systems are strengthened, there should be less need for PEPFAR to rely on separate systems in order to obtain information. “Building national M&E systems requires sustained efforts over long periods of time with local leadership, commitment, and extensive stakeholder engagement” (Porter et al., 2012, p. S122). As PEPFAR continues its efforts, alignment of its M&E systems with those of partner countries, is imperative as a critical component of country ownership (Holzscheiter et al., 2012). Further discussion of PEPFAR and national health information systems can be found in Chapter 9 on health system strengthening.

Harmonization with Global HIV/AIDS Indicators and Global Multilateral Reporting Systems

Harmonization of Global HIV/AIDS Indicators

U.S. government operations in general have accountability [. . . .] But, the monitoring framework for programs [. . .] should really be harmonized with other donors. Because it’s such an incredible waste of money and of time, particularly for the poor nationals who have to fill in a different form for each donor but on the same things. And then each time a little bit different. That undermines capacity in countries.” (NCV-14-ML)

There have been several global initiatives to harmonize HIV/AIDS indicators, although there are still hundreds of indicators in use (NCV-7-USG). One such harmonization initiative, called the Monitoring and Evaluation Reference Group (MERG), is sponsored by UNAIDS (Porter et al., 2012). The goal of this initiative is to harmonize HIV/AIDS indicators globally, and it has played an important role in convening agencies and helping actors come to consensus (NCV-7-USG). MERG uses an anonymous indicator review process for proposed indicators (NCV-7-USG). With MERG input, UNAIDS developed a core set of indicators, known as the United Nations General Assembly Special Session (UNGASS) indicators, which countries report on biannually (OGAC, 2009e; UNAIDS, 2009). The development of these indicators

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

has allowed for comparisons of key HIV/AIDS indicators across countries globally and over time.

TABLE 11-4 Level of Harmonization of Next Generation Indicators with Global Indicators

  Harmonized w/UNGASS Indicators Harmonized w/Other Global Indicators PEPFAR-Specific Indicators (Not Harmonized) Total
Total
(%)
28
(18)
83
(54)
43
(28)
154
Routinely Reported to OGAC 8
(26)
12
(39)
11
(35)
31
Not Routinely Reported to OGAC 20
(16)
71
(58)
32
(26)
123

SOURCE: OGAC, 2009e.

When PEPFAR introduced the NGIs, a stated goal was to increase harmonization with global HIV/AIDS indicators (OGAC, 2009e). Table 11-4 shows the level of harmonization of the NGIs with UNGASS indicators and other global indicators used by WHO, UNAIDS, and the Global Fund, based on OGAC’s self-classification (OGAC, 2009e). Of the NGIs that are required to be routinely reported to OGAC HQ, 35 percent were not harmonized with either UNGASS indicators or other global indicators (OGAC, 2009e). The lack of harmonization with global indicators may be a reflection of activities that are unique to PEPFAR; however, it contributes to the continued need for parallel M&E systems in order to collect PEPFAR program monitoring data.

Harmonization of Reporting with Other Multilateral Organizations

OGAC has worked closely with both UNAIDS and the Global Fund to harmonize data reporting. OGAC meets separately every 6 months to a year with UNAIDS and the Global Fund to compare data, which has led to improved data quality and increased consistency of reported data (NCV-3-USG; NCV-21-ML). The Global Fund and OGAC now do joint releases of results to provide explanations to the public about data overlap, and they have provided shared monitoring guidance to countries (NCV-21-ML) (Global Fund, 2011). The Global Fund and OGAC have started using common indicators but there are always some separate reporting requests from either side. Some countries are partially funded by PEPFAR and partially funded by the Global Fund, so they have come up with financial criteria of minimum

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

thresholds for the contribution of PEPFAR and the Global Fund toward national results (NCV-21-ML).

The next stage was we started to calculate overlaps that we would publish a Global Fund figure, a PEPFAR figure, and then a joint figure, which showed the unique number of individuals reached by both. And I think the next stage is to start to harmonize around the impact and the outcome data.” (NCV-21-ML)

PEPFAR’s Contribution to the Development of Global Indicators

In addition to being an active participant on the MERG, OGAC has worked with multinational organizations to identify and develop program monitoring indicators. For example, in the area of gender, OGAC, together with civil society, national governments, and United Nations (UN) partners, identified eight key areas that should be measured related to gender and HIV (NCV-10-USG). OGAC and UN partners compiled indicators to monitor nutrition and HIV that are not mandated by any one donor but that serve as a resource for countries that are interested in monitoring and evaluating their nutrition and HIV efforts (NCV-17-USG). Generally, OGAC has tried to serve as a resource for countries as they are putting together platforms around their programmatic efforts (NCV-17-USG).

Conclusion: The need to quickly collect data and measure results at the onset of PEPFAR contributed to the development of PEPFAR-specific data collection systems, which has limited harmonization with partner countries and the global HIV/AIDS community. OGAC has worked closely with global actors such as UNAIDS and the Global Fund to harmonize program indicators and validate reporting. With recent efforts, PEPFAR has also made progress in modifying its program monitoring system to reduce reporting burden and to improve alignment with partner country programs. However, further modifications could be made to improve the clarity of indicators, including eliminating PEPFAR-specific language in the indicator guidance, further reducing the reporting burden, improving indicator harmonization with global indicators, and better aligning with partner country HIV monitoring and health information systems for data collection at the program and country level.

Data Quality

PEPFAR implementing partners are responsible for validating and aggregating program monitoring data and ensuring data quality from each of

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

their sites (GAO, 2011a). OGAC has provided data quality assurance tools for use at sites to address common data quality issues such as double counting (multiple counting of individuals for the same indicator). However, data quality remains a challenge (GAO, 2011a; OGAC, 2007b).

Double counting was an issue that was mentioned by several interviewees (331-8-PCNGO; 331-23-USNGO; 587-9-USG; 587-18-PCGOV; 461-14-USG; 461-18-USG; 240-15-USG). The lack of unique identifiers in some countries and people getting tested multiple times in different locations have contributed to double counting in the area of voluntary counseling and testing (331-8-PCNGO; 331-23-USNGO; 587-9-USG; 587-18-PCGOV; 461-14-USG; 461-18-USG). Double counting was also an issue in the area of care and support (240-15-USG). Interviewees reported a few strategies for reducing the amount of double counting, including

Work with implementing partners to verify overlap in the output indicators. If don’t have sufficient proof of service provided, don’t count the services in their reported numbers. Have a process for rationalization to avoid having partners work on the same sites. Partners agree on who gets to count and report cases and services.’ (461-1-USG)

More broadly, implementing partners and mission teams described various data quality assessment initiatives (166-12-USG; 272-27-USG; 934-21-USG). One mission team described a process of re-abstracting medical charts to verify data and asking questions to assess processes and protocol at directly funded NGO and treatment sites (587-9-USG). Other mission teams described using a tool developed by the organization MEASURE/Evaluation for data quality assessments (587-3-USG; 636-18-ONGO). Some mission teams hired partner organizations to do data quality assessments and to work with implementing partners on data quality processes (636-18-ONGO; 461-1-USG; 461-20-PCPS). One such organization described the benefits of its data quality assessment work in this way:

In the beginning, people were apprehensive of the data quality assessments. They thought it would affect their funding, but now implementing partners are grateful and it helps them see their strengths and weaknesses and helps partners strengthen where they are weak. [. . .] There has been spillover from the data quality assessments and now many of the partners are doing data quality assessments themselves.’ (461-20-PCPS)

The PEPFAR SI Liaison in country, representing the Coordinator’s office, is responsible for the final steps of assuring data quality and for submission of data to OGAC. In many countries, SI advisors from HQ validate data before submission (GAO, 2011a). If there are issues with the

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

data, then the OGAC SI team communicates with mission teams to clarify the issues. Recently, OGAC has instituted a more formal process where all SI advisors, as well as the SI support persons working with the various country-level TWGs, contribute to mission team data review (GAO, 2011a).

Because of the lack of an operating program monitoring database as described earlier in this chapter, in fiscal years 2010 and 2011, mission teams reported program monitoring data to OGAC using spreadsheets (OGAC, 2009d, 2010c). Data were verified by OGAC staff using data cleaning spreadsheets (Microsoft Excel) and data cleaning checklists (NCV-2-USG). OGAC prioritized data cleaning of seven key indicators that are submitted annually to Congress with the cleaning of the other indicators ‘taking an additional 2-3 weeks(NCV-2-USG). The data clearance process was described by OGAC SI staff as more than a validity test: ‘OGAC is not just checking to make sure that the files submitted by the countries are there, but there is a creditability and reliability check—is what the countries are reporting sensible for the type of program they have?(NCV-7-USG).

Although OGAC interviewees described their evolving and multistage processes for data collection, validation, and availability, the committee’s experience with PEPFAR program monitoring data raised some concerns. When the committee requested program monitoring indicators, these data, beyond the seven key indicators that are reported annually to Congress, were not readily available. And when indicator data were made available, the committee’s examination of these data revealed numerous discrepancies. A similar observation has been made by others (Bryant et al., 2012). These data discrepancies led the committee to question the mission team and OGAC HQ verification processes. These data issues, along with limited data availability, made it difficult for the committee to fully assess PEPFAR’s efforts. (See also Chapter 2 on the evaluation scope and approach and the more detailed description of methods in Appendix C.)

Use of Program Monitoring Data

Data Use at OGAC HQ

At OGAC HQ, one of the primary uses of program monitoring data is to provide annual reporting to Congress and respond to ad hoc congressional requests (NCV-2-USG). ‘The data reporting played a huge role in getting funding. [PEPFAR was] one of few international programs reporting results(NCV-3-USG). Additionally, program monitoring data reported by mission teams were described as being at the ‘heart of internal decision making at OGAC(NCV-7-USG). These data are used for budgets, models, congressional budget justifications, and for the strategic plan (NCV-7-USG). OGAC also reported using these data to track ‘Where they’ve been and where they think

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

they’re going. [T]he data led, in part, to the ability to push for the 2013 treatment target of six million(NCV-7-USG). Program monitoring data are also used for programmatic decision making, informing engagement with partner countries, providing insights into what technical assistance countries need, and monitoring partner countries’ responses to the HIV epidemic (NCV-2-USG).

Data Use by Mission Teams and Implementing Partners

Mission team interviewees described using program monitoring data for multiple purposes. Most commonly, program monitoring data were used by mission teams to examine achievements from prior years, to review and monitor partner performance, and to guide priority activities (240-15-USG; 636-1-USG; 166-12-USG; 272-22-USG; 461-16-USG; 461-20-PCPS; 587-9-USG; 196-1-USG; 116-1-USG). Data were also used to analyze partner overlap by region (461-16-USG; 461-20-PCPS), to determine programmatic trends for different partners (636-1-USG), and to help partners set appropriate targets (240-33-USG; 636-1-USG).

Although PEPFAR mission teams provided examples of how PEPFAR program monitoring indicator data were useful for evaluating program performance, overall, interviewees described how the burden of reporting indicator data interfered with the ability to consistently use the data in a meaningful way (240-1-USG; 331-48-USG; 587-22-USG; 461-15-USG; 934-2-USG). The high reporting burden limited available time to look at the data (240-3-USG; 461-16-USG), and much of the program monitoring data were not used beyond the purpose of reporting to OGAC (461-3-USG; 331-1-USG; 272-25-USG). As one interviewee stated, ‘PEPFAR has an onerous reporting burden “way, way beyond the pale,”’ and ‘reporting pulls limited staff attention away from where it should be focused—monitoring and continual assessment with field visits(587-22-USG). Additionally, discontinuing the use of COPRS I, as described earlier in the chapter, led to limited ability to manipulate program monitoring data for analysis, which affected mission team use of the data (331-1-USG).

Implementing partners also bemoaned the heavy burden of reporting program monitoring data, particularly on clinical staff (NCV-6-USNGO; NCV-8-USACA; NCV-14-ACA; 396-8-PCNGO), and how it resulted in limited use of the data (461-15-USG). As one interviewee said:

In the current system [. . .] we collect everything but very little [data] will [be] analyzed and utilized for our program or for instance, in making or for policy advocacy. Very little. I think that’s like 20 percent of data. So now we try to collect anything and just put it in the corner. So I think that we can review the term and the workload for the people we brought in the program, which [would] mean that we could save costs and have more time to improve the program.” (396-8-PCNGO)

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

This view was reinforced by a mission team interviewee:

In the service delivery sites and at the districts the numbers are not being analyzed or used. [. . .] Sites are concerned with moving the data out and up. As long as the number goes out the door, the sites are not worried about it, and everybody is happy as long as the report gets out.’ (461-15-USG)

In addition to the burden of reporting, collected data were not perceived by interviewees as being beneficial for improving patient outcomes:

Indicators—doing monthly reporting is totally burdensome. Facilities do not see the data as helping the patient and [data are] seen as something you have to do because of the program. It would be fine if the report was useful in and of itself but that is not always the case. In some clinics, have multiple people working all day on the data and [their] whole job is reporting. [. . .] All of the reporting of indicators is not benefitting the patients; you just have to do it because you are in the PEPFAR program.’ (NCV-8-USACA)

As one USG interviewee stated, ‘If they could show sites how the indicators are useful for program management then the process of collecting the indicators would become meaningful for the sites(587-9-USG).

As described previously, in response to feedback from the field about the burden of reporting program data, the introduction of NGIs reduced the number of centrally reported indicators (see Table 11-2). Although some improvement with the introduction of the NGIs was noted by interviewees (331-23-PCGOV; 331-34-USNGO), the number of indicators being reported was still perceived as burdensome (587-9-USG; 461-15-USG; NCV-6-USNGO; NCV-8-USACA). ‘PEPFAR is asking for low-level indicators to be reported at the high level when they are only needed at the facility level(461-15-USG). Another interviewee stated that there were ‘Too many indicators, overwhelming, purpose is not clear and not useful for country decision making(587-9-USG).

Data Use by Track 1.0 Partners4

The four Track 1.0 partners involved in the early implementation of PEPFAR-supported care and treatment programs represent a subset of

__________________

4 Track 1.0 partners in this report refers to four partners that were the primary large-scale implementers of ART in PEPFAR’s centrally funded Track 1.0 program. (For more information, see Appendix C, “Methods”). These partners also implemented other HIV services and programs, and there were also other centrally funded Track 1.0 partners in other program areas.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

implementing partners that are active in multiple countries and that have distinct opportunities and capabilities for data use. In addition to reporting applicable program monitoring indicator data to mission teams within the partner countries in which they implement programs, Track 1.0 partners also submit a separate, core set of facility-based treatment and care data to CDC. Additionally, Track 1.0 partner country offices report results separately to the HQ of their own organizations (240-19-USACA; NCV-5-USACA; NCV-6-USNGO).

Track 1.0 partners support sites in multiple countries and have been able to use site-level data to identify and share best practices across Track 1.0 partner countries (NCV-4-USACA; NCV-5-USACA; NCV-8-USACA). For example, as one Track 1.0 partner described:

An advantage to being in several countries was that we were able to show differences among countries and site data. It was useful for the programs to see what other countries are doing and learn from each other. Used the data for program improvement, mortality coming from opportunistic infections, adherence failure, and infant infection rates. In-country teams responded to the data and made changes based on what [was] learned from other sites.’ (NCV-4-USACA)

Additionally, Track 1.0 partners have used data to target technical assistance and to improve sites (NCV-4-USACA; NCV-5-USACA). For example, one Track 1.0 partner used data to look at clinics with no children on ART and determined that the cause was related to an issue with availability of pediatric drug formulations (NCV-4-USACA). Another Track 1.0 partner described doing an annual facility survey to understand the context in which its care and treatment sites operated (NCV-5-USACA). The survey, used to monitor sites and target technical assistance, provided information about access to family planning and availability of male circumcision and a range of prevention services as well as information on availability of TB laboratory diagnostics (NCV-5-USACA).

Conclusion: There are some good examples of the use of PEPFAR program monitoring data at the mission team, implementing partner, and HQ levels, but the preponderance of the data collected does not seem to be routinely utilized. One major contributing factor is PEPFAR’s requirement for collection and reporting of a large amount of program monitoring data, which has placed a large administrative burden on implementing partners and mission teams, which detracts from efforts for data analysis and use.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Summation for Program Monitoring Data

PEPFAR has placed a strong emphasis on collecting data to monitor the performance of the programs it supports, for which there have been benefits. As one interviewee stated, ‘There are very few programs like PEPFAR that can give you results on how many people are on treatment, PMTCT, etc.’ (240-33-USG). Also, the PEPFAR approach has led to an increased emphasis on measurement. In the words of one interviewee:

[The partner country] did not see M&E as an important issue in the system. Usually people would do work, give assistance and then go away; documentation was very poor. Now there are systems in place and people appreciate that the systems have to stay—PEPFAR helped with this, though work is still needed.’ (461-20-PCPS)

Although PEPFAR’s emphasis on collecting data for monitoring and evaluation is commendable, its value is limited if the process is so cumbersome that it results in limited ability to utilize the data. OGAC HQ, mission teams, implementing partners, and partner country governments have different constraints that limit use of data. At the OGAC HQ level, only a subset of program monitoring data are cleaned and available in a usable manner, and changes in database systems have limited the ability to access and use data. At the implementing partner and mission team level, the ability to use data is limited because of reporting burden and the perception that these data are of little utility. Partner country governments experience some of the same challenges as implementing partners, and they also, in many cases, have limited capacity for data analysis and use.

As PEPFAR moves toward greater alignment with partner country M&E systems, there will be less ability to attribute results to PEPFAR. This is consistent with an appropriate shift to a focus on measuring contribution to a country-led response. This shift could result in a reduced number of PEPFAR-specific indicators, which could help alleviate the burden of reporting across PEPFAR and contribute to more effective use of data.

PEPFAR SUPPORT FOR EPIDEMIOLOGICAL DATA

PEPFAR has been instrumental in supporting partner country surveillance efforts. At the onset of PEPFAR, there were limited epidemiological data available for understanding the drivers of the HIV epidemic in partner countries and for informing decisions in implementing a response. Consequently, PEPFAR invested heavily in increasing surveillance of HIV/AIDS in partner countries to monitor the epidemic, supporting local surveys and baseline studies, and developing methods to model the scope of the epidemic (GAO, 2012; Lyerla et al., 2012).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

PEPFAR has supported surveillance systems within partner countries including nationally representative household surveys such as the Demographic and Health Survey, behavioral surveys such as the Integrated Biological and Behavioral Surveillance (IBBS), drug resistance surveys, antenatal care surveys, and HIV case and incidence estimation, and has also provided financial and political support for surveys on populations at elevated risk of HIV infection and transmission (116-1-USG; 116-4-USG; 166-4-USG; 196-1-USG; 196-8-ML; 196-10-PCGOV; 196-11-USNGO; 196-13-OGOV; 240-9-USG; 240-12-USG; 272-13-USG; 331-3-USG; 331-10-PCGOV; 331-14-USG; 331-15-USG; 331-24-PCGOV; 396-6-PCGOV; 587-9-USG; 636-1-USG; 934-21-USG; 934-24-PCGOV; 461-1-USG). As one mission team interviewee described PEPFAR-supported surveillance efforts, ‘survey data is very strong and has been useful in giving evidence of what is happening in the epidemic(272-27-USG). As one partner country government interviewee stated:

I think the PEPFAR program enhanced our capacity on surveillance on HIV testing in country to enhance prevention care and support for HIV. [. . .] And I highly appreciate CDC support not only for HIV/AIDS but for many other activities [. . .] we receive [surveillance] support from CDC, both technical support and financial support on many activities. [. . .] we have support from CDC to do IBBS. I think that maybe the best information we have about HIV/AIDS [situation] in country. [. . .] Besides surveillance we receive support for estimation prediction, also very important. [. . .] And we have I think a good picture about HIV/AIDS in the country.” (396-6-PCGOV)

In addition to the use of program monitoring data, as described earlier in this chapter, PEPFAR-supported surveillance and survey data have been used by partner country governments and other country stakeholders to better understand drivers and to monitor trends for country epidemics, to contribute to and influence planning for the national response, and to influence national policies, which, in some cases, has resulted in increased attention to previously underserved populations or service needs (396-6-PCGOV; 396-1920-USG; 396-53-USNGO; 272-22-USG; 272-25-USG; 166-23-USG; 196-11-USNGO; 331-10-PCGOV; 331-ES; 331-24-PCGOV; 196-ES). For example, a national HIV/AIDS behavioral risk survey funded by the USG and other donors with the MOH changed some of the priority areas for both PEPFAR and the MOH in terms of regions targeted and target populations: ‘Now, instead of being more anecdotal, have more evidence and now more able to target programming(166-23-USG). In another partner country, a PEPFAR partner worked with provincial governments to provide training on how to collect and analyze data and then on how to use the data for planning and evaluating programs (396-53-USNGO).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

A range of stakeholders also described how PEPFAR mission team staff used epidemiological data to focus activities and provide services based on evidence (331-43-USG; 331-22-PCNGO; 196-28-USG; 116-1-USG; 116-12-PCNGO; 166-7-PCGOV; 396-1920-USG; 240-9-USG; 272-25-USG). For example, mission teams described how information was gathered on who was infected and what behaviors were driving the epidemic and how interventions were developed based on those behaviors, including focusing on best practices in populations at elevated risk (331-14-USG; 396-12-USG). One mission team interviewee described using epidemiological data to focus PEPFAR-supported activities:

Which districts PEPFAR supports is primarily driven by epidemiology—places with high prevalence with groups that needed to be reached. There is also ongoing dialogue with the government. But, burden is the driver of where PEPFAR works.’ (196-28-USG)

Conclusion: PEPFAR has provided financial and technical support for collecting epidemiological information in partner countries. This was widely seen as a positive contribution and has informed decisions and priorities in planning the HIV/AIDS response and implementing HIV programs, encouraged and facilitated responsiveness to the epidemic, and contributed to identifying the needs in partner countries.

PEPFAR SUPPORT FOR DATA USE BY PARTNER COUNTRY STAKEHOLDERS

In FY 2008, OGAC SI endorsed the overarching goal of “Know your epidemic, know your results,” in keeping with the UNAIDS approach introduced in 2007 of “Know your epidemic, know your response” (OGAC, 2008c, p. 191; UNAIDS, 2007, p. 10). Part of this initiative was aimed at helping partner countries and members of civil society “collect, analyze, critically review, disseminate, interpret, display, and strategically use data at all levels” (OGAC, 2008c, p. 191). PEPFAR’s effort to support the use of routine monitoring data and epidemiological data by partner country stakeholders to inform the HIV response is described only very briefly here. Further discussion of PEPFAR’s efforts on building capacity in partner countries for the collection and use of health data information can be found in Chapter 9 on health system strengthening. In addition, PEPFAR’s support for data collection to support the planning of program portfolios and joint planning with partner countries is discussed where relevant in the program area chapters in Part III of this report.

The PEPFAR approach to implementing programs was described by some interviewees as leading to an increased emphasis on measurement

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

that has had a positive effect (461-14-USG; 272-15-PCNGO; NCV-5-USACA; 240-8-USG; 636-18-ONGO; 396-55-USG; 331-14-USG; 116-23-USPS; 166-23-USG; 272-22-USG; 461-18-USG). One USG interviewee described PEPFAR’s contribution to creating an evidence-based culture as the legacy of PEPFAR’s work in the partner country (240-8-USG). In addition to supporting the collection of epidemiological data, which is discussed in detail in the next section of this chapter, PEPFAR has carried out several activities to increase data use among partner country governments. Efforts have included mentorship (240-12-USG; 636-18-ONGO), providing training to district government employees on data use (636-18-ONGO; 196-1-USG), strengthening data and data use in the MOH (636-18-ONGO; 166-4-USG; 166-9-ML/OBL/USACA/USNGO/PCNGO/PCPS; 272-17-USG; 396-1920-USG), working on creating an evidence-based culture (240-8-USG; 166-4-USG; 461-16-USG), and supporting data use workshops (461-16-USG; 396-5-USNGO). PEPFAR has successfully stressed the importance of local data collection and use for decision making by local governments, including the establishment of district M&E teams (636-18-ONGO; 396-55-USG). PEPFAR has also worked with partner country governments to use data to change policies and better target programming (331-14-USG; 116-23-USPS; 166-23-USG; 272-22-USG; 461-18-USG).

Track 1.0 partners have been active in promoting partner country data use (166-15-USACA). One Track 1.0 partner described providing automated data reports as a mechanism for feedback both at the site and country level, eliminating the burden of report making because in many countries there is a limited statistical analysis capacity (NCV-5-USACA). This Track 1.0 partner also encouraged implementing facilities to present and discuss their data with each other at meetings, contributing to ownership of the data collected (NCV-5-USACA). Additionally, in at least one partner country, a Track 1.0 partner provided funds to help the government develop a data warehouse for patient-level data (NCV-5-USACA). By participating on national TWGs, Track 1.0 partner staff presented and discussed data with government counterparts, encouraging data use by the partner country (NCV-5-USACA).

Despite these efforts, increasing partner country data use has been hampered by an ongoing lack of capacity within partner country governments and partner country organizations (116-23-USPS; 166-4-USG; 166-7-PCGOV; 166-9-OBL/ML/USACA/USNGO/PCNGO/PCPS; 272-15-PCNGO; 461-20-PCPS). High turnover of staff within the government and partner organizations has also posed a challenge (116-23-USPS; 396-56-USNGO). Successfully increasing use of data by partner countries was seen by some interviewees as being linked to the long-term sustainability of PEPFAR’s efforts (331-1-USG; 396-39-USG).

Conclusion: PEPFAR has invested in building the capacity of partner countries to plan for, collect, manage, and use HIV data, which has implications for the larger health system. Through these investments, PEPFAR has contributed to fostering a culture of evidence

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

among country partners, including country-based implementing partners and partner country governments.

PEPFAR-SUPPORTED EVALUATION AND RESEARCH ACTIVITIES

History and Evolution of PEPFAR-Supported Evaluation and Research

From its onset PEPFAR has included an emphasis on evidence-based programming, highlighting the need for evaluation and research in addition to program monitoring data to serve as evidence to inform efforts (OGAC, 2004). As a result, PEPFAR has actively supported some form of evaluation since its inception; however, program priorities and policy constraints on engaging in research limited the initial role of research within PEPFAR (OGAC, 2004, 2011b). As PEPFAR programs and priorities evolved from an emergency response toward a more sustainable response to the HIV epidemic, PEPFAR leadership increasingly recognized the importance of evaluation and research in capturing, utilizing, and maximizing knowledge created through PEPFAR as well as in ensuring contributions to the global knowledge base on effective HIV/AIDS interventions and program implementation5 (OGAC, 2009f; Padian et al., 2011). Subsequently, the role of evaluation and research within PEPFAR has expanded.

Both research and evaluation have important roles to play within PEPFAR and can contribute to implementing effective evidence-informed programs. Although research and evaluation use similar tools and methodologies and may draw from similar data sources, they have notably different aims, uses, and audiences (Fain, 2005; Levin-Rozalis, 2003; Small, 2012). The aims of research include adding new knowledge to a field, proving that a particular factor caused a particular effect, and producing results that are generalizable beyond an individual project or program (Fain, 2005; Levin-Rozalis, 2003; Small, 2012). In contrast to this, the purpose of evaluation is “not to prove, but to improve” (Stufflebeam, 2007, p. 2). Evaluation is specific to a particular project or program; it aims to produce outcomes used by decision makers to determine the best mechanisms to achieve program goals, assess program effectiveness, and assess whether goals are being met or not (Fain, 2005; GAO, 2011b; Levin-Rozalis, 2003). As the role of research in PEPFAR evolved, which is described in the following sections, defining appropriate and allowable research activities within PEPFAR was and remains a challenge, and there remain no clear distinctions between these separate but complementary aims of research and evaluation.

__________________

5 Tom Lantos and Henry J. Hyde United States Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008, P.L. 110-293, 110th Cong., 2nd sess. (July 30, 2008).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

The Role of Research from PEPFAR I to PEPFAR II

The first phase of PEPFAR (PEPFAR I) was initiated as an emergency response to HIV/AIDS that was focused on the rapid implementation and scale-up of prevention, treatment, and care programs (OGAC, 2009f); as such, basic infrastructure for monitoring and evaluation existed (USAID, 2011b), but “state-of the art monitoring, evaluation and research methodologies were not fully integrated or systematically performed” (Padian et al., 2011, p. 1). In PEPFAR I, research was seen as having two roles: to produce new knowledge about HIV/AIDS interventions and implementation, and to assess PEPFAR programs and inform policies through targeted research (OGAC, 2004). As the primary focus of PEPFAR I was the rapid scale-up and implementation of programs, leadership felt that PEPFAR efforts would be better spent on implementation, while other USG organizations better suited to conduct research focused on creating new knowledge (OGAC, 2005b).

At the time, the USG supported a wide variety of HIV/AIDS research through NIH, CDC, and USAID from which PEPFAR could draw new knowledge ranging from basic clinical and social science research to applied and operations research; studies focused on multiple topics, including therapeutic and preventative regimens, microbicides, vaccines, ART, prevention of mother-to-child transmission (PMTCT), ABC, male circumcision, injection safety, nutrition, and psychosocial issues for OVC (IOM, 2007a). The intention was for OGAC to work closely with the leadership at NIH, HHS, and USAID to ensure that their research priorities aligned with PEPFAR’s goals and needs in order to leverage these external research efforts to inform PEPFAR policy and program decisions (OGAC, 2004, 2005b).

Beyond this collaboration, PEPFAR did, in some special cases, fund targeted evaluations and research to address PEPFAR-specific questions (OGAC, 2005b). For many PEPFAR stakeholders, however, it was unclear what research, if any, was allowed with PEPFAR I funding. Descriptions of research in the PEPFAR I legislation and strategy seemed to proscribe against using PEPFAR funds for research, and many country mission teams and implementing partners perceived a ban on using PEPFAR funds for research (IOM, 2007a). This perceived research proscription was frequently mentioned during interviews with HQ and implementing partners involved in PEFPAR from the inception. In the words of one interviewee, “[Y]ou couldn’t use the word research or operational research(NCV-4-USACA). Another interviewee described how people were “baffled as to why there was no research components in the first years of this program, and why it was absolutely disallowed because we did all this work and we’re not able to really learn from it or do anything(NCV-8-USACA). Finally, one interviewee described how research and evaluation became conflated by OGAC to get

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

around research restrictions. In the words of the interviewee ‘In the first phase of PEPFAR, OGAC could not use the word research, so people referred [to research activities] as evaluation(NCV-7-USG).

Over time, leadership recognized that although PEPFAR was not intended to be a research organization, research was important to optimizing programs and maximizing the impact of knowledge and experiences created through PEPFAR (IOM and NRC, 2010). Recognizing this, the PEPFAR II reauthorization legislation and the second Five-Year Strategy helped clarify the research and evaluation policy to encourage these activities within PEPFAR (OGAC, 2009f). The reauthorization legislation and the second Five-Year Strategy called for the integration and expansion of research (e.g., biomedical research, health services research, impact evaluation research, and operations research) within PEPFAR in order to assess program quality, effectiveness, and population-based impact; to optimize service delivery; and to contribute to the global evidence base on HIV/AIDS interventions and program implementation (OGAC, 2009f).6

Evolution of PEPFAR-Supported Evaluation and Research Activities in PEPFAR I and II

As PEPFAR priorities and programming progressed, the frame within which PEPFAR conceptualized evaluation and research activities expanded from the initial Targeted Evaluations (TEs) to Public Health Evaluations (PHEs) Phases I and II to the current Implementation Science (IS) and Impact Evaluations. As the frame has evolved, the scope, allowable methods, funding mechanisms, oversight entities, and priorities of these research and evaluation activities have changed; this evolution is summaried in Table 11-5. Throughout this evolution, research and evaluation remained comingled in the operational structures of TEs and PHEs, with no clear articulation of the distinctions between PEPFAR’s research activities and aims and evaluation activities and aims, which is discussed below. The following sections focus on activities that have been implemented during PEPFAR I and into PEPFAR II, while a subsequent section will discuss in more detail the new research and evaluations activities being implemented under the IS umbrella, which were only just beginning as this evaluation was under way.

Targeted evaluations Targeted evaluations began in 2005 to provide an evidence base, beyond routine program monitoring and evaluation or surveillance, to inform program planning and implementation (OGAC, 2005b, 2006b). The goals of targeted evaluations were to assess program outcomes, indicate whether programs achieved their goals, and identify

__________________

6Supra, note 5.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

potential best practices for scale-up (OGAC, 2005b, 2006a,b). In the words of OGAC staff, TEs were “studies that provide[d] rapid answers to specific, measurable, and focused questions about health program implementation to improve services and identify best practices” (Bouey and Padian, 2011, p. 4). The allowable scope and study methods of TEs were influenced by “legislative sensitivities on use of PEPFAR funds for research” (Bouey and Padian, 2011, p. 4). Randomization was not allowed and study methods mainly included quasi-experimental designs using natural controls or pre- and post-test results with a comparison or control group (Bouey and Padian, 2011; OGAC, 2005b).

Study priorities for TEs were mainly country driven, with most proposals submitted through COPs; these proposals were reviewed and selected by a TE sub-committee, which included representatives from USG agencies involved in PEPFAR. Additionally, this subcommittee, in coordination with a Scientific Steering Committee and implementing agencies, developed the priorities for centrally funded studies, developed proposal selection criteria, and oversaw selected studies (OGAC, 2005b, 2006a). There was little control from HQ level (OGAC) over TEs (Bouey and Padian, 2011; OGAC, 2011e). Studies were funded either through central funds or country-level budgets (OGAC, 2005b).

TEs that were funded in 2005 and 2006 aimed to address questions concerning the efficacy of programs in the areas of prevention, care, treatment, and service delivery for HIV/AIDS (OGAC, 2006b). Specifically, these studies assessed the following areas: abstinence/be faithful, condoms and other prevention, PMTCT, treatment (antiretroviral [ARV] drugs and services), palliative care (for basic health care and support and for TB/HIV), OVC, counseling and testing, and strategic information. The countries that received funding for targeted evaluations in 2005 and 2006 were Guyana, Haiti, Kenya, Mozambique, Nigeria, Rwanda, South Africa, Tanzania, Uganda, Vietnam, and Zambia. A few multi-country studies were also funded (IOM, 2011).

In 2007 the scope of the targeted evaluations expanded from studies focused on questions about individual program implementation to include studies designed to answer questions concerning efficacy and best practices with the goal of producing generalizable results that could contribute to program sustainability. Randomized trials were still not allowed (Bouey and Padian, 2011). In this round, TEs were no longer centrally funded, evaluation priorities were driven by TWGs at the country level, and proposals were submitted and funded solely through the COP and country budgets (OGAC, 2006c, 2011e). According to one interviewee, TWGs and implementing partners developed the research agendas based on gaps that they saw in the field (NCV-3-USG). Proposals were still reviewed and selected by the targeted evaluation subcommittee, which also continued to oversee

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

TABLE 11-5 Evolution of PEPFAR-Supported Evaluation and Research Activities

Program (Fiscal Year) Scope Methods Funding Mechanism Oversight Program Priorities
Targeted Evaluations (2005–2006) Rapid answers to specific, measurable, and focused questions about program implementation to improve services and identify best practices Rigorous assessments including pre- and post-test results with a comparison or control group Mission/country and centrally funded
Funded through COP
Scientific Steering Committee and Targeted Evaluation Subcommittee
Minimal central oversight
Country driven
Targeted Evaluations (2007) Answer specific questions around efficacy and best practices to produce generalizable results and contribute to program sustainability Rigorous assessments including pre- and post-test results with a comparison or control group Funded through COP Scientific Steering Committee and Targeted Evaluation Subcommittee
Minimal central oversight
Technical working group driven
Public Health Evaluations (PHEs) Phase 1 (2008) Answer questions around program effectiveness, compare program models, answer operational questions, and determine program outcome and impact with shift in focus from individuals to communities and populations Rigorous, scientifically sound research methodology using experimental or quasi-experimental designs, including (but not limited to) randomization, modeling, advanced statistical techniques, and comparison groups Funded centrally Scientific Steering Committee and PHE Subcommittee
Subcommittee has more responsibility and is able to convene multi-agency PHE evaluation teams representing CDC, DoD, NIH, HRSA, USAID Increased central support and coordination oversight
Globally significant priorities generated at country, central, and technical working group level
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Public Health Evaluations Phase 2 (2009–2010) Answer questions of global significance, program impact and effectiveness, comparative evaluations of interventions and programs, and operational questions Rigorous, scientifically sound research methodology using experimental or quasi-experimental designs, including (but not limited to) randomization, modeling, advanced statistical techniques, and comparison groups Funded centrally Scientific Steering Committee, PHE Subcommittee, and PHE Evaluation Teams
Increased central support and coordination
Country driven
Align with globally significant priorities
Implementation Science (2011–present) Focused on improving uptake, implementation, and translation of research findings into practice Scientifically rigorous research methods using randomized experimental designs, quasi-experimental methods, or advanced mathematical techniques (e.g., simulation, mathematical optimization, and decision science) Funded centrallya
Concept proposals required through implementing agencies
Implementation Science Steering Committee (formerly known as PHE Subcommittee)
Central oversight
Implementing agency managed
Scientific Advisory Boardb driven
Align with country research needs and priorities
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Program (Fiscal Year) Scope Methods Funding Mechanism Oversight Program Priorities
Impact Evaluations (2012–present) Systematic study of the change attributable to a particular intervention, such as a project, program, or policy Uses experimental or quasi-experimental approaches to establish a counterfactual Funded though COP Implementation Science Steering Committee
Central oversight at concept stage
Country managed at implementation stage
Country driven
Linked to specific programmatic activities

NOTE: CDC = U.S. Centers for Disease Control and Prevention; COP= country operational plan; DoD= U.S. Department of Defense; HRSA = Health Resources and Services Administration; NIH= National Institutes of Health; USAID= U.S. Agency for International Development.

aImplementation Science Requests for Applications (RFAs) are funded centrally and managed through implementing agencies. These mechanisms include the NIH Implementation Science Supplement, NIH Implementation Science RFA, NIH Implementation Science Injection Drug Use (IDU) RFA, NIH Implementation Science PMTCT RFA, CDC Implementation Science Funding Opportunity Announcement, and USAID Implementation Science Annual Program Statement (APS).

bSAB is composed of members that represent academia, advocates, international experts, the HIV/AIDS community, partner country governments, multilateral and bilateral agencies, foundations, and nongovernmental organizations.

SOURCES: Bouey and Padian, 2011; OGAC, 2010b, 2011e, 2012e; Padian et al., 2011.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

selected evaluations, with continued minimal oversight from central HQ (Bouey and Padian, 2011; OGAC, 2011e). One HQ interviewee noted the importance of the introduction of TEs within PEPFAR as important because it provided a mechanism to look at program effectiveness, which the program monitoring indicators did not address (NCV-3-USG). Although TEs were an important step in establishing evaluation and research in PEPFAR, the interviewee also said that they faced certain challenges, including varying quality across evaluations and internal bickering over evaluation priorities and control of funding (NCV-3-USG).

Public Health Evaluations—Phase I and II In 2006, the concept of Public Health Evaluations was introduced as a new approach to evaluation and research within PEPFAR (NCV-7-USG) (OGAC, 2006a). One OGAC interviewee described the introduction as being, in part, a response to the varying quality of TEs (NCV-3-USG). Another HQ interviewee stated that the PHEs would allow OGAC to have more control over what was happening and the quality of the work (NCV-7-USG). The OGAC document Blueprint for Public Health Evaluations in PEPFAR described PHEs as a broader concept than TEs, with an expanded range of allowable methodologies and a new management structure (OGAC, 2006a). Whereas TEs were intended to answer questions about program implementation and efficacy in order to identify models and best practices for potential scale-up, PHEs broadened this scope, recognizing a need for increased studies and methodologies to answer critical questions over time and for allowing investigators to assess the impact of programs on populations (OGAC, 2006a). The study design methodology was expanded to allow rigorous, scientifically sound research methodology using experimental or quasi-experimental designs including, but not limited to, randomization, modeling, advanced statistical techniques, and comparison groups (Bouey and Padian, 2011; OGAC, 2007e, 2011e).

With the introduction of the first phase of Public Health Evaluations (called PHEs I going forward) in 2008, the focus of PEPFAR evaluations shifted from individuals to populations, and the goals evolved to the implementation of studies to guide PEPFAR, inform policy, assess impact, and contribute knowledge to the global HIV/AIDS community (OGAC, 2010b). Some PHE I goals remained similar to those of the TEs, specifically, answering questions related to program effectiveness and quality and also identifying models and best practices, while some goals were new, including determining program outcome and impact (Bouey and Padian, 2011; OGAC, 2007e). PHEs I also focused on encouraging local partner involvement to build capacity. Funding for PHEs was initially provided through both central and country budgets, but eventually PHEs were no longer funded through country budgets and were only funded centrally,

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

with awards based on merit in a competitive funding process (NCV-7-USG) (OGAC, 2007e, 2008b).

In addition to having an expanded scope and additional methodologies compared with TEs, a new oversight mechanism was introduced for PHEs. The TE subcommittee evolved into the PHE subcommittee, an interagency technical policy group with representatives from the U.S. Department of Health and Human Services (HHS), USAID, the Census Bureau, DoD, and the Peace Corps. The PHE subcommittee had increased responsibility and an ability to convene multi-agency PHE evaluation teams (OGAC, 2006a). Study priorities were generated and driven by the central level, country mission teams, TWGs, and other members of the PEPFAR community (OGAC, 2006a). Once per year, proposals were sent to the PHE subcommittee, which would review concepts and recommend funding levels to the Scientific Steering Committee (OGAC, 2008b). The Scientific Steering Committee would then make recommendations to the Global AIDS Coordinator, who made final funding decisions. Topics prioritized for PHEs were prevention, treatment, care, and cross-cutting issues such as gender and also OVC (IOM, 2007a). Multi-country evaluations were also eligible for funding (OGAC, 2007e).

In 2009 a second phase of Public Health Evaluations (called PHEs II going forward) was introduced, which further expanded the scope of PEPFAR evaluations. The goals of PHEs II were to answer questions of global significance, to assess program impact and effectiveness, to perform comparative evaluations of interventions and programs, and to encourage in-depth studies beyond routine program evaluation (Bouey and Padian, 2011; OCAG, 2008b). The main focus was on bridging research and practice with a call for concepts that examined the real-world effectiveness of interventions with proven efficacy, cost-effectiveness of delivering these programs at scale, as well as optimizing efficiency (OGAC, 2010b). Beyond questions of global significance, there was still an emphasis on allocating funding to country-specific questions in order to respond to host country or local implementation needs and to provide partner country capacity building opportunities (OGAC, 2008b, 2009d). The study methodologies did not change from PHEs I to PHEs II (Bouey and Padian, 2011; OGAC, 2010b, 2011e), and funding continued to come centrally with PHE awards granted based on a competitive proposal, review, and selection process (OGAC, 2008b). PHE concepts and priorities were country driven through proposals submitted annually by PEPFAR USG country teams, which were reviewed and selected by the Scientific Steering Committee, PHE Subcommittee, and PHE evaluation teams. This submission, review, and approval process occurred separately from the COP submission and review process (Bouey and Padian, 2011; OGAC, 2009d, 2011e). The FY 2010 PHE guidance and call for concepts emphasized a priority for PHE proposals with the follow-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

ing elements: country-driven concepts answering questions of importance to the partner country, participation of local partner country institutions and investigators, and partner country research capacity building elements (OGAC, 2010b).

In 2012, the Government Accountability Office (GAO) completed a mixed-methods study that examined PEPFAR’s evaluation activities and included a review of PEPFAR PHEs. As part of this study, the GAO requested and received from OGAC a list of 18 PHEs that had been completed as of November 2011. Additionally, OGAC indicated that there were 82 other PHEs initiated and ongoing as of November 2011 (GAO, 2012). According to the GAO, the 18 completed PHE studies covered the content areas of PMTCT, counseling and testing, care and support (adult and pediatric), adult treatment, and prevention of sexual transmission (GAO, 2012). In December 2012 the Institute of Medicine (IOM) committee made a similar request to OGAC for a comprehensive list of all PHEs awarded from 2008 to 2010 (Phase I and II). As described further in the following sections, OGAC was unable to provide the committee with a list of all PHEs awarded and completed, but it did provide a list of PHEs that were currently ongoing as of December 2011. These 83 continuing PHE studies addressed the content areas of PMTCT, prevention, care and treatment (including resistance monitoring), counseling and testing, OVC, service delivery, and health system strengthening. Of these continuing PHEs, 6 were multi-country studies (7 percent), and the remaining 77 (93 percent) were single-country studies. According to the information provided by OGAC, PHEs were being conducted in 17 countries. Figure 11-3 displays the number of ongoing PEPFAR PHE studies in each of these countries (OGAC, 2011b,d).

Additionally, the information OGAC provided listed the implementing partners for 73 of the 83 PHEs. The majority of these PHEs (n=65) were being implemented solely by one partner organization; eight PHEs, however, were being implemented in partnership between two or more implementing partner organizations. This joint implementation usually involved a U.S.-based organization partnering with one or more organizations based in PEPFAR partner countries. Overall, there were 41 unique organizations involved in implementing the 73 continuing PHEs. These organizations were based in the United States, PEPFAR partner countries, and non-PEPFAR countries (see Figure 11-4) and represented academia, government, NGOs, the private sector, and research organizations (OGAC, 2011d).

Although it offers only a snapshot of ongoing PHEs at a particular time, the list provided by OGAC shows that PEPFAR-implemented PHEs covered an array of content areas in a multitude of countries and that these PHEs were implemented by organizations based both in the United States

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

img

FIGURE 11-3 Ongoing PEPFAR Public Health Evaluation (PHE) studies, by country, December 2011.

NOTES: This figure represents the breakdown of research activities as of December 15, 2011. Figure compiled from a list of continuing PHEs for FY 2012 received from OGAC.

a Multi-country study countries: Côte d’Ivoire, Kenya, Mozambique, Rwanda, South Africa, Tanzania, Thailand, Uganda, and Vietnam.

SOURCES: OGAC, 2011b,d.

img

FIGURE 11-4 Organizations implementing ongoing PEPFAR Public Health Evaluation (PHE) studies, by implementing organizations’ country, December 2011.

NOTES: Figure compiled from a list of continuing PHEs for FY 2012 received from OGAC. Figure represents the percentage by country of origin for the 41 unique organizations involved in implementing the 73 continuing PHEs.

SOURCE: OGAC, 2011d.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

and partner countries, with a handful implemented in partnership between U.S.-based and partner country organizations.

Results of TEs and PHEs OGAC was not able to provide the committee with an up-to-date list of completed TEs and PHEs, and it is difficult to identify independently which TEs and PHEs have resulted in published reports or journal publications. In a response to a separate request for PEPFAR-supported publications described later in this chapter, USAID and CDC did provide lists specifying nearly 400 journal publications that resulted from PEPFAR-supported TEs, PHEs, impact evaluations, or operations research. These publications ranged in their purposes, including assessing feasibility, effects, effectiveness, cost-effectiveness, and impact of interventions. The publications also covered a wide range of technical areas, including care and support (i.e., adult and pediatric), counseling and testing, prevention (i.e., PMTCT, biomedical prevention, prevention of sexual transmission, harm reduction, prevention with positives, and prevention in populations at elevated risk), health systems strengthening (i.e., commodities and technologies, financing, integration, workforce, and service delivery), OVC, gender, treatment (i.e., adult and pediatric, resistance, adherence, and retention), vulnerable populations, stigma, and strategic information. A few interviewees noted that analysis and findings from PHE studies were disseminated, published (NCV-5-USACA; 166-20-USG), and used for programmatic changes (NCV-6-USNGO); however, interviewees also noted that, because of the lengthy PHE application, review, and procurement processes, study findings often became obsolete by the time the PHE was completed (461-3-USG; 461-14-USG; NCV-31-USG).

Although this provides some sense of the scope of these efforts, this information is not comprehensive for all TEs and PHEs. Without a more complete understanding of the outputs of the TE and PHE efforts supported by PEPFAR, it was not possible for the committee to draw any conclusions about the extent to which findings from these evaluations have been used and have affected PEPFAR-supported programs.

Other PEPFAR-supported evaluation and research activities In addition to using TEs and PHEs to provide formal support for evaluation and research across a range of technical areas, PEPFAR has also provided support for a wide array of additional evaluation activities employed at the macro level, the HQ level, the implementing agency level, and the country level. At the macro level the GAO, Office of the Inspector General (OIG), and Congressional Budget Office perform periodic audits and evaluations of portions of the PEPFAR program. Additionally, the IOM was tasked with performing an independent evaluation of PEPFAR I and, later, the evaluation presented

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

in this report (Simonds et al., 2012).7 At the HQ level, in 2009 the interagency TWGs were tasked with developing and submitting State of the Program Area (SOPA) documents to OGAC. The SOPAs, completed by TWGs across program areas, were used to review the current status of the program area as well as to identify promising practices, lessons learned, challenges, emerging issues, and future directions of the TWGs (IOM and NRC, 2010; Reyes, 2009). At the implementing agency level, CDC and USAID manage and implement periodic evaluations of programs and interventions covering a wide variety of technical areas, including prevention, treatment, OVC, strategic information, and health system strengthening (GAO, 2012). Finally, at the country level PEPFAR mission teams and implementing partners not only participate in the routine data collection and reporting described previously in this chapter, but also carry out basic program evaluations. Basic program evaluations, similarly to PHEs, are used to guide PEPFAR programming and policies but they are focused specifically on local program implementation and on the direct effects of programs on the population that is receiving program resources (OGAC, 2010b, 2011g). These studies were described as methodologically simpler than PHEs, with results that are not generalizable beyond the individuals enrolled in the program (OGAC, 2010b). Unlike the case with PHEs, funding for basic program evaluations comes through country budgets in the COPs, and the studies are managed through country and regional teams (OGAC, 2010b, 2011g).

According to the GAO study, OGAC generally defers to implementing agency evaluation policies as guidance and also defers to implementing agencies or country and regional teams to plan evaluations of HQ-managed and country-level activities, respectively. The U.S. Department of State, CDC, and USAID all have established evaluation policies or frameworks that are applicable to PEPFAR programs (GAO, 2012). The CDC issued an evaluation framework for all CDC programs in 1999, which “summarizes essential elements of program evaluation, clarifies program evaluation steps, and reviews standards for effective program evaluation” (GAO, 2012, p. 18). In 2011 USAID published an evaluation policy to replace previously issued guidance and to help improve the number and quality of USAID program evaluations. The policy provides a definition of evaluation and gives clear guidance on the purpose of evaluation, what should be evaluated, approaches for conducting evaluation, and the dissemination and use of results (GAO, 2012; Office of Learning, Evaluation, and Research, 2012). Finally, the Department of State issued an evaluation policy in 2012, applicable to OGAC and other state department bureaus, which provides a framework for implementing program and project evaluations

__________________

7Supra, note 5.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

(GAO, 2012). Although the recent 2012 country and regional operational plan guidance provided some direction on evaluation by defining various types of evaluation and research (GAO, 2012), OGAC has not issued its own overarching evaluation framework or evaluation plan to guide evaluation activities occurring at the HQ and country levels.

Beyond the evaluation activities described above, PEPFAR also supports other activities in PEPFAR partner countries linked to generation of evidence, including support for surveillance and surveys, strengthening monitoring systems at the country level, and building partner country research capacity. As described previously in this chapter, PEPFAR has supported HIV surveillance in partner countries, local surveys and baseline studies, and modeling to assess the scope of the epidemic. As discussed in Chapter 9, PEPFAR supports the development and strengthening of monitoring systems including national health management information systems, HIV M&E systems, and facility- and community-based monitoring systems (GAO, 2012). Finally, PEPFAR provides support to build partner country research capacity; several interviewees described how PEPFAR mission teams and implementing partners provide in-country research support to local institutions and government agencies conducting research (934-44-PCACA; 166-1-USG; 396-55-USG; 166-5-USG).

PEPFAR Evaluation and Research in PEPFAR I and II: Successes and Challenges

As described in the previous section, over time PEPFAR has supported an immense number of research and evaluation activities through a variety of mechanisms across a wide range of technical areas. Results and outcomes of PEPFAR-supported research and evaluation activities (e.g., research results, evaluation outcomes, PEPFAR program data, surveys, and publications) have been used to inform and improve PEPFAR programs and strategic planning (196-28-USG; 116-1-USG; 116-12-PCNGO; 166-7-PCGOV; 240-9-USG; 272-25-USG; 331-22-PCNGO; 331-14-USG; 331-43-USG; 396-12-USG; 396-1920-USG), to influence country-level policies and national planning (116-23-USPS; 272-22-USG; 272-25-USG; 272-27-USG; 396-1920-USG; 396-53-USNGO), and to contribute evidence to the knowledge base on improving HIV/AIDS interventions and program implementation (272-24-USG; 272-25-USG; 272-36-USG; 461-4-USG; 461-8-PCGOV; NCV-10-USG). Interviewees noted, for example, how PEPFAR support for surveillance, surveys, evaluation, and operational research has provided data and results that have been used by partner country governments to influence national policies on male circumcision, microbicides, effective PMTCT, and dual therapy (272-22-USG; 272-25-USG; 272-27-USG). One interviewee described the studies that implementing partners conduct and subsequently the influence that results have on policies, guidelines, and standards as ‘tremendous’ and having a ‘major impact(272-22-USG).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Interviewees also described how knowledge created and disseminated through PEPFAR has contributed to the global knowledge base on effective HIV/AIDS interventions and implementation. Interviewees highlighted PEPFAR studies that produced evidence showing nurses could effectively deliver treatment (272-36-USG), influenced PMTCT regimens (272-24-USG; 461-8-PCGOV), provided an opportunity to gather evidence on prevention and treatment in discordant couples (461-4-USG), provided evidence on microbicides (NCV-10-USG), and produced evidence used to change implementing partners’ perceptions on gender (272-25-USG). As described later in this chapter, PEPFAR implementing agencies and partners have created a vast amount of knowledge that has been disseminated internal and external to PEPFAR. Publications, books, conference abstracts, technical guidelines, and training materials have been produced and disseminated as a result of PEPFAR support.

Despite these successes, HQ- and country-level interviewees identified several barriers and limitations related to PEPFAR-supported research and evaluation activities, including barriers to conducting research activities, research gaps, and challenges in monitoring and tracking PEPFAR-supported evaluation and research activities.

Interviewees identified PEPFAR restrictions as barriers to conducting research activities. Specifically, they identified legislative restrictions on the types of research that could be conducted in PEPFAR (NCV-4-USACA; 272-12-USNGO), as well as the cumbersome and lengthy research review and approval processes (NCV-2-USG; NCV-5-USACA; NCV-8-USACA; NCV-31-USG; 461-1-USG; 461-3-USG; 461-14-USG; 396-6-PCGOV; 116-12-PCNGO; 396-5-USNGO; 240-8-USG; 196-1-USG; 461-16-USG; 272-25-USG)G)), as barriers that discouraged researchers from engaging in PEPFAR-supported research. One implementing partner interviewee described research activities his/her organization considered implementing but ultimately did not because the activities might have been considered restricted research. These activities included collecting data on HIV drug resistance; performance-based financing (comparing the performance of costly sites to less costly sites); and systematically using data to improve clinical care and outcomes (NCV-4-USACA).

A multitude of interviewees across countries described the PEPFAR evaluation and research application processes as cumbersome, lengthy, complex, and difficult, which deterred many from participating (NCV-2-USG; NCV-5-USACA; NCV-8-USACA; NCV-31-USG; 461-1-USG; 461-3-USG; 461-14-USG; 396-6-PCGOV; 116-12-PCNGO; 396-5-USNGO; 240-8-USG; 196-1-USG; 461-16-USG; 272-25-USG)G)). Several interviewees described spending a great deal of time working on PEPFAR research proposals that never went anywhere or that became obsolete by the time they were approved (NCV-8-USACA; 461-3-USG; 461-14-USG). In the words of one interviewee, ‘There is so much pressure to do PHEs, but they are the biggest waste of time in PEPFAR. [I] “don’t ever want to be involved in a PHE again”(461-3-USG). Additionally, one HQ interviewee described the PHE central review process as “very long” and not conducive to applying outcomes to programs on

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

the ground that may change quickly. The interviewee went on to say, “if it takes three years to get a study off the ground, it really, you know, it’s really not optimal […], because it’s a bit too late in the day by the time you get your results(NCV-31-USG).

Although PEPFAR has supported some form of evaluation from the beginning of the program and the allowable research and evaluation activities have continued to expand, research gaps remain and were identified by interviewees across countries. These gaps included a need for more program evaluation at the country level (331-5-ML; 272-6-ML; 166-23-USG; 240-15-USG; 272-32-PCNGO); local evidence generation (240-15-USG); costing studies (NCV-2-USG); studies to identify effective, efficient, and affordable service delivery models (396-18-USG; 396-39-USG; 396-45-USNGO; 396-59-USG); evaluation of integration models of HIV services (240-24-USG); more data around populations at elevated risk and drivers of the epidemic (331-7-PCNGO; 331-14-USG; 166-5-USG; 636-9-USACA; 240-9-USG); and more technical area–specific research, such as research in the areas of prevention and treatment.

Beyond the barriers to conducting research and the research gaps, there is also a challenge in tracking and monitoring the PEPFAR-supported evaluation and research activities that are being carried out. OGAC does not have a centralized system that tracks PEPFAR-supported evaluation and research activities over time (OGAC, 2011b). In order to assess the evolution of PEPFAR-supported evaluation and research activities over time, the committee requested from OGAC a comprehensive list or series of lists that documented all PEPFAR-supported research activities from the inception of PEPFAR, including all approved TEs, PHEs (I and II), and IS studies (i.e., completed, closed/terminated, and ongoing). OGAC could not provide the committee with a comprehensive list over time, and instead staff provided several lists of currently ongoing PHEs from 2011 and newly awarded NIH IS grants from FY 2010. These lists included only ongoing studies and excluded studies that had been completed, closed, or terminated between 2005 and 2010. One HQ interviewee stated that ‘OGAC never successfully managed to track the work that was being accomplished [research and evaluation—TEs PHEs, and IS](NCV-7-USGOV). OGAC staff did inform the committee that they are currently constructing a tracking system for PEPFAR evaluations (NCV-31-USG) (OGAC, 2011b).

Overall, interviewees described the manner in which PEPFAR previously handled support for research activities as resulting in missed opportunities to evaluate PEPFAR, learn from and improve programs, and assess impact from the beginning, which interviewees linked to sustainability (NCV-4-USACA; NCV-8-USACA). As noted by one interviewee, there as ‘[a] need for operational research from the beginning’ of PEPFAR but it was not approved in PEPFAR I. In the interviewee’s view, from the beginning PEPFAR should have functioned under the approach of “aim, fire, re-aim, re-fire,” but in-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

stead PEPFAR operated under the approach of “ready, fire, aim,” ‘which is the wrong approach but was the approach pushed by PEPFAR leadership in the beginning of the program(NCV-4-USACA).

Commenting on PEPFAR’s research restrictions, another interviewee said, “[I]t’s a funny thing to run such a huge, large-scale program about a clinical condition and say, ‘No research, absolutely zero research.’ You can’t be learning anything about what you’re doing(NCV-8-USACA). This limited PEPFAR’s ability to perform implementation and operations research that could have further contributed to the evidence base on effective HIV/AIDS interventions and implementation (NCV-5-USACA; 396-55-USG; 240-24-USG; 240-8-USG; 587-12-USG; 587-6-CCM; 272-27-USG). It also contributed to a missed opportunity to collaborate with local institutions and build in-country research capacity, which interviewees also linked to sustainability (NCV-4-USACA; NCV-8-USACA; 196-1-USG; NCV-9-USG; 331-6-CCM). As one commented, “[Y]ou can have the money, but if you don’t have the capability in your country to do research, to answer your own problems locally, [. . .] you’re constantly going to be dependent on folks having to fly in and do all this [. . .] work for you(NCV-9-USG).

One interviewee captured the overall sense, expressed by many, that PEPFAR’s initial lack of support for research activities was a missed opportunity:

It was very disappointing to us that we weren’t able to [. . .] keep the research perspective as we developed programs. [. . . Y]ou could learn a lot from it if you set the systems up correctly and have a nice base for not only collaborators like ourselves, but also our partners in-country to be able to utilize electronic databases and do retrospective clinical analyses or something. [. . .] We were discouraged from PEPFAR from the outset [from doing research], they didn’t want us to have informed consents [. . .] and would do audits to make sure we weren’t doing research. [. . . N]ow PEPFAR is putting a lot of money into implementation science research but it’s a little late.” (NCV-8-USACA)

Conclusion: Despite recent efforts to strengthen research and evaluation activities, the manner in which PEPFAR initially approached research activities was a missed opportunity to establish, from its inception, robust mechanisms to evaluate its programs, assess impact, contribute to the global knowledge base, and develop in-country research capacity.

Summary

Although OGAC has officially supported some form of research and evaluation in PEPFAR since 2005 (through TEs and PHEs), completed stud-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

ies have been few in number, incongruent in the range of questions, and not integrated within a comprehensive framework that articulated the purpose and goals of evaluation and research within PEPFAR (Padian et al., 2011). Additionally, throughout TEs and PHEs research and evaluation remained comingled concepts. There was no overarching strategy that clearly articulated the distinctions between the research aims and the evaluation aims within PEPFAR and how research and evaluation activities would work together to address these aims.

The Way Forward: Implementation Science

To address the challenges described in the preceding sections, in 2010 OGAC introduced and began to adopt what it has termed an IS approach for PEPFAR-supported research and evaluation; this represented an expansion of the traditional country-driven PHEs to research-driven studies implemented by the larger research community (Holmes, 2012). The IS umbrella “runs the gamut from routine monitoring and evaluation through operational research and impact evaluations with more rigorous scientific designs to randomized controlled trials” (OGAC, 2011b, p. 1). The IS framework, which was introduced in a journal editorial, was described as a single framework for the collection and use of information across PEPFAR. The three main components of the IS framework are monitoring and evaluation, operational research, and impact evaluations. IS focuses on improving program delivery; answering questions on the efficiency, effectiveness, and impact of programs; identifying and adopting successful delivery models; answering questions that PEPFAR is uniquely poised to investigate; and making evidence-informed decisions for PEPFAR activities and programs (Holmes, 2012; Padian et al., 2011). Additional goals include aligning with partner country national research priorities and building research capacity among individuals and institutions at the country level (Bouey and Padian, 2011).

Similar to PHE II, IS studies are centrally funded, but IS introduced a new mechanism for the submission, review, and approval of concepts. Applications are submitted in response to a request for applications (RFAs) for research and evaluation issued by one of three USG agencies—NIH, CDC, and USAID; after the concepts are reviewed and selected, funding is awarded, and studies are managed separately through these implementing agencies (NCV-31-USG) (OGAC, 2011g). This new approach was intended to minimize OGAC’s role in awarding and managing the process and to help address the challenges of the lengthy approval and review process from the PHEs (NCV-7-USG; NCV-31-USG).

Specific eligibility criteria vary by implementing agency, but RFAs from each agency are open to U.S.- and non-U.S.-based NGOs, nonprofit or-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

ganizations, and for-profit organizations that are willing to forgo profit, including academic institutions, community-based organizations, foundations, faith-based organizations, and host country organizations. Unlike PHEs, USG agencies and PEPFAR mission teams are not eligible to apply for the RFAs, because USG agencies do not apply for funds from other USG agencies (HHS, 2012; NIH, 2011; OGAC, 2012c; USAID, 2011a), which, according to one HQ interviewee, caused some ‘backlash(NCV-7-USG). Applications also now are required to have an “affiliation with a local partner” to encourage collaboration and to place emphasis on engaging with “either a local government entity or a local university or a local NGO(NCV-31-USG).

Study methods for IS remained similar to PHEs II methods, allowing randomized experimental designs, quasi-experimental methods, or advanced mathematical techniques (e.g., simulation, mathematical optimization, and decision science) (Bouey and Padian, 2011; OGAC, 2011e, 2012d; Padian et al., 2011). However, study concepts are no longer country driven; instead, PEPFAR evaluation and research priorities and direction are now driven by the SAB (OGAC, 2012e,f), which was formed to “properly advise” PEPFAR (NCV-11-USG) and open OGAC up to input from ‘non-USG people(NCV-7-USG).

The SAB includes 51 members who represent the HIV/AIDS community, academia, international experts, partner country governments, multilateral and bilateral agencies, foundations, advocates, and NGOs (OGAC, 2012g). Box 11-2 lists SAB members’ institutions, grouped by institution type, with the number of members who come from a particular institution given in parentheses, as of October 2012 (OGAC, 2012g). The role of the SAB is to inform the science that drives PEPFAR by providing guidance to OGAC on “scientific, implementation, and policy issues related to the global HIV/AIDS response” (OGAC, 2012f, p. 1). Specific roles include advising on both “broad scientific matters” as well as “emergency and short-notice scientific issues” relevant to PEPFAR, reviewing the quality of evidence being used to inform PEPFAR policies and guidance, reviewing research programs, and identifying evidence gaps and new opportunities (OGAC, 2011a, p. 2, 2012f). The SAB functions through semi-annual meetings as well as periodic conference calls. The board’s inaugural meeting was held in Washington, DC, in January 2011; there have been two additional meetings since, the most recent in October 2012. Additionally, the SAB has formed three working groups, the Combination Prevention Working Group, the Most At-Risk Populations Working Group, and the Data Working Group; these groups are composed of subsets of SAB members who hold conference calls on a more frequent basis to speak about the particular topic area of the working group (OGAC, 2011a, 2012f,h).

In 2010, in concert with the introduction of the IS concept, the first RFA for IS was issued by NIH as 1-year funding supplements to investiga-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

BOX 11-2
Institutional Affiliations of Scientific Advisory
Board Members, October 2012

Academia (members=18)

•   Columbia University (1)

•   Division of Infectious Diseases, University of California, San Diego (1)

•   Emory University (1)

•   Harvard University (1)

•   Imperial College, School of Public Health (2)

•   Johns Hopkins Bloomberg School of Public Health (2)

•   Johns Hopkins University School of Medicine, Infectious Disease Division (1)

•   Makerere University School of Public Health (1)

•   Perinatal HIV Research Unit, University of the Witwatersrand/Chris Hani Baragwanath Hospital (1)

•   Rutgers, the State University of New Jersey (1)

•   The London School of Hygiene and Tropical Medicine (1)

•   UCLA Center for Community Health and Global Center for Children and Families (1)

•   University of California, San Francisco (1)

•   University of North Carolina at Chapel Hill (1)

•   University of Zimbabwe College of Health Sciences, Harare (1)

•   Vanderbilt University Institute for Global Health (1)

U.S. Government (members=11)

•   Department of Defense (1)

•   National Institutes of Health/National Cancer Institute (2)

•   National Institutes of Health/National Institute of Allergy and Infectious Diseases (2)

•   National Institutes of Health/NIH Clinical Center (1)

•   Office of the U.S. Global AIDS Coordinator (2)

•   U.S. Agency for International Development (1)

•   Centers for Disease Control and Prevention (2)

Multilateral/Intergovernmental body (members=5)

•   Global Fund (1)

•   Joint United Nations Programme on HIV/AIDS (UNAIDS) (1)

•   The World Bank (1)

•   World Health Organization (2)

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

NGO (members=11)

•   Center for Global Development (1)

•   Desmond Tutu HIV Center (2)

•   Education, Training, Research Associates (1)

•   Elizabeth Glaser Pediatric AIDS Foundation (1)

•   Family Health International (1)

•   Mayo Clinic (1)

•   ONE Campaign (1)

•   Results for Development Institute (1)

•   The Bill & Melinda Gates Foundation (2)

Philanthropy/Private Sector (members=2)

•   Collaborative Fund for HIV Treatment Preparedness (1)

•   Merck & Co., Inc. (1)

Research Organization (members=4)

•   Baron Edmond de Rothschild Chemical Dependency Institute of Beth Israel Medical Center (1)

•   Human Science Research Council (1)

•   MRC Gender and Health Research Unit (1)

•   South African Centre for Epidemiological Modeling and Analysis (1)

SOURCE: OGAC, 2012g.

tors with current NIH funding. Funding was granted “for research and research training being conducted at PEPFAR funded sites” to inform PEPFAR on effective and efficient approaches to HIV prevention, care, and treatment (NIH, 2010, p. 1). Awards were made to 36 applicants (Homes, 2012). Following this, from 2011 to 2012, additional rounds of RFAs were issued by NIH, CDC, and USAID. As of October 2012, 74 IS awards had been made in total (including the initial 36 NIH supplements); studies addressed the content areas of PMTCT (n=23), voluntary medical male circumcision (VMMC) (n=5), early treatment/treatment as prevention (n=3), improving care and treatment cascade performance (n=19), and building on an HIV platform to address multiple health outcomes and multi-sectoral approaches (n=24). Of these IS studies, six are multi-country and the remaining 68 are single-country studies; overall, IS studies are being conducted in 23 countries (see Figure 11-5) (Holmes, 2012).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

img

FIGURE 11-5 Implementation science awards, by country.

NOTE: Figure adapted from Scientific Advisory Board presentation “Implementation Science Updates.”

a Multi-country includes Côte d’Ivoire, Haiti, India, Kenya (x4), Malawi, Mozambique, Peru, South Africa, Southern Africa (IDeA), Tanzania, Uganda (x3), Zambia.

SOURCE: Adapted from Holmes, 2012.

Overall, 41 unique organizations were involved in implementing the 74 IS studies (Holmes, 2012). These organizations were based in the United States, PEPFAR partner countries, and other non-PEPFAR countries (see Figure 11-6) and represented academia, NGOs, private-sector firms, and research organizations. As the committee received only a snapshot of ongoing PHE studies in 2010, not a comprehensive list of awarded and completed TEs and PHEs over time, it was unable to review and assess changes over time in the distribution of study content area, study country, implementing partner organization type, and country between TEs, PHEs, and IS. It would be useful for PEPFAR to track these parameters in order to assess progress toward the IS goal of doing a better job of matching research activities to the research needs and unique research opportunities within PEPFAR as well as the aim of involving more local entities in the research.

Impact Evaluations

An additional component of the IS initiative, begun in 2012, is impact evaluations. These are studies that address questions of local priority that are carried out in coordination with in-country partners and local government (Goosby, 2012). As described by an interviewee, impact evaluations were created to address a gap that the introduction of the IS RFA process

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

img

FIGURE 11-6 Organizations implementing PEPFAR Implementation Science studies, by implementing organizations’ country, October 2012.

NOTE: Figure compiled from information in the Scientific Advisory Board presentation “Implementation Science Updates.”

SOURCE: Holmes, 2012.

created for mission team–led and –managed research (NCV-7-USG). After the RFA structure was established,

It was felt that there was still a bit of a gap because we did not really have a way for the [mission teams], where there were sort of burning questions and particular interest by the partner government, to be able to do a more larger-scale, more rigorous implementation science study. They didn’t really have a mechanism to be able to initiate those. And that was the idea behind the impact evaluations. To be able to set up a process in which the countries would be able to mobilize those.” (NCV-31-USG)

Proposals for impact evaluations are submitted through the COPs from the mission teams, and funding for the studies comes out of the individual mission team’s budget (NCV-31-USG) (Goosby, 2012; OGAC, 2011e). These studies are intended to use the methodologies of randomized experimental or quasi-experimental design and to produce results with causal attribution assessing what would have occured had the program not been implemented (Padian et al., 2011).

Impact evaluation proposals were accepted for the first time in 2011, as part of a pilot phase (NCV-31-USG). According to one interviewee, five proposals came in, three of those were reviewed, and eventually there was “only one that really made it through the whole process. [. . .] But it was a pilot

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

year(NCV-31-USG). This interviewee went on to describe how the 2011 pilot was used to review lessons learned and to develop mechanisms to overcome some of the barriers and stated hope that in 2012 more impact evaluations would meet the criteria for funding (NCV-31-USG).

A major limitation of the current vision for impact evaluations is that evaluation studies using randomized controlled methods, while important for some purposes such as establishing the efficacy of an intervention, are neither appropriate nor reasonable for evaluating most HIV interventions being implemented in real-world settings. Many of the criteria necessary for robust randomized controlled studies, such as random assignment to the intervention and control groups and a high-intensity implementation of the intervention over a long period of time with continued adherence to intervention or control group, are neither feasible nor suitable in the field (Thomas et al., 2011). The design of an evaluation needs to match the interventions being evaluated, the purpose of the evaluation, the desired findings, the target audience for the findings, and what decisions will be made as a result of the findings. Habicht et al. describe three types of evaluation designs: adequacy, plausibility, and probability. Adequacy evaluations assess a program or impact by comparing it with “previously established adequacy criteria;” these evaluations assess whether or not objectives were met and do not require a control group (Habicht et al., 1999, p. 11). The lack of a control group, however, makes it difficult to conclude that outcomes and impacts are due to the program. Plausibility evaluations go a step further than adequacy evaluations—they use “opportunistic” or “non-randomized control groups” and before-and-after comparisons to “rule out” external factors that may have led to observed outcomes or impacts (Habicht et al., 1999, p. 13). Finally, probability evaluations go a step further still, requiring randomized intervention and control groups to ensure that “there is only a small . . . probability” that observed differences between the program and controls are due to “confounding, bias, or chance” (Habicht et al., 1999, p. 14). Depending on the type of intervention and the stage of implementation, evaluation methodologies beyond randomization and probability should be considered for PEPFAR’s IS initiative, including impact evaluations.

Summary

PEPFAR’s new IS approach represents steps toward making a distinction between research and evaluation through the RFAs and the articulation of the impact evaluation concept. The most recent articulation of the IS is the publication in which it was originally described (Padian et al., 2011), and OGAC has not yet released guidelines or a plan of action for its IS agenda. The article does not clearly describe the separate but complemen-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

tary roles of monitoring, evaluation, and research within PEPFAR—failing to clearly articulate each area’s scope and aims, intended audience, and methods and activities—and therefore it is the committee’s assessment that the article does not serves as a sufficient framework for IS.

Conclusion: Despite challenges, PEPFAR has made progress in carrying out evaluation and research activities over time. PEPFAR has moved from an early proscription against research to using TEs and PHEs to work within research restrictions and to the recent creation of what holds promise as more useful processes for establishing priorities, managing activities, documenting “what works,” expanding PEPFAR’s technical leadership, disseminating research and evaluation findings, and continually improving the effectiveness and impact of PEPFAR. However, even as the roles of research and evaluation within PEPFAR have expanded, defining appropriate and allowable activities remains a challenge—there has not been clarity about the separate but complementary activities and aims for evaluation and research within PEPFAR.

Although the committee was not charged with developing a comprehensive research agenda for PEPFAR, the committee did draw on the available information in the content areas of recent, ongoing, and planned evaluation and research efforts supported by PEPFAR to identify some of the major gaps that warrant more emphasis going forward. The information presented here combined with the more topic-specific assessments from prior chapters in this report indicates that some of the major gaps include research on behavioral and structural interventions, especially in the areas of prevention, gender, nonclinical and OVC care and support, and treatment retention and adherence; longitudinal outcome studies, especially for care and treatment and OVC programs; and research on health systems strengthening interventions across the WHO building blocks that assess setting- and system-specific feasibility, effectiveness, quality of services, and costs for innovative models.

KNOWLEDGE TRANSFER AND LEARNING WITHIN PEPFAR

A key aspect of knowledge management is the transferring of insights, experiences, strategic information, best practices, and lessons learned within an organization. The efficient and timely transfer of knowledge is critical to successfully using the knowledge an organization has acquired and created to improve and change the way that work is accomplished. Knowledge has the greatest impact when it is shared broadly (Garvin, 1993). For PEPFAR to capitalize on knowledge to improve the effectiveness of its programs,

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

it must be able to efficiently capture and transfer strategic information, research results, evaluation outcomes, experiences, best practices, and lessons learned within PEPFAR so that PEPFAR staff, implementers, and other stakeholders at all levels can use and apply this knowledge to improve activities and efforts in support of the HIV response.

Levels, Pathways, and Mechanisms of Knowledge Transfer Within PEPFAR

PEPFAR is a large and complex entity composed of multiple levels and stakeholders that span many countries. PEPFAR functions through two main levels, the central HQ level and the country level, and within these levels there are a multitude of stakeholders involved in coordination, oversight, and program implementation. As discussed in detail in Chapter 3, stakeholders at the HQ level include OGAC, which serves as the administrative and formal organizational unit of PEPFAR, as well as several government implementing agencies used in the response for their core expertise: Department of Commerce, DoD, HHS (including CDC, NIH, HRSA, and Food and Drug Administration), Department of Labor, Department of State, Peace Corps, and USAID. At the country level, stakeholders within PEPFAR include the U.S. interagency mission teams comprised of representatives from the implementing agencies that coordinate and oversee PEPFAR program activities at the country level as well as implementing partners (IPs), which can include U.S.- and partner country–based: universities or other academic organizations, governmental organizations, NGOs, international NGOs, multilaterals, and private-sector organizations that implement PEPFAR programs.

For knowledge to have the greatest impact possible within PEPFAR, multiple types of knowledge must be transferred efficiently within, between, and across PEPFAR levels and stakeholders. Because of its size and complexity, there are many potential pathways for knowledge transfer within PEPFAR; these are depicted in Figure 11-7. Through a review of the interview data and OGAC guidance documents, the committee determined that knowledge transfer could occur within each of the entities in PEPFAR, e.g., within OGAC, within a USG mission team in a country, or within a particular implementing partner; this is depicted by the circular arrows in the figure. Furthermore, as depicted by the solid straight arrows, knowledge transfer could occur between levels or entities—between the HQ level and the country level, such as between OGAC and the U.S. interagency mission teams at the country, or between entities in a particular level, such as between a U.S. interagency mission team and implementing partners in a particular country, and so on. Finally, knowledge transfer could occur across PEPFAR countries, e.g., across U.S. interagency mission teams in

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

img

FIGURE 11-7 Potential pathways of knowledge transfer within PEPFAR.

NOTE: IP = implementing partner.

SOURCE: Interview data and review of OGAC guidance documents.

different countries or across implementing partners in different countries; the dotted arrows represent the points at which knowledge transfer could occur across countries. Given the size, span, and complexity of PEPFAR and the vast number of potential pathways of knowledge transfer, it can be difficult to determine which pathways and what scales of knowledge transfer are appropriate, efficient, and sufficient.

PEPFAR stakeholders have established and used a variety of ways to transfer a wide range of knowledge within, between, and across PEPFAR levels, entities, and countries. As described in depth earlier in this chapter, PEPFAR has a system for routine reporting of program monitoring data. OGAC uses semiannual and annual reports and COPS to routinely gather not only programmatic data from country programs but also other information related to PEPFAR programs and activities, progress toward targets, and strategic direction and planning. However, the types of knowledge transferred in PEPFAR and the mechanisms used to share knowledge go well beyond the reporting of program monitoring data and other routine reporting, as described in more detail below from the perspective of interviewees involved in PEPFAR implementation. The types of knowledge shared and the scale of sharing vary, depending on the pathway of transfer.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Pathways of Knowledge Transfer

Interviewees described various specific pathways through which the vast amount of knowledge in PEPFAR is transferred among different stakeholders; these are summarized in Box 11-3. The preponderance of interviewees described knowledge transfer from the HQ level to the country level, within the country level, and across countries. There were fewer reports of knowledge transfer from the country level up to HQ. At the HQ level, however, interviewees did report bi-directional sharing of best practices (NCV-18-USG) and policies (NCV-2-USG) between the country level and HQ, using HQ technical working groups (NCV-18-USG), individuals working within government implementing agencies (NCV-2-USG), or country coordinators (934-43-USG) as conduits of information from the field up to HQ. As one interviewee noted:

It’s important for us who are assigned to different countries to meet about all the countries to discuss what’s happening and to ensure that, [. . . we] know what each other are learning and what’s kind of rising to the top as a promising intervention that should be tested in other areas.” (NCV-18-USG)

Types of Knowledge Transferred by Different Pathways

Table 11-6 lists the types of knowledge, beyond routine program monitoring data, that interviewees described as being shared within PEPFAR along with the multiple different stakeholder pathways through which these types of knowledge are transferred. Interviewees described how information, data and evidence, and policies and guidelines are transferred from the HQ level to the country level as well as at the country level. The preponderance of interviewees described sharing of experiences as primarily occurring at the country level (240-9-USG; 240-14-USG; 331-14-USG; 396-9-PCG; 396-19-USG; NCV-5-USAC) and across countries (116-1-USG; 934-43-USG; NCV-5-USACA; NCV-9-USG; NCV-18-USG; NCV-17-USG); there were fewer reports of experiences being shared between the HQ level and the country level. A few interviewees did, however, describe instances when experiences in the form of best practices (NCV-18-USG) and case studies (NCV-18-USG) were transferred from HQ to the country level.

Mechanisms of Knowledge Transfer

In general, organizations use a variety of mechanisms to spur the process of knowledge transfer, including reports, presentations, study tours, personnel rotation programs, and training. Reports and presentations are popular ways to transfer knowledge because they make it possible to summarize a large amount of knowledge for wide distribution, and study tours are popular among large organizations that want to transfer knowledge

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

BOX 11-3
Pathways of Knowledge Transfer in PEPFAR,
Beyond Routine Reporting

Knowledge transfer from central HQ to all of PEPFAR

•  From OGAC to all of PEPFAR (NCV-2-USG; NCV-3-USG; NCV-7-USG; NCV-10-US; NCV-17-USG)

Knowledge transfer from central HQ to the country level

•  From OGAC to:

o   The country level (stakeholders not specified) (934-43-USG; NCV-7-USG; NCV-10-USG; NCV-17-USG; NCV-18-USG)

o   U.S. interagency mission teams at the country level via government implementing agencies at the HQ level (934-43-USG; NCV-2-USG)

o   Implementing partners at the country level (NCV-10-USG)

•  From OGAC HQ technical working groups to:

o   The country level (stakeholder not specified) (NCV-17-USG; NCV-18-USG)

o   U.S. interagency mission teams at the country level (934-43-USG; NCV-11-USG)

Knowledge transfer from the country level to central HQ

•  From country level (stakeholder not specified) to OGAC HQ TWGs (NCV-18-USG)

•  From U.S. interagency mission teams at the country level to OGAC via government implementing agencies at the HQ level (NCV-2-USG) or PEPFAR Country Coordinators (934-43-USG)

Knowledge transfer within the country level

•  From a U.S. interagency mission team to implementing partners in a country (116-1-USG; 166-12-USG; 272-36-USG; 396-19-USG; 461-20-PCPS; NCV-11-USG)

•  Between implementing partners in a country (116-1-USG; 240-14-USPS; 272-36-USG; 331-14-USG; 396-19-USG; 461-16-USG; 461-20-PCPS; 636-1-USG; NCV-7-USG)

•  From implementing partners to a U.S. interagency mission team in a country (396-5-USNGO)

•  Within a particular implementing partner in a country (396-5-USNGO; 396-56-USNGO; NCV-5-USACA; NCV-8-USACA)

•  Within the U.S. interagency mission team in a country (166-12-USG; 396-19-USG; 396-57-USG)

Knowledge transfer across countries

•  Across U.S. interagency mission teams in different countries (240-9-USG; 272-36-USG; 396-18-USG; 396-1920-USG; 934-43-USG; NCV-9-USG; NCV-17-USG; NCV-18-USG)

•  Across implementing partners in different countries (116-1-USG; 396-9-PCGOV; NCV-7-USG; NCV-18-USG)

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

TABLE 11-6 Types of Knowledge Transferred in PEPFAR, Beyond Routine Reporting

Type of Knowledge HQ to Country Level Country to HQ At Country Level Across Countries Level Not Specified
Information
(116-1-USG; 166-12-USG; 272-36-USG; 396-19-USG; 396-57-USG; 934-43-USG; NCV-2-USG; NCV-7-USG; NCV-8-USACA; NCV-17-USG; NCV-18-USG)
X   X X  
Data and Evidence
Data
(NCV-10-USG; 116-1-USG; 636-1-USG; 461-16-USG; NCV-5-USACA NCV-8-USACA)
X   X    
Programmatic indicator data (other than routine reporting)
(NCV-3-USG; NCV-7-USG; 396-19-USG; 396-5-USG; 166-12-USG; 461-20-USG; NCV-5-USACA)
X   X    
Program results (461-20-USG)     X    
Evidence
(NCV-10-USG; 934-43-USG; NCV-17-USG; NCV-18-USG)
X     X  
Research
(NCV-18-USG)
X        
Policies and Guidelines
Targets
(NCV-2-USG)
X        
Policies
(NCV-2-USG)
X X      
Program guidelines/guidance
(NCV-11-USG; NCV-10-USG; NCV-17-USG)
X   X    
Feedback
(116-1-USG; 396-56-USG; 934-43-USG; NCV-5-USACA)
  X X    
Experiences
Experiences
(NCV-18-USG; NCV-9-USG; NCV-17-USG; 396-9-PCGOV)
X   X X  
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Type of Knowledge HQ to Country Level Country to HQ At Country Level Across Countries Level Not Specified
Best practices
(934-43-USG; 331-14-USG; NCV-18-USG; NCV-5-USACA)
X X X X  
Country case studies (NCV-18-USG) X        
Challenges
(240-14-USG; 116-1-USG)
    X X  
Innovation
(396-19-USG; NCV-5-USACA)
    X    
Intervention targeting (240-9-USG)     X    
Lessons learned
(396-19-USG; NCV-18-USG; 396-18-USG; NCV-5-USACA)
    X X X
Successes
(240-14-USG)
    X    
Successful transition models
(396-18-USG)
        X

across a wide geographic reach (Garvin, 1993). Although reports, presentations, study tours, and other passive mechanisms are useful in transferring certain types of knowledge, such as information, data and evidence, and policies and guidelines, they can prove cumbersome in transferring knowledge that is tacit, difficult to articulate, or resides in the experiences of personnel. The literature indicates that personnel rotation programs are a more effective method to transfer implicit knowledge, as such implicit knowledge is more easily absorbed through experience (Garvin, 1993).

The qualitative interview data indicate that within PEPFAR both active and passive knowledge transfer mechanisms are used, with the types of mechanisms described by interviewees falling into five main categories: intermediaries, meetings (both PEPFAR-supported and external), reports and published guidelines, online technology, and study tours and staff rotation (see Table 11-7). Most interviewees described very similar mechanisms of knowledge transfer across interviews; interviewees did indicate, however, that the scale of use and the particular mechanism used depended on the pathway of knowledge transfer.

Across PEPFAR, similar mechanisms are used to facilitate knowledge transfer, but their use varies depending on the particular pathway of transfer. OGAC HQ shares knowledge with the field through intermediaries (934-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

43-USG; NCV-2-USG; NCV-11-USG; NCV-17-USG; NCV-18-USG), reports and published guidelines (NCV-10-USG), meetings (934-43-USG; NCV-17-USG), and multiple forms of online technology (NCV-2-USG; NCV-17-USG; 396-57-USG; NCV-7-USG). At the country level stakeholders use many of these same mechanisms to transfer knowledge, including meetings (116-1-USG; 240-14-USPS; 272-16-PCNGO; 272-36-USG; 331-14-USG; 461-16-USG; 461-20-PCPS; 636-1-USG; NCV-5-USACA; NCV-11-USG), reports (166-12-USG; 396-5-USNGO; 396-19-USG; 461-20-PCPS), and online technology (396-19-USG; 396-57-USG; NCV-5-USACA; NCV-8-USACA). To transfer knowledge across countries, however, informal staff exchange (396-18-USG; NCV-18-USG; 935-27-USG; 935-28-USG) and study tours (240-9-USG; 396-9-PCGOV; NCV-9-USG) were described as mechanisms of transfer along with intermediaries (934-43-USG; 935-27-USG; 240-33-USG; 396-57-USG; 934-43-USG) and various types of multi-country meetings (116-1-USG; 272-36-USG; 396-19-USG; 934-43-USG; NCV-5-USACA; NCV-7-USG; NCV-8-USACA; NCV-18-USG; 587-25-ML; 166-25-USG). Examples of the various mechanisms are described in more detail in the sections that follow.

Intermediaries Interviewees described intermediaries mainly as a tool specific to transferring knowledge bi-directionally between HQ and the country level (see Table 11-7). Many interviewees described how intermediaries in the form of TWGs (934-43-USG; NCV-11-USG; NCV-17-USG), technical advisors (NCV-2-USG; NCV-18-USG), government implementing agencies at HQ (934-43-USG; NCV-7-USG), and country coordinators at the country level (934-43-USG) served as conduits of knowledge between HQ and the countries. As one HQ interviewee explained it, HQ TWGs and advisors transfer knowledge to mission teams in-country, which then in turn transfer knowledge to implementing partners:

The technical working groups for PEPFAR are comprised of really experts in the field. And they [. . .] spend probably 60 percent or more of their time traveling to countries. And the focus of their work is with our country teams [. . .] like the prevention guidance, when it was rolled out, we did a series of phone calls, we did Q and As, we did open lines, we then had people going out doing prevention portfolio reviews with our country teams. So that they could figure out how to realign and more to the guidance, how to stage their change over time [. . .] But it’s always headquarters, for the most part, is working with our country teams. Not with partners that do the actual implementation. But then our country teams will be having these kinds of meetings with the partners, and working with them. So that’s basically how it moves out and how we use our technical folks.” (NCV-11-USG)

A country-level interviewee echoed this, noting that most knowledge comes through HQ TWGs and government implementing agencies rather than directly from OGAC and describing OGAC as doing a “decent job” of sharing knowledge (934-43-USG). Country coordinators can also serve as in-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

TABLE 11-7 Mechanisms of Knowledge Transfer in PEPFAR

Mechanism of Knowledge Transfer HQ to Country Level Country to HQ At Country Level Across Countries
Intermediaries
Country coordinators
(934-43-USG; 240-33-USG; 396-57-USG; 935-27-USG)
X X X  
HQ TWG
(934-43-USG; NCV-11-USG; NCV-17-USG)
X X    
Technical advisors
(NCV-2-USG; NCV-18-USG)
X X    
Implementing agencies
(934-43-USG; NCV-7-USG)
X      
PEPFAR-Supported and External Meetings
Conference calls
(934-43-USG; 935-27-USG; NCV-11-USG; NCV-17-USG; NCV-18-USG)
X   X X
Conferences
(331-14-USG; 396-19-USG; NCV-7-USG; 587-25-ML; 166-25-USG; 272-16-PCNGO)
    X X
Meetings
(116-1-USG; 272-36-USG; 461-20-PCPS; NCV-5-USACA; NCV-17-USG; NCV-18-USG)
X   X X
Annual coordinators meeting
(934-43-USG; 935-27-USG)
  X   X
Country-level partners meeting
(272-36-USG; 636-1-USG)
    X  
Multi-country meetings within an implementing partner
(NCV-5-USACA; NCV-8-USACA)
    X X
PEPFAR annual HIV/AIDS Implementers' Meeting
(116-1-USG; 272-36-USG; 935-27-USG; NCV-7-USG; NCV-17-USG)
X   X X
Technical area forums
(NCV-18-USG)
      X
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Mechanism of Knowledge Transfer HQ to Country Level Country to HQ At Country Level Across Countries
Workshops
(240-14-USPS; 461-16-USG; 461-20-PCPS)
    X  
Reports/Guidelines
Presentations
(396-5-USNGO)
    X  
Published guidelines
(NCV-10-USG)
X      
Responses to data requests
(166-12-USG; 461-20-PCPS)
    X  
Routine reports
(396-19-USG; 461-20-PCPS)
    X  
Online Technology
Databases
(396-19-USG; NCV-5-USACA; NCV-8-USACA)
    X  
Listservs
(396-57-USG)
    X  
PEPFAR extranet site
(NCV-2-USG; NCV-17-USG)
X   X  
Routine electronic bulletins
(NCV-5-USACA; NCV-18-USG)
X   X  
SharePoint website
(396-57-USG; NCV-7-USG)
X   X  
Technical area-specific websites
(NCV-10-USG; NCV-18-USG)
X      
Webinar
(NCV-5-USACA; NCV-18-USG)
X   X  
Staff Rotation/Study Tours
Staff rotation
(396-18-USG; 396-23-USG; 935-27-USG; 935-28-USG; NCV-2-USG; NCV-9-USG; NCV-18-USG; NCV-31-USG)
  X   X
Study tour
(240-9-USG; 396-9-PCGOV; NCV-9-USG)
      X
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

termediaries to transfer knowledge between OGAC and the country teams (934-43-USG) as well as across countries (240-33-USG; 396-57-USG; 934-43-USG; 935-27-USG); these intermediaries were described by one interviewee as a “clear channel of communication” between OGAC and the country teams (934-43-USG). Another interviewee described communication between country coordinators as a mechanism to transfer knowledge across countries, sharing differences across programs as well as ideas of how to handle similar challenges (935-27-USG). The use of intermediaries was described by many interviewees as a main mechanism for transferring knowledge from HQ to the country level (934-43-USG; NCV-2-USG; NCV-7-USG; NCV-11-USG; NCV-17-USG; NCV-18-USG). Intermediaries in turn use other mechanisms to transfer knowledge within the partner countries (NCV-11-USG).

Meetings The preponderance of interviewees described various meeting types as tools used to share knowledge through three pathways: (1) between HQ and the country level, (2) at the country level, and (3) across countries (see Table 11-7). Specific meeting types mentioned by interviewees included single- and multi-country conference calls (934-43-USG; NCV-11-USG; NCV-17-USG; NCV-18-USG), attendance at conferences (331-14-USG; 396-19-USG; NCV-7-USG; 587-25-ML; 166-25-USG; 272-16-PCNGO), periodic country coordinator meetings (in person or by phone) (934-43-USG; 935-27-USG), country-level implementing partners meetings (272-36-USG; 636-1-USG), multi-country meetings of a particular implementing partner (NCV-5-USACA; NCV-8-USACA), PEPFAR’s annual HIV/AIDS Implementers’ Meeting (116-1-USG; 272-36-USG; 935-27-USG; NCV-7-USG; NCV-17-USG), technical area–specific forums (NCV-18-USG), and workshops (240-14-USPS; 461-16-USG; 461-20-PCPS).

PEPFAR’s annual Implementers’ Meeting is a forum for PEPFAR staff and implementing partners to exchange knowledge, discuss issues, transfer information, and share lessons learned and best practices (IOM, 2007a). Six HIV/AIDS implementers’ meetings were held between 2004 and 2009 (IOM, 2007a; OGAC, 2007a, 2008a, 2009a). The first meeting was held in South Africa with 100 attendees, limited to USG personnel, and focused on the management and structure of PEPFAR (IOM, 2007a). Over time the HIV/AIDS Implementers’ Meeting included more and more attendees and expanded to include individuals and organizations involved in PEPFAR implementation as well as other stakeholders involved in the global HIV/AIDS response. The meeting evolved to become a forum for networking and dialog among implementers and PEPFAR staff to share implementation lessons learned, best practices, barriers experienced, and various other types of information (IOM, 2007a; OGAC, 2007a, 2008a, 2009a).The most recent meeting, entitled Optimizing the Response: Partnerships for Sustainability, was held in 2009 in Namibia with more than 1,500 participants, 220 oral presentations, and 125 poster presentations (OGAC, 2009a). One HQ interviewee said that these meetings were originally designed to allow PEPFAR implementing partners to share information with each

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

other and with multilaterals; that interviewee also described how the role had expanded to become an opportunity for sharing among countries with increased participant involvement, with abstracts and presentations coming from country-specific programs, and with expanded attendance from international NGOs and local country researchers and implementers (NCV-7-USG).

Similar to the implementers’ meetings, some interviewees described periodic country-level partners meetings, either in person or by video, as a means to transfer information, data, and best practices among partners at the country level (272-36-USG; 636-1-USG). Beyond these PEPFAR-wide or stakeholder-specific meetings, there are technical area–specific forums, which one HQ TWG interviewee described as a way to transfer knowledge across countries on a particular topic:

We have every couple of years a [. . .] forum where we bring all the field people together, and we just had one in February that was you know spent 4 days just going over the latest evidence and discussing how they should be implementing this latest evidence and discussing what’s working, what’s not working. So that’s been another excellent means of communicating.” (NCV-18-USG)

In addition to the formal PEPFAR Implementers’ Meeting and country-level partners meetings, some PEPFAR HQ technical working groups have organized various technical exchanges and field-driven learning meetings, which serve as mechanisms for internal knowledge transfer focused on specific topic areas. Summary reports of these workshops and meetings are made available online to potentially increase the knowledge transfer to a wider audience. The reports describe the meetings as platforms for the sharing and exchange of various types of knowledge, including experiences, programming, best practices, strategic planning, successes, opportunities, and challenges. The representatives at these meeting vary by topic, but mainly include PEPFAR staff and partners (JSI, 2012). Illustrative examples of these meetings include

  •  Strengthening Gender Exchange in PEPFAR: Technical Exchange of Best Practices, Program Models, and Resources, organized by the PEPFAR Gender Technical Working Group and held in South Africa in 2009;

  •  Field Driven Learning Meeting: Linkages to and Retention in HIV Care and Support Programs, organized by the PEPFAR HIV Care and Support Technical Working Group and held in Mozambique in 2010; and

  •  Meeting the HIV; Maternal, Newborn, and Child Health; and Social Support Needs of Mothers and their Young Children, a

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

      field-driven learning meeting organized by several working groups (Care and Support, Prevention of Mother-to-Child Transmission, Orphans and Vulnerable Children, and Food and Nutrition) and held in Ethiopia in 2011 (Bergmann, 2011; Fullem, 2012; Spratt and Bergmann, 2011).

Finally, in addition to meetings organized by PEPFAR, interviewees pointed out that various informal opportunities for knowledge transfer among PEPFAR stakeholders exist, even though they are not organized by PEPFAR. These include attendance at international and regional conferences or in-country workshops and participation in national-level technical working groups (116-1-USG; 240-14-USPS; 396-19-USG; 461-16-USG; NCV-7-USG; 587-25-ML; 166-25-USG).

Reports and published guidelines As the literature points out, reports and presentations are a popular tool used by organizations to transfer explicit knowledge widely to an array of stakeholders (Garvin, 1993). PEPFAR has established formal routine reporting systems, including annual progress reports (APRs), semi-annual progress reports (SAPRs), and COPs to transfer knowledge from the country level to HQ (GAO, 2011a). In addition to these formal reporting channels, interviewees identified presentations (396-5-USNGO), responses to data requests (166-12-USG; 461-20-PCPS), and country-level reporting (396-19-USG; 461-20-PCPS) as tools for knowledge transfer at the country level (see Table 11-7). Mission team interviewees described producing and disseminating country- and site-level reports and data visualizations of program monitoring data (166-12-USG; 396-19-USG; 461-20-PCPS) and responding to ad hoc data requests (166-12-USG; 461-20-PCPS) as ways to share program monitoring data with implementing partners. Additionally, a HQ interviewee identified guidelines published by HQ as a tool for knowledge transfer between HQ and the country level (NCV-10-USG).

Online technology Advances in technology have produced new mechanisms of knowledge transfer in organizations; the Internet, intranets, and other online technologies are extending the reach, accessibility, and diffusion of knowledge throughout organizations. PEPFAR stakeholders have embraced the use of online technology to transfer knowledge internally, utilizing intranet sites, electronic bulletins, websites, and webinars as tools of knowledge transfer (396-57-USG; NCV-2-USG; NCV-5-USACA; NCV-7-USG; NCV-10-USG; NCV-17-USG; NCV-18-USG) (see Table 11-7).

In response to an expressed need at the 2005 Annual Implementers’ Meeting for improved peer-to-peer communications to share information and best practices in the field, OGAC established the PEPFAR extranet site, PEPFAR.net (OGAC, 2007d). This secure website, open to USG personnel working on PEPFAR, was introduced in 2006 as a space for OGAC to

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

share information with the field, including a “News to the Field” e-mail bulletin, presentations, policies and guidelines, and public affairs and public diplomacy resources. The website was also described as a space for mission teams, technical working groups, and PEPFAR staff to network, collaborate, and share lessons learned, best practices, presentations, articles, guidelines, and resources (OGAC, 2007c).

The committee did not have access to the PEPFAR extranet site to review its current content and how it is being used for knowledge transfer, but interviewees did describe the site as ‘the central node’ for knowledge transfer within PEPFAR and mentioned that HQ TWGs use it to share guidance, scientific literature, reports, and tools with the field (NCV-2-USG; NCV-17-USG). Despite this, one interviewee had mixed messages on how effective the site is as a tool for knowledge transfer, and noted that it was currently being updated to improve its use for knowledge management (NCV-2-USG). In addition to the extranet site, OGAC uses the “News to the Field” e-mail bulletin as a way to transfer recent news, updates, and information (e.g., guidance, FAQs, best practices, policies, etc.) from HQ to the field (OGAC, 2007d; Simonds et al., 2012).

In addition to PEPFAR’s official online communication tools described above, HQ TWGs and implementing partners have used technology in innovative ways to facilitate knowledge transfer between HQ and the country level as well as at the country level (see Table 11-7). Examples include developing and using country- and partner-specific databases to overcome the barrier of not having a centralized database at OGAC from which to store, manage, and share program data (396-19-USG; NCV-5-USACA; NCV-8-USACA); us-ing listservs and SharePoint websites to share information among mission teams (396-57-USG; NCV-7-USG); supporting technical area–specific websites that give implementing partners access to evidence, research, and information on a particular topic (NCV-10-USG; NCV-18-US); and hosting online webinars (NCV-5-USACA; NCV-18-USG).

Staff rotation and study tours Although not formally instituted as a standard knowledge transfer mechanism throughout PEPFAR, study tours and staff rotation were described by some HQ- and country-level interviewees as informal tools for transferring knowledge across countries as well as between the country level and HQ (see Table 11-7) (240-9-USG; 396-9-PCGOV; 396-18-USG; 396-23-USG; 935-27-USG; 935-28-USG; NCV-2-USG; NCV-9-USG; NCV-18-USG; NCV-31-USG). In one partner country, for example, the prevention TWG wanted to learn about targeting activities to men who have sex with men and did a study tour to another PEPFAR country to learn from its model (240-9-USG). Another partner country interviewee expressed gratitude for being able to use PEPFAR funds for study tours:

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

PEPFAR is very big donors for [us], especially for the national M&E system of our country. Thanks to PEPFAR funding we can [. . .] organize different study tours to overseas to different countries in order to learn from their experience.” (396-9-PCGOV)

Interviewees also gave examples of staff rotation being used for knowledge transfer. Staff rotation programs have been characterized in the scientific literature as “one of the most powerful methods of transferring knowledge,” especially knowledge that is tacit and difficult to learn through passive means (Garvin, 1993, p. 9). Interviewees described various forms of staff rotation that occur within PEPFAR, including staff members moving between PEPFAR countries, staff moving from the country level to the HQ level and vice versa, and staff working on detail between USG agencies, e.g., a staff member from CDC brought to work on detail at OGAC (396-18-USG; 396-23-USG; 935-27-USG; 935-28-USG; NCV-2-USG; NCV-9-USG; NCV-18-USG; NCV-31-USG). In one partner country, members of a TWG identified PEPFAR staff who had worked in other countries as an opportunity to garner lessons from the experiences of other PEPFAR countries, specifically, in this case, on transitioning from service delivery to technical assistance. One of the interviewees noted that although this was not a “formalized exchange,” it was used as an opportunity to transfer knowledge (396-18-USG). Additionally, a HQ TWG interviewee described successful integration of staff rotation in the TWG as a mode of knowledge transfer across countries. In the words of the interviewee:

So in other words our South Africa [. . .] technical lead has spent time in Nigeria working on their program and our Namibia person is about to go to Zambia and spend some time there because we’re not the only experts obviously. There’s a lot of different very valuable expertise out there and we think it’s really great that people have the opportunity to do that.” (NCV-18-USG)

Finally, HQ interviewees shared that their previous experiences working for PEPFAR at the country level provided them with insight, unique perspective, and ideas that contributed to their current positions at the HQ level (NCV-9-USG; NCV-18-USG; NCV-31-USG). Similar sentiments were shared by an interviewee working at the country level after previously working at the HQ level (935-27-USG; 935-28-USG).

Study tours and staff rotation were described as successful tools to transfer experiences, implementation models, lessons, and expertise across PEPFAR, but these methods of knowledge transfer are used on a small scale and not systematically. Staff rotation, if scaled up, used intentionally and strategically, and formally adopted as a standard mechanism in PEPFAR, could provide a useful tool to effectively transfer successful country/partner

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

models of sustainability, country ownership, and technical assistance across PEPFAR.

Summary

In summary, OGAC and PEPFAR stakeholders have developed and used a multitude of formal and informal mechanisms to transfer many types of knowledge throughout the different levels of PEPFAR stakeholders. Formal mechanisms systematically implemented by OGAC to facilitate knowledge transfer include routine reporting from the country-level APRs, SAPRs, and COPs; periodic teleconferences; the PEPFAR Annual Implementers’ meeting; the PEPFAR extranet site; the use of intermediaries as conduits of information to the field; and weekly “News to the Field” e-mails. Informal mechanisms of knowledge transfer not systematically implemented across PEPFAR include various meetings, conferences, and forums; country-level reports; the use of online technology databases, websites, and webinars; and staff rotation and study tours.

PEPFAR Knowledge Transfer Barriers and Limitations

Simply creating, collecting, or acquiring knowledge within an organization is not enough for an organization to learn and improve performance. Knowledge must be disseminated widely and used for the greatest impact (Garvin, 1993). Although PEPFAR has been successful in establishing and using a wide variety of formal and informal mechanisms to transfer knowledge both systematically and intermittently throughout PEPFAR, barriers to knowledge transfer exist and there is still a perceived need for more formalized mechanisms for transferring experiences across countries and implementing partners.

One major barrier to knowledge transfer arises from the lack of a strategic approach to knowledge transfer and the limitations of current systems and processes. OGAC has not articulated goals, purposes, and a plan of action for knowledge transfer in PEPFAR. OGAC has issued several strategy, progress, and guidance documents: PEPFAR Five-Year Strategies, annual reports to Congress, guidance documents on collecting and reporting program monitoring data, guidance documents on evaluation and research activities, and a published article outlining PEPFAR’s IS plan (OGAC, 2004, 2005c, 2006a, 2007f, 2009e,f, 2010b, 2012a; Padian et al., 2011). Although these documents describe knowledge creation and acquisition and may mention an increased emphasis on transparency or dissemination, none of them describes clear goals or provides a detailed plan for how program data, results from evaluations, and experiences (e.g., les-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

sons learned, best practices, and models) will be systematically transferred and shared, either internally or externally, to PEPFAR, beyond the routine reporting to OGAC.

Other knowledge transfer barriers identified by interviewees were related to determining the right knowledge to transfer, the appropriate scale of knowledge transfer, and the correct mechanisms of transfer (272-36-USG; 396-57-USG), and a lack of stakeholder engagement in transfer processes (396-57-USG). As one mission team interviewee observed, ‘[T]he problem is knowing when to and when not to share something; inclusiveness can lead to an excess of information(396-57-USG). Another interviewee stated that in-country partners meetings, in that particular country, did not work as a knowledge transfer mechanism because of the vast number of implementing partners in country but that video conferences were a useful and effective alternative (272-36-USG). Finally, one interviewee pointed out that having a knowledge transfer mechanism in place—in this case, a SharePoint site to share data across agencies and teams—did not guarantee that the site would be utilized by staff (396-57-USG).

Beyond the challenges in the current knowledge transfer systems and processes there is a perceived need for more formalized mechanisms for transferring experiences, such as lessons learned, best practices, innovations, and models. Much of this perceived need was articulated by mission teams and country-level implementing partners who reported a need for more formalized mechanisms for transferring knowledge across countries, implementing partners, and implementation sites. Country-level interviewees recognized the opportunity to learn from the experiences of other countries and voiced a desire for information and best practice sharing regionally and globally (196-8-ML; 934-46-PCGOV), cross-country exchange programs (331-22-PCNGO), and opportunities for the transfer of lessons learned between countries (331-23-USNGO; 587-23-USG; 587-25-ML; 272-36-USG; 542-11-PCNGO; 396-18-USG). This need for increased knowledge transfer across countries was expressed strongly by multiple interviewees. In the words of one interviewee:

I think that a lot more work can be done, cross-fertilization of lessons learned in different countries. A specific example within the [Region X] is that when I was working in [Country A], for example, oodles of effort and resources went into making sure that they developed this awesome electronic medical record. [. . .] Wonderful work done around that. Then I go to [Country B] and I see amazing work being done [. . . on] their community-based [. . .] data collection system. Amazing work that we weren’t able to get done in [Country A]. Two countries that were PEPFAR countries in this region. Why there hasn’t been like exchange visits, or some means of sharing those lessons learned, I don’t know. But evidently,

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

that’s not necessarily a priority to someone. So that might be a good thing to look towards in the future. But regionally, you know, making sure that any resources that have shown [. . .] promise for the future are shared with other countries.” (587-25-ML)

When describing sharing within regions and among countries, one interviewee said that there were lessons to be learned across countries but that the processes for sharing were not systemized and that PEPFAR doesn’t seem to want to spend money on regional meetings (272-36-USG). Another interviewee expressed a desire to see more collaboration among regions and countries with programs facing similar epidemics, noting that while some of this sharing happens on an ad hoc basis, it is not a PEPFAR-run activity (396-57-USG). Echoing the country-level interviewees’ viewpoints, one HQ interviewee reinforced the need for mechanisms of cross-country exchange after attending a Global Health Initiative meeting and seeing knowledge transfer across countries. In the words of the interviewee:

It was fascinating, just kind of like the hunger that was there, the hunger that got satisfied from people actually having that opportunity to exchange information, to talk about problems. And we [OGAC] all realize that for years we’ve been talking about south-to-south learning and you know here we were, an opportunity to bring a bunch of countries together. There were never enormous meetings.” (NCV-9-USG)

In addition, when describing knowledge transfer at the country- and implementing partner–levels, interviewees voiced a need for an increased exchange of information and success stories across implementing partners and service delivery sites within a particular partner country (116-23-USPS; 166-23-USG; 196-26-USG; 396-25-PCGOV; 396-32-PCGOV; 396-44-PCGOV; 587-23-USG). Some interviewees suggested standardized mechanisms to share data between implementing partners (396-5-USNGO) and study tours (396-25-PCGOV; 396-32-PCGOV; 396-44-PCGOV) as ways to increase knowledge transfer. Many of the informal knowledge transfer mechanisms currently used on a small scale by implementing agencies, technical working groups, mission teams, and implementing partners could be useful tools for more formal scale-up and routine use across PEPFAR to enhance knowledge transfer and overcome barriers in the current systems.

Conclusion: Although a wide variety of mechanisms have been successfully established and used to transfer an array of knowledge throughout PEPFAR, more progress is needed to address limitations in current processes and systems and to establish formalized mechanisms to transfer experiences across countries, implementing

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

partners, and sites as systematically as is desired by stakeholders. Without this, there will be missed opportunities to capitalize on best practices and internal lessons learned that could be applied to improve PEPFAR-supported programs.

Lessons Learned on Knowledge Transfer from Implementing Partners

The Track 1.0 partners and other international NGOs, as large PEPFAR implementing partners that manage HIV programs operating in multiple countries with multiple stakeholders involved, face similar challenges to those faced by OGAC in transferring an array of knowledge across different stakeholders, organizational levels, and countries. At HQ and country levels, interviewees at these large implementing partners offered examples of innovative mechanisms used to transfer both explicit and tacit (experiential) knowledge effectively and efficiently throughout their organizations. The mechanisms they reported include partner-developed databases to store, manage, and disseminate data; periodic webinars; publications; feedback sessions; and multi-country meetings and conferences (NCV-5-USACA; NCV-8-USACA). OGAC could look to these large implementing partners that operate in multiple countries for innovative models to scale up across PEPFAR to overcome knowledge transfer barriers and increase the efficiency and effectiveness of knowledge transfer throughout PEPFAR.

To illustrate some of the mechanisms used by these partners, Box 11-4 describes innovative approaches used by one particular PEPFAR implementing partner, “Organization X” (the organization name has been de-identified for confidentiality purposes). Organization X operates in multiple countries and has implemented several knowledge management mechanisms to promote the use of routinely collected data for strategic planning, information sharing with others, and program evaluation. This implementing partner shares knowledge acquired and created by the organization through both active and passive knowledge transfer systems, including a sophisticated database reporting system, reports and periodic bulletins, dissemination meetings, feedback meetings, conferences, and webinar series.

PEPFAR’S KNOWLEDGE DISSEMINATION EXTERNAL TO PEPFAR

As described earlier in the chapter, PEPFAR acquires and creates vast amounts of knowledge in the form of routine PEPFAR program monitoring data, additional program and clinical data collected by implementing partners, outcomes and results of evaluation and research activities (TEs, PHEs, basic program evaluations, IS, and impact evaluations), and data acquired through support of partner country surveillance systems and surveys, as well as tacit knowledge gained through experiences. To have the

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

BOX 11-4
“Organization X” Innovative Knowledge Transfer

Database Reporting System

Organization X developed a central database reporting system to capture, integrate, store, manage, analyze, and disseminate reporting data from multiple organizational activities and programs. The Web-based system was launched to streamline and standardize routine data collection as well as to increase the use of routine data by organization staff for program monitoring, strategic planning, and improving the quality of service delivery. All organizational staff from HQ to the country and site levels have access to the system. System features include standardized data entry with built-in data checks, data navigation by site or activity, a system dashboard to summarize data with “real-time” updates, automated summary tables and reports by country and site, exportable reports and raw data, interactive map features to indicate location of services and sites, and triangulation of data sources.

Reports

The organization’s database system can be used by staff at any organizational level to produce reports for monitoring programs, documenting progress, and reporting to donors. The organization HQ disseminates routinely collected data through quarterly, semi-annual, and annual summary reports to partner country governments and funding partners. Additionally, the organization HQ produces easy-to-read facility- and region-level feedback reports used for guiding program implementation.

Meetings

Organization X uses several types of meetings to transfer tacit and explicit knowledge across implementing partners and sites, including but not limited to:

•   Annual meetings of in-country teams, implementing partners, and sites for the purpose of sharing data, lessons learned, innovations, and tools across implementing partners and countries.

•   Feedback meetings where representatives from implementing sites in a particular region come together to share data and see how they are doing.

•   Organization- and country-wide data dissemination meetings held three to four times per year, where representatives from the organization HQ share their in-depth analysis of the routinely collected aggregate program data and patient-level data with the implementing partners.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Webinars and Online Technology

The organization uses webinars and online technology to transfer and disseminate knowledge across the organization, specifically:

•   The organization HQ runs online webinars series that are available to all staff. These focus on clinically relevant topics, such as improving effectiveness of adherence in HIV treatment settings, the role of microbicides in HIV prevention, and nurse-initiated and managed antiretroviral therapy, as well as dissemination, highlighting data that have been collected, and data collection experiences in the field. Webinars are archived online and available for viewing.

•   The organization HQ distributes a periodic electronic bulletin that is sent to implementing partners and sites to highlight aggregate reported program data and illustrate the utility of routinely collected data.

•   The organization has a publicly accessible webpage that includes online resources such as guidance, manuals, and toolkits for different program areas and also monthly newsletters that feature the webinar schedules, program success stories, and current research and activities. Archives of the newsletter are available online.

SOURCE: Website Organization X, Interview with Organization X HQ.

greatest public health impact, this knowledge needs not only to be transferred within PEPFAR but also to be disseminated beyond PEPFAR for use by partner countries and the international HIV/AIDS community, including other funders, researchers, evaluators, and the public.

Recently there has been increasing recognition by PEPFAR leadership of their responsibility to maximize the knowledge created in PEPFAR by disseminating it widely. The PEPFAR reauthorization under the 2008 Lantos-Hyde Act8 and PEPFAR’s second Five-Year Strategy both placed an emphasis on improving efforts to disseminate PEPFAR data and findings, expand the publicly available data for analysis to inform public health, and continue to contribute to the global HIV/AIDS response evidence base (OGAC, 2009f). Additionally, the 2011 article outlining PEPFAR’s IS plan underscored the need for the next phase of PEPFAR to place an emphasis on the “development and contribution of knowledge about HIV/AIDS

__________________

8Supra, note 5.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

program implementation to the global community” (Padian et al., 2011, p. 202). Finally, the 2011 PEPFAR Annual Report to Congress stated that PEPFAR is “redoubling its efforts to apply and disseminate” PEPFAR lessons learned and data (OGAC, 2011k, p. 5). Even prior to the recent emphasis on knowledge dissemination, however, PEPFAR was already, to some degree, disseminating gained knowledge to external stakeholders, as this section will illustrate.

Knowledge dissemination external to PEPFAR occurs on many levels. PEPFAR stakeholders share knowledge with Congress, other global HIV/AIDS partners, partner country governments, and the public. Additionally, PEPFAR contributes knowledge to the global evidence base around HIV/AIDS program implementation. Across these levels, PEPFAR utilizes various platforms to disseminate knowledge, including published reports, online technology, participation in conferences, routine and ad hoc reporting, and publications.

Reporting to Congress

OGAC reports a portion of the PEPFAR program monitoring data and updates on progress to Congress on a routine basis through annual reports to Congress as well as ad hoc reports that meet special congressional or White House requests (NCV-2-USG; NCV-3-USG). In addition to having been provided to Congress, all of the annual reports to Congress from 2005 to 2012 are archived and available to the public online at the PEPFAR website, along with more than 25 other PEPFAR reports to Congress from 2004 to present (February 2013). In addition to the reports to Congress, every 5 years OGAC releases the Five-Year Strategy of the U.S. President’s Emergency Plan for AIDS Relief. The most recent strategy, released in 2009 and publicly available on the PEPFAR website, reflects on lessons learned during the previous 5 years and outlines the future direction of the program (OGAC, 2009f, 2012i).

Knowledge Sharing with Other Partners and Funders in the Global HIV/AIDS Response

Beyond reporting to Congress, OGAC and PEPFAR stakeholders have implemented a few official and informal mechanisms to share knowledge with other partners and funders in the global HIV/AIDS response. As described previously, OGAC meets separately every 6 months to a year with both UNAIDS and the Global Fund to share and compare data (NCV-3-USG; NCV-21-ML). PEPFAR’s relationship and coordination with the Global Fund is discussed in more detail in Chapter 10.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Furthermore, interviewees described the Implementers’ Meeting as an official mechanism developed by OGAC for systematic active sharing of knowledge with other global partners in the HIV/AIDS response; other mechanisms described by interviewees for this type of exchange were informal and implemented on a country-by-country basis (NCV-7-USG; NCV-11-USG; 116-1-USG; 116-2-USG; 196-13-OGOV; 240-15-USG; 461-4-USG; 461-25-ML; 636-9-USACA; 934-43-USG)G)). The Implementers’ Meeting, described previously in the chapter, was created by OGAC as an official mechanism of knowledge transfer that is open to a multitude of stakeholders, including outside partners and funders involved in the global HIV/AIDS response (OGAC, 2009a). The Implementers’ Meeting serves as a formal forum created by OGAC for cross-stakeholder sharing; however, an open Implementers’ Meeting has not been held since 2009. There was no Implementers’ Meeting in 2011 because of scheduling and cost issues, and in 2012 the meeting was not held because of the need for PEPFAR to have representation at the 2012 International AIDS Conference (NCV-7-USG).

One HQ interviewee described the Implementers’ Meeting as an opportunity for PEPFAR implementing partners to share information with each other as well as with other multilaterals, such as WHO, UNAIDS, and the Global Fund, but went on to state that multilateral engagement in the meetings has been more limited (NCV-7-USG). One HQ interviewee reported that OGAC is currently working on developing new methods to share information about PEPFAR countries with the Global Fund to use in decision making and funding selection (NCV-11-USG).

In addition to the Implementers’ Meeting, multiple interviewees gave examples of informal mechanisms used to share knowledge with external partners and funders, including contributions to partner country reports for UNAIDS and WHO as well as meetings and forums. One HQ interviewee said that PEPFAR program monitoring data are circulated to some external HIV/AIDS partners by indirect means when these data are used to contribute to partner country national data, which are then reported to the United Nations (UN) and WHO (NCV-7-USG). One mission team interviewee said that the team shares PEPFAR data directly with UN agencies, and another described regular country-level meetings for the purpose of sharing data with other donors (196-1-USG; 461-4-USG), but this was not heard across interviewees as an official mechanisms of knowledge transfer established by OGAC. Finally, in several countries interviewees described participation in country-level stakeholder and development partner forums (i.e., the Global Fund country coordinating mechanism, the Health Development Forum, the AIDS Development Partners Forum, and development community meetings) as opportunities used by PEPFAR to share knowledge among stakeholders in the response (116-2-USG; 196-13-OGOV; 240-15-USG; 461-25-ML; 636-9-USACA; 934-43-USG).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

A lack of knowledge sharing across partners in the HIV/AIDS response emerged as a theme in interviews with multilateral funders, bilateral funders, and other HIV response partners (196-13-OGOV; 166-9-OBL/ML/USACA/USNGO/PCNGO/PCPS; 272-6-ML). One bilateral interviewee reflected on the need for better collaboration among funders to work closely together planning from the beginning, saying that “when we’re doing designs rather than [when we are] tailoring them and starting to implement them [. . .] that’s way too late(196-13-OGOV). The interviewee went on to express frustration with PEPFAR for keeping his/her organization in the dark while re-designing the country program in a country where both PEPFAR and his/her organization were engaged, stating that PEPFAR’s processes occurred in a box “with things happening inside but we [the bilateral organization] didn’t really know exactly what was happening in the box(196-13-OGOV). This view was reiterated by an interviewee from a multilateral organization in another country, who noted that PEPFAR was working with the partner country government to streamline HIV/AIDS indicators but that PEPFAR was not very clear on what was happening. The interviewee emphatically stated, “[J]ust tell PEPFAR to share what they do, we’re partners(272-6-ML). Finally, other multilateral and bilateral organizations working in a partner country pointed out that within a country PEPFAR holds technical meetings limited to the USG and PEPFAR partners, but it would be beneficial if these conversations happened in a broader environment involving more stakeholders (166-9-OBL/ML/USACA/USNGO/PCNGO/PCPS). Across interviews, the theme emerged that OGAC lacks formal mechanisms to exchange knowledge with external partners (e.g., bilateral organizations, multilateral organizations, and other partners) involved in the HIV/AIDS response.

Knowledge Sharing with Partner Country Governments

At the country level PEPFAR knowledge dissemination occurs when mission teams and implementing partners share knowledge with partner country governments through meetings and presentations as well as through routine and ad hoc reporting. These multiple avenues of knowledge sharing with partner country governments are described below, followed by the many challenges that remain in transparently sharing information between PEPFAR and partner country governments.

Avenues for Knowledge Sharing with Partner Country Governments

Many country-level strategic information TWGs and implementing partners reported sharing PEPFAR data with the national government (587-9-USG; 166-12-USG; 396-19-USG; 396-56-USNGO; 461-16-USG). One partner country government interviewee confirmed such data sharing between PEPFAR and the national

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

government, saying, “[I]n terms of data management I think that up to now we have no problem, we share together(396-6-PCGOV). Meetings with partner governments were described as a common mode to exchange knowledge, including routine coordination meetings (116-24-USG; 272-36-USG; 240-5-PCGOV; 240-7-PCGOV; 240-15-USG; 331-4-PCGOV; 587-7-PCGOV), technical area-specific meetings (272-36-USG), Partnership Framework meetings and processes (396-57-USG), and PEPFAR staff participation in national TWGs (196-20-PCNGO; 396-19-USG; 461-15-USG; 934-2-USG).

Beyond meetings, PEPFAR mission teams share program data and research findings with partner governments through various country-level reports and presentations on an ad hoc basis (240-3-USG; 272-22-US; 396-56-USNGO; 461-1-USG; 542-21-USNGO; 587-3-USG; 934-38-PCACA). Additionally, in several countries, the mission teams share country APR and SAPR results and COPs (240-33-USG) directly with partner country governments (166-12-USG; 240-33-USG; 272-36-USG; 396-19-USG). Some interviewees reported that disseminated program data and research findings were then used by partner country governments for UNGASS reporting (396-56-USNGO; 461-15-USG), program planning (272-22-USG), and intervention implementation (542-21-USNGO; 934-38-PCACA). The frequency and scale of use of these mechanisms of sharing, meetings, and reports, seemed to be at the discretion of the individual country mission team or implementing partner.

PEPFAR implementing partners routinely report data to partner government systems. Interviewees from mission teams in several countries described how implementing partners are required or encouraged to report and share PEPFAR program monitoring data directly with partner country governments at the national, regional, and district levels as well as with government agencies such as the ministry of health, national AIDS commissions, and provincial AIDS commissions (116-1-USG; 196-22-PCGOV; 272-36-USG; 331-34-USNGO; 496-19-USG; 461-18-USG). Interviewees did not, however, mention how such required or encouraged reporting was enforced or tracked. Finally, a couple of PEPFAR implementing partners and mission teams reported sharing data in response to partner country government data requests (166-12-USG; 396-5-USNGO; 461-20-PCPS), although another interviewee said that specific data requests from the government to PEPFAR mission teams may be limited by varying capacities across the government to formulate requests (166-12-USG).

Challenges in Knowledge Sharing with Partner Country Governments

Despite these mechanisms used by mission teams in partner countries to share with partner country governments, many interviewees described barriers to knowledge dissemination with partner country governments, including a lack of transparency from PEPFAR concerning financial data (166-16-PCGOV; 166-19-PCGOV; 240-33-USG; 396-16-PCGOV; 461-8-PCGOV), differing FYs (396-1920-USG; 396-56-USNGO; 461-4-USG), a lack of harmonization between PEPFAR and partner

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

country reporting systems (166-16-PCGOV; 396-7-PCGOV; 461-20-PCPS), and a lack of routine systems for sharing (166-16-PCGOV).

Partner country government interviewees criticized PEPFAR for a lack of transparency concerning expenditure data. They described challenges in accessing data on how money is being spent and which implementing partners are being funded and at what level as well as data on the costs of activities. Interviewees pointed out that this lack of transparency leads to challenges in monitoring, coordinating, procurement, and planning the national HIV/AIDS response (166-16-PCGOV; 166-19-PCGOV; 396-16-PCGOV; 461-8-PCGOV). This critique has also been voiced by outside researchers who claim that PEPFAR is restricting access to data on how public funds are spent and who point out that other large funders, such as the Global Fund, are able to have financial transparency while still protecting proprietary implementing partner information (Grosso et al., 2012). Furthermore, differing fiscal years and reporting timeframes lead to challenges with sharing data between PEPFAR and partner country governments in a timely and meaningful way (396-19-USG; 396-56-USNGO; 461-4-USG). One mission team interviewee pointed out that when the COP and APR reports are due relatively close together, there is little time to get feedback from the government prior to report submission (461-4-USG).

A few partner country government and private-sector interviewees cited a lack of alignment between country-level M&E systems and PEPFAR data collection systems (described previously in the chapter) as a barrier to knowledge dissemination (166-16-PCGOV; 396-7-PCGOV; 461-20-PCPS). Although several mission team interviewees reported that implementing partners are required or encouraged to report program monitoring data to the national M&E systems (116-1-USG; 272-36-USG; 331-34-USNGO; 496-19-USG; 461-18-USG), in some cases interviewees noted that some PEPFAR implementing partners report data only to PEPFAR and not into the national M&E system (166-16-PCGOV; 396-7-PCGOV; 461-20-PCPS). One partner country government interviewee observed that implementing partners do not feel obliged to report data to the national HIV/AIDS M&E system because they are already sharing the data with PEPFAR (166-16-PCGOV). Another interviewee involved in collecting and managing data for PEPFAR described the repercussions of partners reporting only to PEPFAR and not the national system (461-20-PCPS):

So we get data from them [PEPFAR implementing partners] that comes into our system [PEPFAR’s system], but that data that they provide us may not necessarily be in the national system because they haven’t been, you know, reporting their data into the national system. [. . .] If we are to strengthen national systems, that’s not good because you know it’s not appropriate for the national level to be missing out on data of that nature because it’s an important

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

contribution. [. . . W]e are not here permanently [Contractor], we are here to support programs, but at the end of the day the country will remain and its systems will need to be strengthened.” (461-20-PCPS)

Finally, one partner government interviewee noted that PEPFAR provides data to the national government when they “knock on [the] door” and request it, but that the national government would prefer data (programmatic and financial) to flow automatically between PEPFAR and the partner government (166-16-PCGOV).

In summary, PEPFAR stakeholders have mechanisms in place at the country level to disseminate knowledge to partner country governments, but these processes were as not as transparent, routine, and systemized as preferred by some interviewees.

Conclusion: OGAC could contribute to increased coordination among partners in the HIV/AIDS response by developing official routine and systematic mechanisms for knowledge exchange with other partners involved in the response at both the global and country levels, including partner country governments, other donors, and multilateral organizations.

PEPFAR Knowledge Dissemination to the Public and Contribution to the Global Knowledge Base

As one of the largest funders addressing the global HIV/AIDS epidemic, PEPFAR has both the capacity and the responsibility to play a significant leadership role in ensuring that knowledge created through the HIV/AIDS response is disseminated broadly for the greatest public health impact and that data, research results, and evaluation outcomes are available to other researchers, evaluators, and the public to help accelerate the pace of new knowledge creation. PEPFAR could lead the way in setting new standards of transparency and knowledge dissemination and in contributing solutions to common knowledge sharing barriers. As this section will describe, PEPFAR has begun taking steps to facilitate this broader knowledge dissemination to the public as well as contributing to the knowledge base. As noted earlier, PEPFAR leadership has recognized the importance of knowledge dissemination beyond PEPFAR (OGAC, 2009f; Padian et al., 2011); some knowledge dissemination mechanisms are in place or in the process of being implemented to facilitate sharing with the public; OGAC has stated that it is planning a formalized dissemination platform (OGAC, 2011b); and PEPFAR implementing agencies and partners have contributed vast amounts of evidence to the global knowledge base on effective HIV/AIDS interventions and program implementation through publications, reports,

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

technical guidance, tools, and participation in conferences. Despite these steps, however, significant strides are still needed to ensure that all knowledge created through PEPFAR is transparent, available, and disseminated widely and efficiently.

Public Knowledge Dissemination Mechanisms

PEPFAR stakeholders at the HQ level and the country level use various platforms to disseminate knowledge created in PEPFAR to the public and to contribute to the global knowledge base on effective HIV/AIDS interventions and program implementation; these platforms include online technology (NCV-7-USG; NCV-10-USG; NCV-11-USG; NCV-18-USG) (OGAC, 2011b), participation in meetings and conferences (NCV-4-USACA; NCV-6-USNGO; NCV-7-USG; 116-1-USG; 272-27-USG; 331-23-USNGO; 331-44-USNGO)O)), and reports and publications (NCV-2-USG; NCV-4-USACA; NCV-7-USG; NCV-8-USACA; 196-11-USNGO; 272-22-USG; 272-27-USG; 331-23-USNGO; 396-19-USG; 396-53-USNGO; 461-16-USG).

Online technology OGAC staff, HQ TWGs, and PEPFAR implementing agencies use various online technologies to raise awareness about PEPFAR, share information with the public, and disseminate research and evaluation findings; these technologies include blog posts, Twitter, Facebook, YouTube, Flickr, the official PEPFAR website, and other PEPFAR-supported websites (NCV-7-USG; NCV-10-USG; NCV-11-USG; NCV-18-USG) (OGAC, 2011b). Box 11-5 describes the various websites that PEPFAR supports and the types of knowledge disseminated through them.

Meetings and conferences In addition to online technology, both HQ- and country-level interviewees cited participation at international conferences (NCV-4-USACA; NCV-6-USNGO; NCV-7-USG; 116-1-USG; 331-44-USNGO), at scientific meetings (935-27-USG), and at periodic stakeholder meetings (331-23-USNGO) as mechanisms used to ensure PEPFAR results are disseminated widely. One HQ interviewee stated that PEPFAR staff are actively encouraged to submit abstracts to conferences and highlighted examples of staff attendance and participation at CROI (Conference on Retroviruses and Opportunistic Infections) and the 2012 International AIDS Conference and also some limited participation at ICASA (International Conference on AIDS and STIs in Africa) (NCV-7-USG). Interviewees at the country level described dissemination through presentations at international conferences (331-44-USNGO), abstracts with PEPFAR programmatic data (935-27-USG), and presentations of unpublished data at country-level conferences (116-1-USG). A HQ interviewee did note, however, that USG quotas on the number of USG staff attending overseas meetings are a limitation on using conferences and meetings as a dissemination tool (NCV-7-USG).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

BOX 11-5
PEPFAR-Supported Websites

PEPFAR public website (www.PEPFAR.gov)

Launched in 2006 to raise awareness about PEPFAR, share information with the public, highlight events and recent news, and disseminate research and evaluation findings, the website provides links to several resources, including the PEPFAR Five-Year Strategy, Reports to Congress, COPs, Partnership Frameworks, and limited information on budget information, programmatic data, and publications.

AIDstar-One (www.aidstar-one.com)

Managed by USAID, the website provides access to a variety of HIV/AIDS resources, including technical area-specific webpages to access resources by topic (e.g., HIV prevention and HIV treatment), a comprehensive database of HIV program best practices and innovation shared by implementers and program planners, and a technical resources section with cases studies, strategic plans, reports, publications, success stories, guidelines, and more to assist public health practitioners develop evidence-informed HIV programs. The resource section can be searched and filtered by resource type, focus area, region, and country. A search of the resource section for PEPFAR returned more than 500 resources, including guidelines, case studies, tools, reports, documents, and events.

Development Experience Clearinghouse (DEC) (dec.usaid.gov/dec)

A searchable database of USAID’s technical and program related-

Reports and publications Finally, a preponderance of HQ- and country-level interviewees highlighted reports and publications as tools to disseminate PEPFAR program activities and program data, lessons learned, research results, and evaluation outcomes (NCV-2-USG; NCV-4-USACA; NCV-7-USG; NCV-8-USACA; 196-11-USNGO; 272-22-USG; 272-27-USG; 331-15-USG; 331-23-USNGO; 396-19-USG; 396-53-USNGO; 461-16-USG)G)). Interviewees noted annual reports to Congress, regional and provincial pamphlets, and contributions to UNGASS reports and Universal Access Reports as mechanisms to share programmatic data and special survey data (331-15-USG; 461-16-USG; 396-19-USG; 396-53-USNGO; NCV-2-USG).

In addition to programmatic data, reports and publications are used to share lessons learned. Recently, OGAC contributed to the development and public release of two special issues of scientific journals focused on lessons learned in the first two phases of PEPFAR (NCV-7-USG). The Health Affairs special issue “Assessing the President’s Emergency Plan for AIDS Relief,” released July 2012, examined PEPFAR successes, lessons learned, and next steps and the Journal of Acquired Immune Deficiency Syndromes

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

documentation, it allows users to search, rate, and download documentation and lets USAID staff and contractors upload documents. PEPFAR-related documents and publications are available here. An advanced search of documents in the database for “HIV/AIDS” and “PEPFAR” resulted in 740 documents.

OVCSupport.net (www.ovcsupport.net)

A platform for “the exchange of experience, practice, and tools on policy and programming” related to children and HIV, the website has sections for news and exchange, policy and research, programming resources, and discussion forums and also contains a library that is searchable by category. HG TWGs use this website to disseminate research and evaluation findings related to children and HIV. A library search for “PEPFAR” returned 41 results.

What Works for Women and Girls (www.whatworksforwomen.org)

The website provides a review of the data from HIV/AIDS interventions for women and girls from roughly 100 countries. It gives the public and PEPFAR partners access to literature and evidence on interventions that work for women and girls. Evidence can be sorted by strategy effectiveness (e.g., “works,” “promising”), strategy (e.g., prevention, strengthening the enabling environment), keyword, location, and gray rating.

SOURCES: (NCV-10-USG; NCV-17-USG); AIDStar-Two, 2012; Gay et al., 2012; JSI, 2012; OGAC, 2012j; USAID, 2012.

special issue “PEPFAR: Its Vision, Achievements, and New Directions,” released August 2012, examined PEPFAR experiences and insights from a scientific and programmatic perspective (Dentzer, 2012; Ho et al., 2012). In addition, HQ- and country-level interviewees described other dissemination of PEPFAR data, activities, research results, and evaluation outcomes through publications in professional journals (NCV-4-USACA; NCV-8-USACA; 272-22-USG; 272-27-USG; 331-23-USNGO; 196-11-USNGO), with one interviewee stating that ‘hundreds of papers’ have resulted from PEPFAR evaluation activities (272-22-USG). HQ- and country-level interviewees identified multiple successful examples of using reports and publications to disseminate knowledge created in PEPFAR. Interviewees, however, also identified barriers to using publications as a mechanism for the dissemination of knowledge gained through PEPFAR. These included a lack of in-country capacity to publish (116-1-USG; 331-24-PCGOV), restrictions on conducting research (NCV-4-USACA), and multiple levels of approval needed to collect, present, and publish data (NCV-4-USACA).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Dissemination Successes

As noted previously, PEPFAR implementing agencies and partners have contributed vast amounts of evidence to the global knowledge base on effective HIV/AIDS interventions and program implementation through publications, reports, technical guidance, and participation in conferences. In some cases, PEPFAR publications, research results, and evaluation outcomes have been used to influence policy and to inform modifications in HIV/AIDS interventions and program implementation (116-1-USG; 116-12-PCNGO; 116-23-USPS; 166-7-PCGOV; 196-28-USG; 240-9-USG; 272-22-USG; 272-25-USG; 272-27-USG; 331-14-USG; 331-22-PCNGO; 331-43-USG; 396-12-USG; 396-1920-USG; 396-53-USNGO). To assess the scale of PEPFAR’s contribution to the global knowledge base, the committee requested from each of the four Track 1.0 partners, as well as from OGAC, CDC, and USAID, a list of publications resulting from PEPFAR support; several of the entities noted that the lists they provided were not completely comprehensive back to the beginning of PEPFAR. The separate lists were combined in EndNote, and duplicate references were removed to create one extensive, non-redundant list of PEPFAR-supported publications over time. Based on the information provided, from 2004 through to March 2012 more than 1,700 journal publications had been produced as a result of PEPFAR (see Figure 11-8). This is certainly an underestimate of the total number of publications produced with PEPFAR support because not all stakeholders were surveyed and the count does not include other publications beyond those that appeared in

img

FIGURE 11-8 PEPFAR-supported journal publications, by year, 2004–2011.

NOTE: The figure represents journal publications through 2011, the final complete year in the lists available, which were all received by March 2012 (161 publications were listed as being published between January and March 2012).

SOURCE: Publication lists received from OGAC, USAID, CDC, and Track 1.0 Partners.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

journals (e.g., reports, pamphlets, case studies, tools, etc). OGAC was not able to provide a comprehensive list of publications.

Although they had not been requested to do so, a few entities provided the committee with lists of other dissemination products resulting from PEPFAR funding in addition to the lists of journal publications that were requested. Although the numbers are neither comprehensive nor representative of all PEPFAR stakeholders, it is worth noting that the PEPFAR entities from whom the committee requested publication lists also reported having disseminated, from the inception of the program to March 2012, 10 books (entities reporting=3), more than 950 abstracts (entities reporting=4), more than 75 technical guidelines or training materials (entities reporting=3), and more than 100 WHO-supported documents (entities reporting=1) as a result of PEPFAR support.

Beyond producing numerous dissemination products, many HQ- and country-level interviewees described how knowledge created and disseminated through PEPFAR (e.g., research results, evaluation outcomes, PEPFAR program data, surveys, and publications) has been used to influence country-level policies and planning (116-23-USPS; 272-22-USG; 272-25-USG; 272-27-USG; 396-1920-USG; 396-53-USNGO). Interviewees also noted the contribution of PEPFAR to the global knowledge base on improving HIV/AIDS interventions and program implementation (272-24-USG; 272-25-USG; 272-36-USG; 461-4-USG; 461-8-PCGOV; NCV-10-USG). As evidenced above, a large volume of knowledge has been disseminated as a result of PEPFAR in the form of publications, abstracts, technical guidelines, and reports, and in many cases this knowledge has been used to influence policies and improve HIV/AIDS interventions and program implementation.

Measuring and Tracking Contributions to the Global Knowledge Base

As previously noted, PEPFAR implementing agencies and partners have created and disseminated vast amounts of knowledge since the inception of the program, yet it is difficult to assess PEPFAR’s contribution to the knowledge base because of the lack of a centralized system or approach to track publications, abstracts, guidelines, and reports that result from PEPFAR funding. In order to assess the scale of PEPFAR’s contribution to the global knowledge base on effective HIV/AIDS interventions and program implementation, the committee requested a comprehensive list of publications that resulted from PEPFAR support from the beginning of PEPFAR. OGAC could not provide the committee with such a comprehensive list and instead provided a couple of lists that together, when duplicates were removed, contained 169 publications and 3 abstracts published between 2006 and 2011. OGAC informed the committee that publications generated through PEPFAR-funded activities are “currently tracked through ad hoc systems

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

and coordination with PEPFAR’s implementing agencies” (OGAC, 2011b, p. 1). The lack of a system to track PEPFAR publications was reiterated by a HQ interviewee, who said, ‘OGAC never successfully managed to track the work that was being accomplished [research and evaluation TEs, PHEs, and IS]’ or ‘any reports that come out of that work(NCV-7-USGOV). After receiving the initial publication list from OGAC, the committee asked the Track 1.0 Partners, USAID, and CDC to provide their own lists of PEPFAR-supported publications; while all the organizations were able to provide lists, a couple stated that they were unable to provide comprehensive lists. The lists were combined in EndNote, with duplicate publications removed, which resulted in a single extensive, but not comprehensive, list of more than 1,700 journal publications supported by PEPFAR from the inception of the program.

Although the committee was able to determine one measure of PEPFAR’s contributions to the global knowledge base, this number is most likely an underestimate because there are an unknown number of additional dissemination products that have not been captured, tracked, or enumerated in a central platform. OGAC informed the committee that, to address this limitation, it is currently constructing a tracking system for PEPFAR evaluations that will also provide a “comprehensive central repository for PEPFAR-funded publications when it is completed” (OGAC, 2011b, p. 1). Furthermore, HQ interviewees reported that, to better track PEPFAR-funded publications, OGAC is working with USG agencies to track PEPFAR-funded evaluation and research reports, encouraging agencies to cite PEPFAR as a source of funding when they publish, and working with the National Library of Medicine to get a search term for “PEPFAR” in Medline and to have anything funded by PEPFAR be tagged with the PEPFAR search term (NCV-7-USG).

In addition to the lack of a system to track PEPFAR-supported publications, PEPFAR lacks a central repository or tool that the international HIV/AIDS community (e.g., partner countries, other funders, researchers, evaluators, and the public) can use to access PEPFAR-funded publications, abstracts, reports, and tools and build off the knowledge base created through PEPFAR. The PEPFAR website provides links to some reports, guidance, publications, and other resources, but they represent only a small subset of the actual number of knowledge contribution products resulting from PEPFAR funding. As of October 2012, the publications section of the website, for example, provided access to only 21 publications that had resulted from PEPFAR support (OGAC, 2012j). The GAO noted this limitation, for evaluation results, in its review of PEPFAR’s evaluation activities. The study found that PEPFAR stakeholders use a variety of mechanisms to share evaluation findings but that not all of the evaluation reports are online, which limits their availability to the public and also limits the utility

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

of the results for stakeholders in the HIV/AIDS response (GAO, 2012). One HQ-level interviewee described one challenge that results from not having a centralized repository of PEPFAR dissemination products: ‘It’s hard to search for and find PEPFAR reports(NCV-7-USG). As a result of PEPFAR’s efforts, OGAC, implementing agencies, and implementing partners have successfully contributed vast amounts of evidence to the global knowledge base on effective HIV/AIDS interventions and program implementation in the form of publications, reports, guidelines, tools, and participation in conferences. There is, however, no formalized system for tracking and no central repository for accessing these knowledge dissemination products, which makes it difficult to assess PEPFAR’s contribution to the global knowledge base and also makes it difficult for the resulting knowledge to be fully utilized for maximum impact.

Conclusion: OGAC would benefit from having a more systematic method to track PEPFAR-funded dissemination products (e.g., publications, reports, abstracts, and guidelines) in order to measure and manage their contribution to the global knowledge base on effective HIV/AIDS interventions and program implementation. Additionally, the international HIV/AIDS community would benefit from PEPFAR having a more robust publicly available central repository of PEPFAR-funded publications, abstracts, reports, and tools from which to share, collaborate, and accelerate knowledge creation.

Data Sharing

Practitioners in the field of public health are starting to realize the importance of data sharing for maximum impact of research outputs and data collection efforts, yet public health still lags behind other research fields in terms of the necessary infrastructure, standards, and incentives to facilitate this data sharing (Walport, 2011). While data sharing is an important goal, prior to establishing effective dissemination systems, stakeholders must find ways to ensure equitable access to knowledge, to disseminate knowledge efficiently, to protect data privacy, and to overcome researcher resistance to sharing (Walport, 2011). As one of the largest funders addressing the global HIV/AIDS epidemic, PEPFAR has the opportunity to play a significant leadership role in making HIV/AIDS monitoring, evaluation, and research data available to other researchers, evaluators, and the public so that these data are used for the greatest public health impact and to accelerate the pace of new knowledge creation. PEPFAR could lead the way in establishing infrastructure, standards, and incentives to encourage sharing of HIV/AIDS data and in contributing solutions to addressing barriers to data sharing.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Currently PEPFAR does not have a data-sharing policy in place that promotes access to data collected by implementing partners, mission teams, and contractors with PEPFAR support (i.e., program monitoring data, financial data, evaluation outcomes, and research results). As a result, PEPFAR stakeholders collect an immense amount of data with PEPFAR funds that are not readily available to outside researchers, evaluators, and the global HIV/AIDS community for use in new knowledge creation. As described previously, only a limited amount of PEPFAR program indicators and financial data are made publicly available via the PEPFAR website, in annual reports to Congress, and in other reports (NCV-2-USG) (OGAC, 2012j), and the program and financial data that are available are usually presented in a report format that is not easily usable by researchers and evaluators instead of being offered in Excel files or datasets. Furthermore, as previously mentioned, when the committee requested PEPFAR program monitoring data from OGAC for use in this evaluation, there were no data beyond the seven key indicators that are reported annually to Congress that were readily available. Next Generation Indicator (NGI) data were eventually provided to the committee, but only a limited number of PEPFAR I indicators were available and provided for use in the evaluation.

Since the inception of PEPFAR many large implementing partners, such as the Track 1.0 partners, have been collecting additional data beyond routinely reported program data; this additional data includes cohort data, data on retention and adherence, and mortality data. These data, collected with PEPFAR support, could add significant value to the global knowledge base on HIV/AIDS program implementation and interventions, yet many of these data are not publicly available in a usable format to maximize knowledge creation. Because there is no PEPFAR policy requiring data sharing, Track 1.0 partners and other stakeholders are not obligated to make data collected with PEPFAR support publicly available. When the evaluation committee requested access to portions of Track 1.0 partner supplemental data for purposes of the PEPFAR evaluation, all of the partners were very willing to work with the committee to share insight, information, publications, and presentations and were open to exploring data-sharing possibilities, but only one of the three organizations with supplemental data was ultimately willing to share data with the committee. Among the reasons that the other two Track 1.0 partners gave for not sharing their data were readiness of the data, the time and resource burden required to prepare the data, and concerns about preserving data publication rights.

OGAC has created a data working group within the SAB to address the fact that although large amounts of data are collected through PEPFAR, “researchers, clinicians and even OGAC do not have access to it” (OGAC SAB Data Working Group, 2011, p. 1). The mission of this data working group is to advise OGAC on “how it can best gather, disseminate and set

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

policy regarding information generated through the PEPFAR program,” and one objective of the group is to “recommend policy and procedures on data management, data access, data sharing, and release of appropriate data” (OGAC, 2012h). Other research fields, USG agencies, and public health entities with established data-sharing policies could serve as models for PEPFAR and the SAB data working group for developing infrastructure and standards for data sharing as well as for how to select data of appropriate readiness and utility to share. These entities could also provide lessons learned about ensuring equitable access to knowledge, efficient dissemination, and the protection of data privacy. A fair and effective data-sharing policy could maximize the use of data created through PEPFAR and spur innovation and discovery.

Conclusion: As PEPFAR evolved, leadership increasingly recognized the importance of maximizing the impact of knowledge acquired and created through PEPFAR by disseminating it widely beyond PEPFAR. HQ- and country-level stakeholders have developed mechanisms and taken steps to share PEPFAR knowledge with Congress, other partners and funders in the HIV/AIDS response, partner country governments, and the public. Additionally, PEPFAR implementing agencies and partners have contributed evidence and vast amounts of publications to the global knowledge base on effective HIV/AIDS interventions and program implementation. Despite these successes, more progress is needed in disseminating knowledge external to PEPFAR, particularly in sharing knowledge with partner country governments and other partners involved in the HIV/AIDS response, increasing the amount of PEPFAR data (e.g., routinely collected program monitoring data, evaluation outcomes, and research results) publicly available for use by researchers and evaluators, and tracking and measuring PEPFAR’s contribution to the global knowledge base.

SUMMATION

As shown throughout this chapter, PEPFAR has made significant strides in knowledge management—acquiring, creating, capturing, sharing, and using knowledge. However, there are many areas where more progress is needed to address limitations in PEPFAR’s current knowledge management approaches in order to (1) ensure successful monitoring and evaluation of PEPFAR goals and activities, especially as the model of implementation shifts; (2) continually improve programs; and (3) maximize the impact of knowledge created in PEPFAR to contribute to sustainable, country-owned HIV/AIDS responses.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

PEPFAR has developed and contributed to various systems to acquire and generate knowledge, including creating a PEPFAR-specific program monitoring data collection system to track activities and program results, supporting epidemiologic and surveillance activities in partner countries, strengthening partner country health information systems, implementing various program evaluation approaches, and supporting research across a wide range of technical areas. As a result, OGAC, implementing agencies, and implementing partners have successfully acquired and created vast amounts of knowledge, often at a scale not seen in other development programs. This includes program monitoring, epidemiological, and surveillance data; evaluation outcomes and research results; and best practices, lessons learned, successes, challenges, and innovation.

At the HQ level PEPFAR has utilized various sorts of knowledge (e.g., program monitoring data, epidemiologic data, normative guidance, and intervention effectiveness data) to drive program activities and inform efforts, and at the partner country level there are good examples of data use by PEPFAR stakeholders. Additionally, PEPFAR has carried out initiatives to build capacity and increase data use among partner country governments and PEPFAR implementing partners, contributing to the fostering of a culture of evidence use among partner countries.

OGAC and PEPFAR stakeholders have developed and used a wide variety of formal and informal mechanisms to transfer knowledge within PEPFAR, including reporting, intermediaries, meetings, conferences, published guidelines, online technology, study tours, and staff rotation. In addition to internal knowledge transfer and use, PEPFAR leadership has increasingly recognized the importance of knowledge dissemination beyond PEPFAR. There are some dissemination mechanisms in place to share knowledge with Congress, other global HIV/AIDS partners, partner country governments, and the public; these mechanisms include online technology, publications, reports, technical guidance, tools and training materials, and participation in meetings and conferences. As a result of PEPFAR’s dissemination efforts, vast amounts of evidence have been contributed to the global knowledge base on effective HIV/AIDS interventions and program implementation. PEPFAR program data, publications, research results, evaluation outcomes, surveys, and surveillance results have, in some cases, been used to influence country-level policies, modify HIV/AIDS interventions and program implementation, and contribute to national- and global-level reports.

Despite the success and progress described above, significant gaps remain in PEPFAR’s knowledge management approaches, and more progress is needed to address challenges in the areas of knowledge use, knowledge transfer within PEPFAR, and knowledge dissemination external to PEPFAR, because much of the knowledge acquired and created in PEPFAR does not

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

seem to be routinely disseminated and used. A fundamental gap identified by the committee is that PEPFAR lacks a conceptual framework for knowledge management that articulates the vision, goals, and role of knowledge within PEPFAR and that details what knowledge will be needed in the short and long term; how knowledge acquired and created in PEPFAR will be transferred internal to PEPFAR; how the knowledge should be used; how it will be disseminated beyond PEPFAR; and how the different activities of program monitoring, evaluation, and research will be used in a complementary manner to achieve goals. Articulating this comprehensive framework will require addressing current challenges related to program monitoring, research and evaluation, and knowledge transfer and knowledge dissemination, but it will lead to a more strategic and efficient approach in the future. PEPFAR’s largest and most sustained effort for generating data is the PEPFAR-specific program monitoring data collection system. The need to quickly measure results at the outset of PEPFAR contributed to this system developing in parallel to partner country M&E systems, but over time OGAC has modified the system, working closely with global partners in the HIV/AIDS response and partner country governments, to increasingly harmonize indicators and alignment with partner country HIV/AIDS monitoring and evaluation systems. However, the system requires the collection and reporting of a large amount of program monitoring data, the majority of which does not seem to be routinely used.

The use of the program monitoring data is limited by lack of utility of some of the PEPFAR indicators, technological challenges, reporting burden, and a lack of indicator harmonization. Additionally, for a period of time PEPFAR lacked a central database from which to report, manage, and disseminate program monitoring data, which limited access to and use of the data. Finally, PEPFAR reporting requirements place a large administrative burden on implementing partners and mission teams, which detracts from their ability to analyze and use data. Further modifications are needed to improve harmonization with global indicators and to better align with partner country systems in order to further reduce the number of PEPFAR-specific indicators, reduce reporting burden, increase data use by PEPFAR partner countries, and, ultimately, contribute to country-owned HIV/AIDS responses.

Furthermore, PEPFAR indicators do not capture sufficient information on all of PEPFAR’s stated priorities, goals, and activities. PEPFAR’s indicators, like many program monitoring systems, are focused primarily on outputs, which serve an important function in monitoring the implementation of activities but do not reflect quality, efficiency, and effectiveness. Measuring program progress and effectiveness in achieving desired outcomes is not always best achieved through program monitoring systems; this is also largely true for areas of increasing emphasis in PEPFAR, such

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

as technical assistance, capacity building, systems strengthening, sustainability, and country ownership, that are currently not well captured in existing knowledge management efforts. Therefore, strategically targeted and well-coordinated evaluation and research are critical complementary activities for assessing meaningful outcomes and for continual improvement to maximize the effectiveness and impact of PEPFAR investments. Despite recent efforts to strengthen PEPFAR research and evaluation activities and to develop mechanisms for internal knowledge transfer, challenges remain. Defining appropriate and allowable research activities remains a challenge; research gaps exist both across the whole of PEPFAR-supported programs and in some PEPFAR-supported countries. In addition, establishing formal mechanisms to transfer experiences (e.g., lessons learned, best practices, innovations, and models) across countries, implementing partners, and implementing sites would make it possible to more systematically synthesize and capitalize on best practices and internal lessons learned.

PEPFAR has made progress in disseminating knowledge external to PEPFAR, but more progress is needed to maximize the impact of knowledge created in PEPFAR, particularly by sharing knowledge with partner country governments and other partners involved in the HIV/AIDS response, tracking and measuring PEPFAR’s contribution to the global knowledge base, and increasing the availability of data collected with PEPFAR funds (routinely collected and reported program monitoring and other data from implementing partners and contractors, evaluation outcomes, and research results) for use by external researchers, evaluators, and other interested parties. The committee identified a number of challenges related to PEPFAR’s sharing of information with partner country governments, including a lack of financial transparency, different fiscal years, a lack of harmonization between PEPFAR and partner country reporting systems, and a lack of routine systems for sharing. The lack of formal mechanisms for sharing limits knowledge exchange between PEPFAR and other partners in the HIV/AIDS response.

Additionally, PEPFAR lacks a comprehensive central repository or tool with which to track and make available PEPFAR-funded research and evaluation activities, reports, and other dissemination products (e.g., publications, abstracts, tools, and guidelines) to both internal stakeholders and the international HIV/AIDS community. The absence of a centralized system makes it difficult to assess PEPFAR’s contribution to the global knowledge base and to use knowledge fully for maximum impact. Finally, PEPFAR does not have a data-sharing policy in place that promotes access to data collected by implementing partners, mission teams, and contractors with PEPFAR support. As a result, PEPFAR stakeholders collect an immense amount of data with PEPFAR funds that are not readily available to outside researchers, evaluators, and the global HIV/AIDS community.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

It will be critical for PEPFAR to evolve its knowledge management approaches and systems so that knowledge acquired and created through PEPFAR is useful, accessible, and used by PEPFAR stakeholders to monitor, inform, and improve the performance, effectiveness, and efficiency of efforts supported by PEPFAR. The need to adapt to changing circumstances and requirements with a new knowledge management framework is particularly important as the model for PEPFAR implementation is evolving toward supporting country-led responses in partner countries, with a transition to less emphasis on direct support for delivery of services and programs and more support and technical assistance for systems strengthening, capacity building, and sustainable management of the response by partner country stakeholders. With this transition, PEPFAR’s approach to knowledge management must also be transformed so that it can assess its own efforts going forward. PEPFAR needs to invest now in developing reliable, credible approaches to assessing the effectiveness of efforts beyond support for service delivery. Importantly, with this shift, the ability to attribute results by counting the services provided or the beneficiaries reached will be diminished, and direct attribution will no longer be an appropriate expectation for accountability. Instead, PEPFAR can seize this opportunity to be forward-looking and to work with others in the global health and development assistance communities to develop appropriate and credible ways to assess contributions to the improved performance and effectiveness of national efforts. Support for epidemiological data collection through surveillance and special studies in partner countries has been a cornerstone of PEPFAR’s contribution and should continue to be a critical component of knowledge management activities to support joint planning with partner countries.

Additionally, as one of the largest funders addressing the global HIV/AIDS epidemic, PEPFAR has both the capacity and the responsibility to play a significant leadership role in ensuring that knowledge created through the HIV/AIDS response is widely disseminated and available to outside partners, researchers, evaluators, and the public to spur innovation, accelerate the pace of knowledge creation, and maximize the public health impact of interventions.

Overall Conclusion: PEPFAR has made progress in managing knowledge and learning by developing systems for data creation and collection, streamlining program-monitoring data collection, advancing PEPFAR’s role and approach to evaluation and research, and using a wide variety of mechanisms to transfer knowledge. Yet, like other entities involved in the global HIV/AIDS response, it struggles with creating, acquiring, and transferring the right knowledge at the appropriate scale and in a manner that facili-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

tates use. PEPFAR has the potential to lead the global HIV/AIDS community in knowledge management by creating and following a conceptual framework that articulates the vision, purposes, intended audiences, and goals of knowledge; how knowledge will be acquired, created, transferred, used, and disseminated to achieve these goals; and the complementary roles of program monitoring, evaluation, and research in achieving these goals. PEPFAR has the opportunity to optimize program efficiency and effectiveness through an improved strategy that (1) streamlines and focuses knowledge creation within PEPFAR, (2) increases the acquisition of knowledge external to PEPFAR, (3) improves the efficiency and effectiveness of knowledge transfer within and external to PEPFAR, and (4) institutionalizes the use of knowledge to improve the way work is accomplished.

A clearer conceptual framework that incorporates the core elements of knowledge management (illustrated in Figure 11-9) combined with purposeful planning and implementation of PEPFAR’s monitoring, evaluation, research, and dissemination efforts would allow for the strategic allocation of limited personnel, time, and financial resources while reducing the burden of collecting and reporting data and other information that is not useful. Achieving a comprehensive strategy will require

1.  identifying what knowledge PEPFAR needs, in the short and long term, to inform, plan, monitor, evaluate, and improve efforts;

2.  determining which internal and external stakeholders need to know what information, at what level of the PEPFAR operational infrastructure, covering what scope of PEPFAR’s efforts, and with what frequency;

3.  planning which knowledge will be acquired from outside PEPFAR and which knowledge will be created though PEPFAR and mapping these knowledge needs to appropriately matched monitoring, evaluation, or research efforts;

4.  establishing mechanisms to transfer and disseminate knowledge; and

5.  determining the appropriate amount of personnel, time, and financial resources to devote to knowledge management in the context of the strategic use of the overall PEPFAR investment, and also determining how best to allocate these resources to find an appropriate balance among the generation, use, and dissemination of knowledge.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

RECOMMENDATIONS

Recommendation 11-1: The Office of the U.S. Global AIDS Coordinator (OGAC) should develop a comprehensive knowledge management framework, including a program monitoring and evaluation strategy, a prioritized and targeted research portfolio, and systems for knowledge dissemination. This framework should adapt to emerging needs to assess PEPFAR’s models of implementation and contribution to sustainable management of the HIV response in partner countries.

This knowledge management framework will require that PEPFAR implement and strategically allocate resources for the following:

A.   To better document PEPFAR’s progress and effectiveness, OGAC should refine its program monitoring and evaluation strategy to streamline reporting and to strategically coordinate a complementary portfolio of evaluation activities to assess outcomes and effects that are not captured well by program monitoring indicators. Efforts should support innovation in methodologies and measures where needed. Both monitoring and evaluation should be specifically matched to clearly articulated data sources, methods, and uses at each level of PEPFAR’s implementation and oversight.

B.   To contribute to filling critical knowledge gaps that impede effective and sustainable HIV programs, OGAC should continue to redefine permitted research within PEPFAR by developing a prioritized portfolio with articulated activities and methods. The planning and implementation process at the country and program level should inform and be informed by the research portfolio, which should focus on research that will improve the effectiveness, quality, and efficiency of PEPFAR-supported activities and will also contribute to the global knowledge base on implementation of HIV/AIDS programs.

C.   To maximize the use of knowledge created within PEPFAR, OGAC should develop systems and processes for routine, active transfer and dissemination of knowledge both within and external to PEPFAR. As one component, OGAC should institute a data-sharing policy, developed through a consultative process. The policy should identify the data to be included and ensure that these stipulated data and results generated by PEPFAR or through PEPFAR-supported activities are made available in a timely manner to PEPFAR stakeholders, external evaluators, the research community, and other interested parties.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
STEP ACTION
Acquire
Acquire knowledge from outside PEPFAR
Plan which knowledge will be acquired from outside PEPFAR. Acquire knowledge from the global knowledge base (partner country surveys and surveillance systems, global monitoring systems, research and evaluation, and experiences of other partners and stakeholders).
Create
Create knowledge within PEPFAR
Plan which knowledge will be created within PEPFAR. Create knowledge through tacit experiences, routine program monitoring, frequent program evaluation, periodic outcome and impact evaluations at different levels of PEPFAR implementation, and targeted, purposeful research activities.
Synthesize
Synthesize diverse knowledge streams
Across all levels of PEPFAR, perform iterative syntheses of acquired and created knowledge to inform decision making and action.
Transfer
Transfer acquired and created knowledge within PEPFAR
Determine which knowledge will be transferred within PEPFAR and at which levels. Plan, develop and establish processes and mechanisms to transfer knowledge throughout PEPFAR.
Disseminate
Disseminate knowledge external to PEPFAR
Determine which knowledge will be disseminated external to PEPFAR and at which levels. Plan, develop and establish processes and mechanisms to ensure wide dissemination.
Use
Institutionalize the use of knowledge to maximize outcomes
Use knowledge at the implementing partner, country, agency, OGAC and global levels to improve the global response to HIV/AIDS.
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

         and ensure that these stipulated data and results generated by PEPFAR or through PEPFAR-supported activities are made available in a timely manner to PEPFAR stakeholders, external evaluators, the research community, and other interested parties.

The following sections describe additional considerations for implementing each of the components of this recommendation.

Further considerations for implementation of Recommendation 11-1A: Program monitoring and evaluation

•   OGAC’s current tiered program monitoring indicator reporting structure (illustrated in Figure 11-10) should be further streamlined to report upward only those indicators essential at each PEPFAR level:

o   Tier 1: A small set of core indicators, fewer than the current 25, to be reported to central HQ level. These data should be used to monitor performance across PEPFAR as a whole, for congressional reporting, and to document trends; as such, these indicators should remain consistent over time. Whenever possible and appropriate, these indicators should be harmonized with existing global indicators and national indicators; therefore, some centrally reported indicators will reflect PEPFAR’s contribution rather than aim to measure direct attribution.

o   Tier 2: A larger menu of indicators defined in OGAC guidance, from which a subset are selected for their applicability to country programs to be reported by implementing partners to the U.S. mission teams but not routinely reported to HQ. These data should be used to monitor the effectiveness of the in-country response and to support mutual accountability with partner countries and their citizens. These data could be considered for occasional centralized use to inform special studies or respond to congressional requests but aggregation and comparability across countries may be limited in this tier because all mission teams may not collect the same data.

o   Tier 3: Indicators selected by implementing partners to monitor and manage program implementation and effectiveness that are not routinely reported to mission teams. Implementing partners should select appropriate indicators defined in OGAC guidance

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

     and augment these with other indicators as needed for their programs. Implementing partners should work with mission teams in developing their program monitoring plans with selected indicators. Mission teams should provide oversight and technical assistance to ensure implementation of these plans and to promote local quality data collection, use, and mutual accountability. Although not routinely reported, some of these data could be considered for occasional country-level and centralized use.

o   OGAC should create mechanisms for implementing partners, mission teams, and agency HQ to mutually contribute to a periodic review across all tiers of indicator development, applicability, and utility and to make modifications if necessary.

o   Tier 1 indicators should be harmonized whenever possible and appropriate with existing global indicators and national indicators. For indicators that are not routinely reported centrally (Tiers 2 and 3), country program planning should facilitate alignment of indicator selection and data collection with partner country HIV monitoring and health information systems.

•   OGAC should complement program monitoring with a unified evaluation portfolio that includes periodic program evaluation at the PEPFAR country program and implementing partner levels to assess process, progress, and outcomes as well as periodic impact evaluations at the country, multi-country, and HQ levels.

o   OGAC evaluation guidance should provide information about prioritizing areas for evaluation, the types of evaluation questions, methodological guidance, potential study designs, template evaluation plans, examples of key outcomes, and how evaluation results should be used and disseminated. PEPFAR should support a range of appropriate methodologies for program evaluation, including mixed qualitative and quantitative methods, and should shift emphasis from probability designs to plausibility designs that provide valid evidence of impact.

o   To allow for some comparability across countries and programs, OGAC and HQ technical working groups should, with input from country teams, strategically plan and coordinate a subset of evaluations within programmatic areas that include (but are not limited to) a minimum set of centrally identified and defined outcome measures and methodologies.

o   Within PEPFAR-supported evaluation activities there should be an emphasis on the use of in-country local expertise to en-

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

     hance capacity building for program evaluation and contribute to country ownership.

•   For both program monitoring and evaluation OGAC should continue its work on defining and developing measures to assess progress in the currently under-measured areas of country ownership, sustainability, gender, policy, capacity building, and technical assistance.

Further considerations for implementation of Recommendation 11-1B: Research

•   OGAC should clearly define which activities and methodologies will be included under the umbrella of PEPFAR-supported research as distinguished from program evaluation.

•   OGAC should draw on input from implementing agencies, mission teams, partner countries, implementing partners, the SAB, and other experts to identify and articulate research priorities and appropriate research methodologies. The research proposals and funding mechanisms should be designed to ensure that these priorities are met and that methodologies are applied through RFAs and other investigator-driven research proposals as well as through targeted solicitations of research in gap areas not met through open requests.

•   Given PEPFAR’s legislative and programmatic objectives to support research that assesses program quality, effectiveness, and population-based impact; optimizes service delivery; and contributes to the global evidence base on HIV/AIDS interventions and program implementation, at the time of this evaluation the committee identified the following gaps in PEPFAR’s research activities:

o   Behavioral and structural interventions, especially in areas such as prevention, gender, nonclinical and OVC care and support, and treatment retention and adherence. These research activities should employ appropriate methodologies and study designs without being unduly limited to random assignment designs.

o   Costs, benefits, and feasibility of integrating gender-focused programs with clinical and community-based activities.

o   Health systems strengthening interventions across the WHO building blocks, with a prioritized goal of determining setting-and system-specific feasibility, effectiveness, quality of services, and costs for innovative models.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

o   To contribute to country ownership, PEPFAR should facilitate in-country local participation and research capacity building through simplified, streamlined, and transparent application and review processes that encourage submissions from country-based implementing partners and researchers.

Further considerations for implementation of Recommendation 11-1C: Knowledge transfer and dissemination

•   The knowledge created within PEPFAR that should be more widely documented and disseminated includes program monitoring data, financial data, research results, evaluation outcomes, best practices, and informal knowledge such as implementation experience and lessons learned.

•   To institutionalize internal and external knowledge transfer and learning, PEPFAR should develop appropriate systems and processes for the most needed types and scale of knowledge transfer. To achieve this, PEPFAR should draw on broad stakeholder input to assess the strengths and weaknesses in current processes and to identify needs and opportunities for improved knowledge transfer.

•   PEPFAR should invest in innovative mechanisms and technology to facilitate knowledge transfer across partner countries and implementing partners. Mechanisms currently used successfully on a small scale and an ad hoc basis could be formally scaled up across PEPFAR. OGAC should also look to other organizations with wide geographic reach and organizational complexity, such as multi-country PEPFAR implementing partners, other large global health initiatives, and global corporations, for models of successful knowledge transfer systems.

•   OGAC should develop a policy for data sharing and transparency that facilitates timely access to PEPFAR-created knowledge for analysis and evaluation. The purpose of this policy would be to ensure that, within a purposefully and reasonably defined scope, specified program monitoring data and financial data, evaluation outcomes, and research data and results generated with PEPFAR support by contractors, grantees, mission teams, and USG agencies be made available to the public, research community, and other external stakeholders. OGAC and the PEPFAR implementing agencies should consult with both internal and external parties who would be affected by this policy to help identify the data that are most critical for external access and that can be reasonably subject

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

      to data-sharing requirements, as well as to help develop feasible mechanisms to implement a data-sharing policy.

o   For routinely collected financial and program monitoring data, a limited set of essential data should be identified and made available for external use in a timely way.

o   Evaluation and research reports and publications using data collected through PEPFAR-supported programs should be tracked and made available in a publicly accessible central repository. USG agencies with similar repositories can be considered as models.

o   For research data and other information that is expressly generated for new knowledge, the policy should respect time-bound exclusivity for the right to engage in the publication process, yet also ensure the timely availability of data, regardless of publication, for access and use by external evaluators and researchers. OGAC should look to USG agencies with similar research data policies as models.

o   In developing the policy and specifying the scope of data to be included, several key factors and potential constraints that can affect the implementation of the policy will need to be addressed. These include patient and client information confidentiality; the financial resources, personnel, and time needed to make data available; and issues of data ownership, especially in the context of increasing responsibility in partner countries and the provision of PEPFAR support through country systems or through activities and programs supported by multiple funding streams.

REFERENCES

AIDStar-Two. 2012. OVCsupport.Net—a global hub on children and HIV. http://www.ovcsupport.net/s (accessed October 5, 2012).

Bergmann, H. 2011. Field driven learning meeting: Linkages to and retention in HIV care and support programs. Arlington, VA: USAID’s AIDS Support and Technical Assistance Resources, (AIDStar-One), Task Order 1.

Bouey, P., and N. Padian. 2011. PEPFAR-funded evaluations presentation. Presentation at the PEPFAR Scientific Advisory Board Meeting, Washington, DC.

Bryant, M., J. Beard, L. Sabin, M. I. Brooks, N. Scott, B. A. Larson, G. Biemba, and C. Miller. 2012. PEPFAR’s support for orphans and vulnerable children: Some beneficial effects, but too little data, and programs spread thin. Health Affairs 31(7):1508-1518.

Dentzer, S. 2012. Assessing the President’s Emergency Plan for AIDS Relief. Health Affairs 31(7).

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Donnelly, J. 2012. The President’s Emergency Plan for AIDS Relief: How George W. Bush and aides came to “think big” on battling HIV. Health Affairs 31(7):1389-1396.

Fain, J. 2005. Editorial: Is there a difference between evaluation and research? Diabetes Educator 31(2):150-155.

Fullem, A., M. Levy, and M. Sharer. 2012. Meeting the HIV; maternal, newborn, and child health; and social support needs of mothers and their young children. Field driven learning meeting, Addis Ababa, Ethiopia, November 8 to 10, 2011. Arlington, VA: AIDStar-One, Task Order 1.

GAO (U.S. Government Accountability Office). 2011a. PEPFAR program planning and reporting. GAO-11-785. Washington, DC: GAO.

GAO. 2011b. Performance measurement and evaluation: Definitions and relationships. Washington, DC: GAO.

GAO. 2012. President’s Emergency Plan for AIDS Relief—agencies can enhance evaluation quality, planning, and dissemination: Report to congressional committees. Washington, DC: GAO.

Garvin, D. A. 1993. Building a learning organization. Harvard Business Review 71(4):78-91.

Gay, J., M. Croce-Galis, and K. Hardee. 2012. What works for women and girls: Evidence for HIV/AIDS interventions. 2nd edition. www.whatworksforwomen.org (accessed October 5, 2012).

Global Fund (Global Fund to Fight AIDS, Tuberculosis, and Malaria). 2011. Monitoring and evaluation toolkit: HIV, tuberculosis, malaria and health and community systems strengthening. Part 2: HIV. Geneva: Global Fund.

Goosby, E. 2012. The President’s Emergency Plan for AIDS Relief: Marshalling all tools at our disposal toward an AIDS-free generation. Health Affairs 31(7):1593-1598.

Grosso, A. L., K. Hoan Tram, O. Ryan, and S. Baral. 2012. Countries where HIV is concentrated among most-at-risk populations get disproportionally lower funding from PEPFAR. Health Affairs 31(7):1519-1528.

Habicht, J. P., C.G. Victoria, J.P. Vaughan. 1999. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. International Journal of Epidemiology (28):10-18.

HHS (U.S. Department of Health and Human Services). 2012. Operations research (implementation science) for strengthening program implementation through the President’s Emergency Plan for AIDS Relief (PEPFAR). http://www.grants.gov/search/search.do?mode=VIEW&oppId=136553 (accessed October 5, 2012).

Ho, D. D., P. A. Volberding, and W. A. Blattner. 2012. The United States President’s Emergency Plan for AIDS Relief (PEPFAR): Its vision, achievements, and new directions. Journal of Acquired Immune Deficiency Syndromes 60(Suppl 3).

Holmes, C. 2012. Presentation to SAB: Implementation science updates. Washington, DC: OGAC.

Holmes, C. B., J. M. Blandford, N. Sangrujee, S. R. Stewart, A. DuBois, T. R. Smith, J. C. Martin, A. Gavaghan, C. A. Ryan, and E. P. Goosby. 2012. PEPFAR’s past and future efforts to cut costs, improve efficiency, and increase the impact of global HIV programs. Health Affairs 31(7):1553-1560.

Holzscheiter, A., G. Walt, and R. Brugha. 2012. Monitoring and evaluation in global HIV/AIDS control-weighing incentives and disincentives for coordination among global and local actors. Journal of International Development 24(1):61-76.

IHME (Institute for Health Metrics and Evaluation). 2011. Financing global health 2011: Continued growth as MDG deadline approaches. Seattle: IHME.

IOM (Institute of Medicine). 2007a. PEPFAR implementation: Progress and promise. Washington, DC: The National Academies Press.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

IOM. 2007b. Preventing HIV infection among injecting drug users in high risk countries: An assessment of the evidence. Washington, DC: The National Academies Press.

IOM. 2011. IOM staff internal review of PEPFAR phase I evaluation—list of targeted evaluations document. Washington, DC: IOM.

IOM and NRC (National Research Council). 2010. Strategic approach to the evaluation of programs implemented under the Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008. Washington, DC: The National Academies Press.

JSI (John Snow International). 2012. AIDStar-One: AIDS support and technical assistance resources. http://www.aidstar-one.com (accessed October 5, 2012).

Levin-Rozalis, M. 2003. Evaluation and research: Differences and similarities. Canadian Journal of Program Evaluation 18(2):1-31.

Loermans, J. 2002. Synergizing the learning organization and knowledge management. Journal of Knowledge Management 6(3):285-294.

Lyerla, R., C. S. Murrill, P. D. Ghys, J. M. Calleja-Garcia, and K. M. Decock. 2012. The use of epidemiological data to inform the PEPFAR response. Journal of Acquired Immune Deficiency Syndromes 60(Suppl 3):S57-S62.

Needle, R., J. Fu, C. Beyrer, V. Loo, A. S. Abdul-Quader, J. A. McIntyre, Z. Li, J. Mbwambo, M. Muthui, and B. Pick. 2012. PEPFAR’s evolving HIV prevention approaches for key populations—people who inject drugs, men who have sex with men, and sex workers: Progress, challenges, and opportunities. Journal of Acquired Immune Deficiency Syndromes 60(Suppl 3):S145-S151.

NIH (National Institutes of Health). 2010. Limited competition: Administrative supplements for HIV/AIDS implementation science in PEPFAR settings. http://grants.nih.gov/grants/guide/notice-files/NOT-AI-10-023.html (accessed 2012).

NIH. 2011. NIH/PEPFAR collaboration for implementation science and impact evaluation (R01). http://grants.nih.gov/grants/guide/rfa-files/RFA-AI-11-003.html (accessed October 5, 2012).

Obama, B. 2011. Remarks by the President on World AIDS Day, December 1, Washington, DC.

Office of Learning, Evaluation and Research. 2012. USAID evaluation policy: Year one; first annual report and plan for 2012 and 2013. Washington, DC: USAID.

OGAC (Office of the U.S. Global AIDS Coordinator). 2003. PEPFAR country operational plan guidelines for FY04. Washington, DC: OGAC.

OGAC. 2004. The President’s Emergency Plan for AIDS Relief: U.S. five-year global HIV/AIDS strategy. Washington, DC: OGAC.

OGAC. 2005a. Emergency Plan for AIDS Relief Fiscal Year 2005 operational plan: June 2005 update. Washington, DC: OGAC.

OGAC. 2005b. President’s Emergency Plan for AIDS Relief: FY06 country operational plan final guidance. Washington, DC: OGAC.

OGAC. 2005c. The President’s Emergency Plan for AIDS Relief: Indicators, reporting requirements, and guidelines for focus countries. Washington, DC: OGAC.

OGAC. 2006a. A blueprint for public health evaluations in the President’s Emergency Plan for AIDS Relief. Washington, DC: OGAC.

OGAC. 2006b. The President’s Emergency Plan for AIDS Relief. FY2007 supplemental COP guidance resource guide. Washington, DC: OGAC.

OGAC. 2006c. The President’s Emergency Plan for AIDS Relief: FY2007 country operational plan guidance. Washington, DC: OGAC.

OGAC. 2006d. The U.S. President’s Emergency Plan for AIDS Relief fiscal year 2006: Operational plan. 2006 August update. Washington, DC: OGAC.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

OGAC. 2007a. 2007 HIV/AIDS implementers meeting: Scaling up through partnerships; program and abstract book.Washington, DC, OGAC.

OGAC. 2007b. Data quality assurance tool for program-level indicators. Washington, DC, OGAC.

OGAC. 2007c. Factsheet: PEPFAR extranet (PEPFAR.net), edited by OGAC. Washington, DC: U.S. DoS, USAID, DoD, Department of Commerce, DoL, HHS, Peace Corps.

OGAC. 2007d. The power of partnerships: The President’s Emergency Plan for AIDS Relief, third annual report to Congress. Washington, DC: OGAC.

OGAC. 2007e. The President’s Emergency Plan for AIDS Relief: FY2008 country operational plan guidance. Washington, DC: OGAC.

OGAC. 2007f. The President’s Emergency Plan for AIDS Relief: Indicators, reporting requirements, and guidelines. Indicators reference guide: FY2007 reporting/FY2008 planning. Washington, DC: OGAC.

OGAC. 2007g. The U.S. President’s Emergency Plan for AIDS Relief fiscal year 2007: Operational plan. 2007 June update. Washington, DC: OGAC.

OGAC. 2008a. 2008 HIV/AIDS implementers’ meeting: Scaling up through partnerships—overcoming obstacles to implementation, program and abstract book. Washington, DC: OGAC.

OGAC. 2008b. The President’s Emergency Plan for AIDS Relief: FY2009 country operational plan guidance. Washington, DC: OGAC.

OGAC. 2008c. The U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2008: PEPFAR operational plan. Washington, DC: OGAC.

OGAC. 2009a. 2009 HIV/AIDS implementers’ meeting program and abstract book: Optimizing the response, partnerships for sustainability. Washington, DC: OGAC.

OGAC. 2009b. Guidance for PEPFAR partnership frameworks and partnership framework implementation plans. Version 2.0. Washington, DC: OGAC.

OGAC. 2009c. The President’s Emergency Plan for AIDS Relief: FY2010 country operational plan guidance. Washington, DC: OGAC.

OGAC. 2009d. The President’s Emergency Plan for AIDS Relief: FY2010 country operational plan guidance: Programmatic considerations. Washington, DC: OGAC.

OGAC. 2009e. The President’s Emergency Plan for AIDS Relief: Next generation indicators reference guide. Version 1.1. Washington, DC: OGAC.

OGAC. 2009f. The U.S. President’s Emergency Plan for AIDS Relief: Five-year strategy. Washington, DC: OGAC.

OGAC. 2010a. Comprehensive HIV prevention for people who inject drugs, revised guidance. Washington, DC: OGAC.

OGAC. 2010b. The President’s Emergency Plan for AIDS Relief: Public health evaluation (PHE) concept submission guidance. Washington, DC: OGAC.

OGAC. 2010c. The President’s Emergency Plan for AIDS Relief: FY2011 country operational plan guidance. Washington, DC: OGAC.

OGAC. 2010d. The U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2009: PEPFAR operational plan. November 2010. Washington, DC: OGAC.

OGAC. 2011a. Charter of the President’s Emergency Plan for AIDS Relief (PEPFAR) Scientific Advisory Board. Washington, DC: OGAC.

OGAC. 2011b. E-mail communication between OGAC staff and IOM: “Request from IOM outcome and impact evaluation of PEPFAR team.” Washington, DC: OGAC.

OGAC. 2011c. Guidance for the prevention of sexually trasmitted HIV infections. Washington, DC: OGAC.

OGAC. 2011d. List of continuing PHEs for FY 2012. Washington, DC: OGAC.

OGAC. 2011e. OGAC technical review of table—“Evolution of PEPFAR supported research activities.” Washington, DC: OGAC.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

OGAC. 2011f. PEPFAR response to data request from IOM for the PEPFAR II evaluation. Washington, DC: OGAC.

OGAC. 2011g. The President’s Emergency Plan for AIDS Relief: FY2012 country operational plan guidance Washington, DC: OGAC.

OGAC. 2011h. The President’s Emergency Plan for AIDS Relief: FY2012 country operational plan guidance appendices. Washington, DC: OGAC.

OGAC. 2011i. The U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2010: PEPFAR operational plan. Washington, DC: OGAC.

OGAC. 2011j. The U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2011: PEPFAR operational plan. Washington, DC: OGAC.

OGAC. 2011k. The U.S. President’s Emergency Plan for AIDS Relief: Seventh annual report to Congress. Washington, DC: OGAC.

OGAC. 2012a. Annual reports to congress on the President’s Emergency Plan for AIDS Relief. http://www.pepfar.gov/press/c19573.htm (accessed October 5, 2012).

OGAC. 2012b. E-mail communication between OGAC staff and IOM: “Clarification regarding number of NGIs.” Washington, DC: OGAC.

OGAC. 2012c. E-mail communication between OGAC staff and IOM: “Follow-up queries for IOM PEPFAR evaluation.” Washington, DC: OGAC.

OGAC. 2012d. Email communication between OGAC staff and IOM: “RE: Follow-up request: Partnership frameworks and implementation plans.” Washington, DC: OGAC.

OGAC. 2012e. E-mail communication between OGAC staff and IOM: “re: Request from IOM outcome and impact evaluation of PEPFAR team.” Washington, DC: OGAC.

OGAC. 2012f. PEPFAR Scientific Advisory Board. http://www.pepfar.gov/sab (accessed October, 2012).

OGAC. 2012g. PEPFAR Scientific Advisory Board membership list. http://www.pepfar.gov/sab/160067.htm (accessed October 30, 2012).

OGAC. 2012h. PEPFAR Scientific Advisory Board working groups. http://www.pepfar.gov/sab/workinggroups/index.htm (accessed October 30, 2012).

OGAC. 2012i. Reports: Annual reports to Congress and other reports to Congress. www.pepfar.gov/reports/progress/index.htm (accessed October 30, 2012).

OGAC. 2012j. The United State’s President’s Emergency Plan for AIDS Relief. http://www.pepfar.gov (accessed October 30, 2012).

OGAC SAB Data Working Group. 2011. SAB Data Working Group (DWG) conference call notes. http://www.pepfar.gov/documents/organization/166969.pdf (accessed October 30, 2012).

Padian, N. S., C. B. Holmes, S. I. McCoy, R. Lyerla, P. D. Bouey, and E. P. Goosby. 2011. Implementation science for the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). Journal of Acquired Immune Deficiency Syndromes 56(3):199-203.

Porter, L. E., P. D. Bouey, S. Curtis, M. Hochgesang, P. Idele, B. Jefferson, W. Lemma, R. Myrick, H. Nuwagaba-Biribonwoha, D. Prybylski, Y. Souteyrand, and T. Tulli. 2012. Beyond indicators: Advances in global HIV monitoring and evaluation during the PEPFAR era. Journal of Acquired Immune Deficiency Syndromes 60(Suppl 3):S120-S126.

Reyes, M. 2009. Summary of PEPFAR State of the Program Area (SOPA): Care and support. http://caps.ucsf.edu/uploads/pubs/presentations/pdf/PEPFARCareandSupportSOPASummary.pdf (accessed January 25, 2013)

Simonds, R. J., C. A. Carrino, and M. Moloney-Kitts. 2012. Lessons from the President’s Emergency Plan for AIDS relief: From quick ramp-up to the role of strategic partnership. Health Affairs 31(7):1397-1405.

Small, P. 2012. Four differences between research and program evaluation. Blog: NonProfit Capacity Building, Authenticity Consulting, LLC.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×

Spratt, K., and Heather Bergmann. 2011. Strengthening gender programming in PEPFAR: Technical exchange of best practices, program models and resources. Arlington, VA: AIDStar-One, Task Order 1.

Stufflebeam, D. 2007. CIPP evaluation model checklist: A tool for applying the CIPP model to assess long-term enterprises. Kalamazoo, MI: Western Michigan University.

Swan, J., H. Scarbrough, and J. Preston. 1999. Knowledge management: The next fad to forget people. Paper read at 7th European Conference on Information Systems, Copenhagen.

Thomas, J. C., S. Curtis, and J. B. Smith. 2011. The broader context of implementation science. Journal of Acquired Immune Deficiency Syndromes 58(1):e19-e21.

UNAIDS (Joint United Nations Programme on HIV/AIDS). 2004. Consultation on harmonization of international AIDS funding, end-of-meeting agreement, Washington consulation of April 25, 2004. Geneva: UNAIDS.

UNAIDS. 2007. Practical guidelines for intensifying HIV prevention: Towards universal access. Geneva: UNAIDS.

UNAIDS. 2009. Monitoring the declaration of commitment on HIV/AIDS: Guidelines on construction of core indicators: 2010 reporting. Geneva: UNAIDS.

USAID (U.S. Agency for International Development). 2011a. Implementation science research to support programs under the President’s Emergency Plan for AIDS Relief (PEPFAR). http://www.grants.gov/search/search.do;jsessionid=w6lxTG8QGvxLLXk0BZbXcSl2FrTyCGQLLPcZV3qXrvyyvsCLFnkC!-25395513?oppId=114673&mode=VIEW (accessed October 30, 2012).

USAID. 2011b. USAID’s implementation science investment: Improving HIV/AIDS programming through the translation of research to practice. Washington, DC: USAID.

USAID. 2012. Development experience clearinghouse. https://dec.usaid.gov/dec/home/Default.aspx (accessed October 5, 2012).

Walport, M. and P. Breast. 2011. Sharing research data to improve public health. Lancet 377(9765):537-539.

WHO (World Health Organization). 2005. World Health Organization knowledge management strategy. Geneva: WHO.

WHO. 2012. Voluntary medical male circumcision for HIV prevention. http://www.who.int/hiv/topics/malecircumcision/fact_sheet/en/index.html (accessed September 20, 2012).

World Bank. 2003. Sharing knowledge: Innovations and remaining challenges. Washington, DC: World Bank.

Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 609
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 610
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 611
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 612
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 613
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 614
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 615
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 616
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 617
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 618
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 619
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 620
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 621
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 622
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 623
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 624
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 625
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 626
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 627
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 628
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 629
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 630
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 631
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 632
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 633
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 634
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 635
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 636
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 637
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 638
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 639
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 640
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 641
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 642
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 643
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 644
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 645
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 646
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 647
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 648
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 649
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 650
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 651
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 652
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 653
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 654
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 655
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 656
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 657
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 658
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 659
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 660
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 661
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 662
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 663
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 664
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 665
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 666
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 667
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 668
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 669
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 670
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 671
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 672
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 673
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 674
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 675
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 676
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 677
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 678
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 679
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 680
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 681
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 682
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 683
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 684
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 685
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 686
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 687
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 688
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 689
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 690
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 691
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 692
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 693
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 694
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 695
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 696
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 697
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 698
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 699
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 700
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 701
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 702
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 703
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 704
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 705
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 706
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 707
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 708
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 709
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 710
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 711
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 712
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 713
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 714
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 715
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 716
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 717
Suggested Citation:"11 PEPFAR's Knowledge Management." Institute of Medicine. 2013. Evaluation of PEPFAR. Washington, DC: The National Academies Press. doi: 10.17226/18256.
×
Page 718
Next: Appendixes »
Evaluation of PEPFAR Get This Book
×
Buy Paperback | $98.00 Buy Ebook | $79.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The U.S. government supports programs to combat global HIV/AIDS through an initiative that is known as the President's Emergency Plan for AIDS Relief (PEPFAR). This initiative was originally authorized in the U.S. Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 2003 and focused on an emergency response to the HIV/AIDS pandemic to deliver lifesaving care and treatment in low- and middle-income countries (LMICs) with the highest burdens of disease. It was subsequently reauthorized in the Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008 (the Lantos-Hyde Act).

Evaluation of PEPFAR makes recommendations for improving the U.S. government's bilateral programs as part of the U.S. response to global HIV/AIDS. The overall aim of this evaluation is a forward-looking approach to track and anticipate the evolution of the U.S. response to global HIV to be positioned to inform the ability of the U.S. government to address key issues under consideration at the time of the report release.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!