11

PEPFAR’s Knowledge Management

INTRODUCTION

Knowledge is generated across all levels of PEPFAR including implementing partners, partner country mission teams, PEPFAR implementing agencies, and Office of the U.S. Global AIDS Coordinator (OGAC) headquarters. This knowledge, if appropriately synthesized, transferred, disseminated, shared, and used, has the potential not only to contribute to program improvement and the sustainability of PEPFAR’s efforts, but also to help the global community in its response to the HIV/AIDS epidemic. As the largest donor currently addressing the global HIV/AIDS epidemic, PEPFAR has both the ability and the responsibility to play a significant leadership role in this realm (IHME, 2011).

Knowledge management has been defined by Swan and colleagues as “any process or practice of creating, acquiring, capturing, sharing and using knowledge wherever it resides, to enhance learning and performance in organizations” (Swan et al., 1999, p. 669). Knowledge management is a strategy used by many organizations to harness and respond to both existing and created knowledge and has been adopted by organizations such as the World Bank and the World Health Organization (WHO) (Loermans, 2002; WHO, 2005; World Bank, 2003).

An organization that is skilled in knowledge management is able to efficiently and effectively manage knowledge that has been created (Loermans, 2002). As discussed in this chapter, the types of knowledge PEPFAR has created and utilized include developing a system for collecting extensive



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 609
11 PEPFAR’s Knowledge Management INTRODUCTION Knowledge is generated across all levels of PEPFAR including imple- menting partners, partner country mission teams, PEPFAR implement- ing agencies, and Office of the U.S. Global AIDS Coordinator (OGAC) headquarters. This knowledge, if appropriately synthesized, transferred, disseminated, shared, and used, has the potential not only to contribute to program improvement and the sustainability of PEPFAR’s efforts, but also to help the global community in its response to the HIV/AIDS epidemic. As the largest donor currently addressing the global HIV/AIDS epidemic, PEPFAR has both the ability and the responsibility to play a significant leadership role in this realm (IHME, 2011). Knowledge management has been defined by Swan and colleagues as “any process or practice of creating, acquiring, capturing, sharing and us- ing knowledge wherever it resides, to enhance learning and performance in organizations” (Swan et al., 1999, p. 669). Knowledge management is a strategy used by many organizations to harness and respond to both exist- ing and created knowledge and has been adopted by organizations such as the World Bank and the World Health Organization (WHO) (Loermans, 2002; WHO, 2005; World Bank, 2003). An organization that is skilled in knowledge management is able to effi- ciently and effectively manage knowledge that has been created (Loermans, 2002). As discussed in this chapter, the types of knowledge PEPFAR has created and utilized include developing a system for collecting extensive 609

OCR for page 609
610 EVALUATION OF PEPFAR program monitoring data, supporting epidemiologic and surveillance activi- ties in partner countries, strengthening partner country health information systems, implementing various program evaluation approaches, supporting research, and the creation of both tacit and experiential knowledge as a result of program implementation. Evaluating PEPFAR’s knowledge management was not an explicit part of the committee’s congressional mandate, but because availability and access to information was key to every aspect of this evaluation, the com- mittee felt strongly that to help guide PEPFAR’s future efforts, examining and making recommendations regarding PEPFAR’s knowledge management approach was critical. The committee determined that PEPFAR has made strong efforts in generating knowledge, often at a level not seen in other development programs. Yet, as reflected in prior chapters of this report, there are key areas where the information needed to assess efforts and guide future activities is unavailable or insufficient. Significant gaps remain in PEPFAR’s knowledge management approach, especially in the realms of knowledge creation, dissemination, and utilization, and to date, OGAC has not articulated a clear and comprehensive strategy for managing knowledge to optimize PEPFAR’s performance and effectiveness. This chapter shifts its focus away from assessing and addressing the limitations in the available information that affected the committee’s ability to respond to the specific charge mandated by Congress; these were dis- cussed in Chapter 2. Rather, the aim of this chapter is to offer an assessment to guide PEPFAR to more strategically and efficiently meet its information needs going forward. This chapter will review and assess PEPFAR’s current approach to knowledge management, culminating with recommendations from the committee for future directions to address current gaps and to strengthen PEPFAR’s ability to generate, share, and utilize knowledge more effectively. Strategic Information PEPFAR articulated a goal of having evidence-based programs from the outset (OGAC, 2004). To meet this goal, the OGAC Office of Strategic Information (SI), which is responsible for using SI to guide and coordi- nate PEPFAR performance planning and reporting, was established (GAO, 2011a). The first Five-Year Strategy defined strategic information as “the systematic collection, analysis, and dissemination of information about reaching the Emergency Plan’s objectives, as well as the related program- matic activities funded to reach these goals” (OGAC, 2004, p. 73). Strategic information was used as an organizing concept because ‘WHO was just starting to use the term strategic information, and that resonated with us— the use of information for program improvement and operations—so, we

OCR for page 609
PEPFAR’S KNOWLEDGE MANAGEMENT 611 decided to use that name. Gathering of information—it had to be strategic and it had to be used’ (NCV-3).1,2 At the peak of funding around 2007–2008, the OGAC SI office had an annual budget of around $33 million for centrally funded SI activities, but over time funding has been reduced to less than $10 million annu- ally (NCV-2-USG). At the partner country level, from fiscal year (FY) 2006 to FY 2011, approximately 4 to 5 percent of total funding for PEPFAR partner country activities was budgeted for SI activities, excluding staff salaries (see Figure 11-1) (OGAC, 2005a, 2006d, 2007g, 2008d, 2010d, 2011i,j). These activities have included monitoring and reporting partner results, as well as surveillance, surveys, and efforts to strengthen partner country health information systems (OGAC, 2008b, 2009d, 2010c). Reflecting an increased focus on country ownership, FY 2012 Country Operational Plan (COP) guidance advised mission teams that activities planned under the SI budget code should aim “to build individual, institutional, and organi- zational capacity in country” for strategic information activities (OGAC, 2011h, p. 68). PROGRAM TARGETS AND PRIORITIES Setting Program Targets Setting priorities and targets is one important aspect of planning and managing programs. Subsequently monitoring and assessing progress and performance in meeting these targets is critical for program management. When PEPFAR was authorized in 2003,3 it was established with an em- phasis on accountability by setting specific performance targets and with a recognition of the necessity of monitoring and evaluation to assess the performance of PEPFAR-supported programs. The initial 5-year goals for 1  Single quotations denote an interviewee’s perspective with wording extracted from tran- scribed notes written during the interview. Double quotations denote an exact quote from an interviewee either confirmed by listening to the audio-recording of the interview or extracted from a full transcript of the audio-recording. 2  Country Visit Exit Synthesis Key: Country # + ES Country Visit Interview Citation Key: Country # + Interview # + Organization Type Non-Country Visit Interview Citation Key: “NCV” + Interview # + Organization Type Organization Types: United States: USG = U.S. Government; USNGO = U.S. Nongovernmental Organization; USPS = U.S. Private Sector; USACA = U.S. Academia; Partner Country: PCGOV = Partner Country Government; PCNGO = Partner Country NGO; PCPS = Partner Country Private Sector; PCACA = Partner Country Academia; Other: CCM = Country Coordinating Mechanism; ML = Multilateral Organization; OBL = Other (non-U.S. and non-Partner Coun- try) Bilateral; OGOV = Other Government; ONGO = Other Country NGO. 3  United States Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 2003, P.L. 108-25, 108th Cong., 1st sess. (May 27, 2003).

OCR for page 609
612 EVALUATION OF PEPFAR Annual Country-Level Funding SI as % of Total PEPFAR Funding $1,400 6% $1,200 Constant 2010 USD Millions 5% $1,000 4% $800 3% $600 2% $400 $200 1% 0 0 FY06 FY07 FY08 FY09 FY10 FY11 Total FIGURE 11-1 PEPFAR funding for country-level strategic information in constant 2010 dollars and as percentage of total PEPFAR funding. NOTES: This figure represents funding for all PEPFAR countries as planned/approved through PEPFAR’s bud- get codes for country-level Strategic Information activities. The budget codes are the only available source of funding information disaggregated by type of activity and are therefore used in this report as the most reasonable and reliable approximation of PEPFAR investment by programmatic area. Data are presented in constant 2010 USD for comparison over time. See Chapter 4 for a more detailed discussion of PEPFAR’s budget codes and the available data for tracking PEPFAR funding. SOURCE: OGAC, 2006d, 2007g, 2008d, 2010d, 2011i,j. the 15 focus countries were to “provide treatment to 2 million HIV-infected people; prevent 7 million new HIV infections; and provide care to 10 mil- lion people infected and affected by HIV/AIDS, including orphans and vulnerable children” (OGAC, 2004, p. 7). The treatment and care 5-year targets were based on meeting 50 percent of the estimated need for the focus countries, using estimates made with input from economists based at the Joint United Nations Programme on HIV/AIDS (UNAIDS) and the Na- tional Institutes of Health (NIH) (NCV-2-USG) (Donnelly, 2012; IOM, 2007a). The 5-year prevention targets were based on cost estimates from UNAIDS and on approximately half of the expected new infections in the focus countries (Donnelly, 2012; IOM, 2007a). With reauthorization under the 2008 Lantos-Hyde Act and ongoing PEPFAR activities, the main cumula- tive targets for treatment, prevention and care have increased steadily (see Table 11-1). In December 2011, on World AIDS Day, President Obama announced an increase in PEPFAR’s target number of people on treatment from 4 million to 6 million by the end of 2013 (Obama, 2011). To accomplish the overall PEPFAR I targets, each partner country mis- sion team was assigned a target to achieve during the initial 5-year imple- mentation period (OGAC, 2003). Starting in FY 2009, under PEPFAR II,

OCR for page 609
PEPFAR’S KNOWLEDGE MANAGEMENT 613 TABLE 11-1 Key PEPFAR Targets Under Legislation and Strategy Mandates Leadership Second Presidential Acta and First Lantos-Hyde PEPFAR Declaration, PEPFAR Five- Reauthorization Five-Year World AIDS Day, Year Strategy Actb Strategy 2011 Target FY 2004– Through Through Through 2013 Timeframe FY 2008 FY 2013 FY 2014 Targets Treatment for Treatment for at Treatment for Treatment for 2 million least 3 million more than 6 million 4 million Prevention of Prevention of Prevention of 7 million new 12 million new more than infections infections 12 million new infections Provision Provision Provision of of care to of care to care to more 10 million, 12 million, than 12 million, including OVC including 5 including 5 million OVC million OVC Training and Training and retention of retention of 140,000 new more than health care 140,000 new workers health care workers NOTE: OVC = orphans and vulnerable children. a United States Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 2003, P.L. 108-25, 108th Cong., 1st sess. (May 27, 2003). b Tom Lantos and Henry J. Hyde United States Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008, P.L. 110-293, 110th Cong., 2nd sess. (July 30, 2008). SOURCES: Obama, 2011; OGAC, 2004, 2009f. targets were determined at the partner country level by PEPFAR mission teams (OGAC, 2008b, 2009d, 2010c, 2011h). To inform the targets for each upcoming fiscal year, which are deter- mined as part of the process of developing the COP, mission teams look at programmatic results from previous years (240-33-USG; 636-1-USG). Ideally, tar- gets should be set based on data, including estimated need, and in at least one partner country there appears to have been an evolution toward an increased use of data by the mission team to determine program targets (240-33-USG). However, the epidemiological data needed to support rational targeting are not always available, and the data that are available vary in their reliability (461-16-USG; 461-18-USG). Mission teams described working closely with implementing partners to set program targets (116-1-USG; 461-16-USG; 461-18-USG). One mission team described the target setting process in this way: ‘[We] work with implementing partners to set targets based on the partners’ budget, disease burden, and previous performance. [We] then aggregate implementing partner’s targets and adjust for over-

OCR for page 609
614 EVALUATION OF PEPFAR lap to get the overall PEPFAR target. Always need to ensure that their target doesn’t exceed the national number.’ (461-16-USG) However, the targets are not always realistic and achievable; as one imple- menting partner interviewee stated about the organization’s program tar- gets, “[It] feels like being asked to make an elephant fly” (166-10-USNGO). Use of Program Targets Interviewees described using targets for program accountability and planning. At the headquarters (HQ) level, OGAC interviewees described comparing data reported by mission teams to the targets set in the COP (NCV-2-USG; NCV-7-USG). At the partner country level, mission teams used targets for COP planning and to assess whether implementing partners met their goals (196-1-USG; 636-1-USG; 461-16-USG). Some mission teams saw program targets as having limited utility for program management (461-16-USG; 196-1-USG). Other mission teams, however, found the information useful for program planning (116-1-USG; 636-1-USG): ‘In particular for the PMTCT [prevention of mother-to-child trans- mission] and treatment indicators, the PEPFAR team has had a process to look back at programmatic results from previous years to inform the targets for the upcoming fiscal year. These program- matic results are useful when developing consensus around the targets and planning of the activities to be implemented in the next year. So, indicator data are used programmatically to inform the managers on how to implement the program especially when trying to scale up.’ (636-1-USG) OGAC is working toward linking program monitoring targets more closely with financial information. Initially, targets were set using best-guess estimates of what the money could buy, given the costs at the time, without knowing the real costs or knowing what the partner country health system could absorb, particularly in the areas of treatment and care (NCV-11-USG). In 2012 OGAC began an expenditure analysis in 10 countries to better un- derstand the range of unit costing for PEPFAR’s core services in order to help mission teams build budgets and more accurately estimate costs (NCV-11- USG) (Holmes et al., 2012). This type of expenditure analysis will become a routine process after this initial study (Holmes et al., 2012). The increased emphasis on tying targets to financial cost may be due to the fact that, as one interviewee put it, ‘the budget now provides constraints and [we] have to really think about how to leverage resources’ (NCV-2-USG). The targets are ‘more useful and more realistic now’ (NCV-2-USG).

OCR for page 609
PEPFAR’S KNOWLEDGE MANAGEMENT 615 Interviewees described an inherent tension between trying to meet pro- gram targets while also trying to implement interventions such as investing in quality programs, health systems strengthening efforts, building capacity, and focusing on prevention. These types of activities contribute to PEPFAR goals but could result in lower numbers reached toward the program targets as compared to investing directly in implementing service delivery (331-43-USG; 587-12-USG; 166-3-USG; 166-6-USG; 166-10-USNGO; 116-7-USG; 272-15-PCNGO). As an example, one mission team interviewee described how the focus on program targets can conflict with efforts to build capacity: ‘PEPFAR funds civil society to do specific projects but this doesn’t teach them how to engage the government, motivate staff, etc. PEPFAR is set up to fund organizations to achieve PEPFAR out- comes/targets. It is hard for PEPFAR to help civil society grow into these roles while also achieving PEPFAR targets.’ (166-4-USG) A PEPFAR-funded nongovernmental organization (NGO) described trying to achieve the targets in this way: ‘Ultimately it becomes a number crunching exercise. We are chas- ing the numbers. We have to find a balance of achieving the target but also rendering a quality service to the OVC. Sometimes it is just the figures that makes a difference—if you do not achieve the target you get “rapped on the knuckles” but if you achieve the target no- body ever asks if you can ensure the quality of the services. We try and render quality services and also meet the targets.’ (272-15-PCNGO) Use of Evidence to Prioritize Activities PEPFAR has emphasized the use of epidemiological data and interven- tion effectiveness data to determine which activities and target populations should be prioritized for implementation in partner countries (NCV-13-ML; NCV- 16-USG; NCV-27-ML; NCV-28-ML; NCV-29-ML) (see also the sections later in this chapter on PEPFAR support for epidemiological data and for evaluation and research). Despite this emphasis on using evidence to drive PEPFAR activities, there are examples, particularly from early in PEPFAR’s implementation, where evidence-informed strategies were not employed, such as the emphasis on the abstinence and be faithful components of the “Abstinence, Be faithful, and correct and consistent Condom use” approach (also known as “ABC”) prevention strategy and the lack of approval for needle exchange programs despite epidemiological data supporting the success of comprehensive pro- grams that included needle exchange among people who inject drugs (IOM, 2007b; Lyerla et al., 2012).

OCR for page 609
616 EVALUATION OF PEPFAR However, there are also clear examples where PEPFAR has functioned as a learning organization, shifting the focus of its activities in response to new evidence. One such example is in the area of using voluntary medical male circumcision (VMMC) to prevent HIV/AIDS via sexual transmission. After WHO and UNAIDS released normative guidance regarding the ben- efits of VMMC in 2007, PEPFAR began implementing VMMC in countries with high HIV prevalence and low male circumcision rates and it has since become the largest supporter of VMMC for HIV prevention globally (NCV-7- USG) (Goosby, 2012; WHO, 2012). Other examples of PEPFAR’s programs evolving over time to reflect the available knowledge and evidence include moving to a combination prevention approach for the prevention of sexual transmission, moving to comprehensive prevention approaches for people who inject drugs, and shifting the initiation threshold for treatment to higher CD4 counts (Lyerla et al., 2012; Needle et al., 2012; OGAC, 2010a, 2011c). (See also Chapter 5, “Prevention,” and Chapter 6, “Care and Treatment.”) Although PEPFAR policies have changed in response to emerging scien- tific evidence, PEPFAR has typically not moved ahead of global standards. PEPFAR usually changes its internal policies only after normative bodies, such as WHO, release appropriate guidelines (NCV-7-USG). The U.S. govern- ment (USG) is, however, heavily involved in the process for developing these normative guidelines. For example, OGAC technical working groups (TWGs) include representatives from WHO and UNAIDS, and when these organizations develop new guidelines, they are typically cleared by members of the OGAC TWGs (NCV-7-USG). In terms of implementing policy changes within PEPFAR, HQ-level TWGs are engaged in putting evidence together, which then goes to the Deputy Principals, followed by the ambassador (U.S. Global AIDS Coor- dinator), who makes the final decisions about a policy change or moving forward on new topics (NCV-7-USG). PEPFAR previously had a Scientific Steer- ing Committee that met “regularly to ensure that PEPFAR programs [were] scientifically sound” (OGAC, 2007d, p. 168). Since 2011, the PEPFAR Scientific Advisory Board (SAB) has provided guidance to the ambassador on “scientific, implementation and policy issues” related to the HIV/AIDS response (OGAC, 2011a) (NCV-7-USG). See section titled “Implementation Sci- ence: The Way Forward” later in this chapter for additional discussion of the SAB. Conclusion: Target setting has been used to focus PEPFAR ac- tivities and for program planning and accountability. PEPFAR has utilized epidemiologic data, normative guidelines, and intervention effectiveness data to drive program activities. Despite some excep- tions, especially in the first phase of implementation, PEPFAR has

OCR for page 609
PEPFAR’S KNOWLEDGE MANAGEMENT 617 based its programs on available evidence and has responded to new knowledge and scientific evidence as it has emerged. Alignment of Targets and Priorities with Partner Countries In PEPFAR II, there has been an increased emphasis on aligning PEPFAR targets with partner country priorities. As stated in the FY 2010 COP guidance, “Annual technical area summary targets should be based on USG support and should feed into the national program five-year goals set through a strategic planning process led by the host country government and supported by key stakeholders” (OGAC, 2009c, p. 52). One mission team interviewee spoke of the need for increased coordination with partner country governments in setting PEPFAR targets: ‘Going forward we need to have more discussions and involvement with the Ministry of Health. If PEPFAR is going to support the national program then targets should be based on that. There needs to be more communication among [PEPFAR mission team] TWGs and with the ministry when target setting.’ (461-18-USG) One mission team described its current alignment with the government as ‘the PEPFAR team takes the government vision and targets (from the HIV plan) and tries to align by saying, “Here’s what we can do to meet your goals”’ (240-9-USG). One mechanism for increased partner country alignment is the PEPFAR Partnership Framework structure (OGAC, 2009b) described in more de- tail in Chapter 10. Partnership Frameworks are intended “to provide a 5-year joint strategic framework for cooperation between the USG, the partner government, and other partners to combat HIV/AIDS in the coun- try through technical assistance and support for service delivery, policy reform, and coordinated financial commitments” (OGAC, 2009b, p. 3). As of July 2012, 19 partner countries and 2 regions had signed Partner- ship Frameworks (OGAC, 2012d). Ideally, targets and priorities that are set based on a country’s Partnership Framework would result in alignment between PEPFAR and the partner country government (OGAC, 2009b). In one country, USG interviewees described how the Partnership Framework process is a key aspect of annual COP planning and has helped to align PEPFAR and national priorities (116-1-USG; 116-4-USG), with PEPFAR saying to the partner country government, “We will put your priorities ahead of ours” (116-2-USG). A partner country government interviewee in this same country commented on the USG’s efforts to align with the partner country priorities: “Must commend the efforts that PEPFAR has made in recent years. PEPFAR has tried as much as possible to harmonize and

OCR for page 609
618 EVALUATION OF PEPFAR align with country priorities, for example, through the Partnership Framework. This has been advocated at the highest level. This is a strong achievement.” (116-16-PCGOV) PEPFAR’s priorities and country government priorities, however, may not always align. In one partner country, for example, government inter- viewees described how during the Partnership Framework process, the country’s prevention priorities did not match PEPFAR’s priorities (587-7-PCGOV; 587-8-PCGOV). Despite some exceptions, however, interviewees across part- ner countries felt that the Partnership Framework structure was helping to improve alignment of PEPFAR and partner country priorities (116-4-USG; 116-16-PCGOV; 166-10-USNGO; 272-5-PCGOV; 272-36-USG). PROGRAM MONITORING DATA PEPFAR’s largest and most sustained effort to create knowledge has been the generation of program monitoring data to track results and report on PEPFAR achievements to Congress. The following sections describe several interrelated aspects of PEPFAR’s program monitoring system: col- lection and reporting, indicator selection and appropriateness, alignment and harmonization with partner countries and other stakeholders, data quality, and data use. Collection and Reporting Program monitoring data are collected by staff at PEPFAR-supported sites such as clinics and community-based programs. Partners who imple- ment programs with PEPFAR funds collate PEPFAR indicator data from the sites that they operate or support and report these data to their respective PEPFAR funding agency in country, e.g., the U.S. Agency for International Development (USAID), the U.S. Centers for Disease Control and Prevention (CDC), etc. (GAO, 2011a). Data from different implementing partners are aggregated by agency and then across mission team agencies before being submitted to OGAC by the in-country PEPFAR SI Liaison (GAO, 2011a). Often, implementing partners and site-level staff also carry out data collec- tion and reporting to meet their own organizational reporting requirements, as well as the reporting requirements for partner countries. The degree to which this data reporting uses indicators and processes that overlap with PEPFAR varies; these issues are discussed in more detail later in this section of the chapter. OGAC provides guidance defining the indicators in PEPFAR’s program monitoring system and the level at which each indicator is to be reported (see Table 11-2). PEPFAR mission teams report data for the required indi-

OCR for page 609
PEPFAR’S KNOWLEDGE MANAGEMENT 619 cators centrally to the OGAC SI office on an annual basis, with a subset of indicators (7 indicators in PEPFAR I and 8 indicators in PEPFAR II) also reported semi-annually (NCV-2-USG) (GAO, 2011a,f). Over the course of PEPFAR, there have been a few iterations of program monitoring indicator guidance released by OGAC (OGAC, 2005c, 2007f, 2009e). As summa- rized in Table 11-2, the first round of indicator guidance, released in 2005, defined 65 indicators to be reported annually to OGAC. The next indicator guidance, issued in 2007, increased the number of centrally reported indica- tors to 76. The Next Generation Indicators (NGIs) guidance, introduced in 2009 for reporting beginning in FY 2010, reduced the number to 31 centrally, routinely reported indicators (25 programmatic indicators, 1 ad- ditional programmatic indicator if a partner country has a signed Partner- ship Framework, and 5 national-level indicators). With the introduction of the NGIs, OGAC created a new category of indicators (n=31) that are essential for mission teams to collect but that do not have to be routinely reported centrally (see Table 11-2). The rationale for these indicators is to ensure that mission teams have specific data available at the partner country level to respond to ad hoc requests for information from Congress (NCV-2-USG). OGAC indicator guidance also includes definitions for additional indicators that are recommended for mis- sion teams to use for program management, if applicable to that country’s program; however, these indicators are not reported centrally. The number of this type of indicator increased substantially with the introduction of the NGIs, from 23 indicators to 92 recommended indicators (see Table 11-2). The evolution of indicators in the new guidance is discussed in more detail later in this section of the chapter. From FY 2006 to FY 2009, COPs and program monitoring data were submitted from PEPFAR mission teams to OGAC via an electronic, Internet-based system called the Country Operational Plan Reporting Sys- TABLE 11-2 Number of PEPFAR Indicators by Reporting Status and Year of Indicator Guidance 2005 2007 2009 Routinely Reported to OGAC 65 76 31 Not Routinely Reported to OGAC 23 23 123 Essential for PEPFAR mission teams — — 31 Recommended for PEPFAR mission teams 23 23 92 Total 88 99 154 NOTES: One indicator defined in the 2009 guidance is routinely reported only from programs that have signed a Partnership Framework with the partner country. One indicator that was previously not routinely reported was elevated to being routinely reported starting in FY 2011, increasing the total number of routinely reported indicators to 32. SOURCES: OGAC, 2005c, 2007f, 2009e, 2012b.

OCR for page 609
708 EVALUATION OF PEPFAR and ensure that these stipulated data and results generated by PEPFAR or through PEPFAR-supported activities are made available in a timely manner to PEPFAR stakeholders, exter- nal evaluators, the research community, and other interested parties. T  he following sections describe additional considerations for imple- menting each of the components of this recommendation. Further considerations for implementation of Recommendation 11-1A: Program monitoring and evaluation • OGAC’s current tiered program monitoring indicator reporting structure (illustrated in Figure 11-10) should be further streamlined to report upward only those indicators essential at each PEPFAR level: o ier 1: A small set of core indicators, fewer than the current 25, T to be reported to central HQ level. These data should be used to monitor performance across PEPFAR as a whole, for congres- sional reporting, and to document trends; as such, these indica- tors should remain consistent over time. Whenever possible and appropriate, these indicators should be harmonized with exist- ing global indicators and national indicators; therefore, some centrally reported indicators will reflect PEPFAR’s contribution rather than aim to measure direct attribution. o ier 2: A larger menu of indicators defined in OGAC guidance, T from which a subset are selected for their applicability to coun- try programs to be reported by implementing partners to the U.S. mission teams but not routinely reported to HQ. These data should be used to monitor the effectiveness of the in-country response and to support mutual accountability with partner countries and their citizens. These data could be considered for occasional centralized use to inform special studies or respond to congressional requests but aggregation and comparability across countries may be limited in this tier because all mission teams may not collect the same data. o ier 3: Indicators selected by implementing partners to monitor T and manage program implementation and effectiveness that are not routinely reported to mission teams. Implementing partners should select appropriate indicators defined in OGAC guidance

OCR for page 609
FIGURE X PEPFAR Information Management Global PEPFAR Program Reporting Monitoring Reporting System Systems Tier 1 Data • Small set of key indicators reported Office by mission teams to OGAC of the • Aggregated across countries to Global AIDS monitor performance across whole Coordinator of PEPFAR and report to Congress (OGAC) Data Feedback Tier 2 Data Data Reporting Mechanism • Larger number of indicators reported by implementing partners Mission Teams to mission teams • Used by mission teams for monitoring the effectiveness Data of the in-country response and Data Reporting Feedback accountability to partner countries Partner Mechanism Country Health Implementing Partners Tier 3 Data Information • Broad set of Indicators collected and used by Systems implementing partners • Used to manage program implementation FIGURE 11-10 Recommended PEPFAR tiered reporting in the context of partner country and global reporting systems. 709

OCR for page 609
710 EVALUATION OF PEPFAR and augment these with other indicators as needed for their pro- grams. Implementing partners should work with mission teams in developing their program monitoring plans with selected indi- cators. Mission teams should provide oversight and technical as- sistance to ensure implementation of these plans and to promote local quality data collection, use, and mutual accountability. Although not routinely reported, some of these data could be considered for occasional country-level and centralized use. o  GAC should create mechanisms for implementing partners, O mission teams, and agency HQ to mutually contribute to a periodic review across all tiers of indicator development, ap- plicability, and utility and to make modifications if necessary. o ier 1 indicators should be harmonized whenever possible and T appropriate with existing global indicators and national indica- tors. For indicators that are not routinely reported centrally (Tiers 2 and 3), country program planning should facilitate alignment of indicator selection and data collection with partner country HIV monitoring and health information systems. • OGAC should complement program monitoring with a unified evaluation portfolio that includes periodic program evaluation at the PEPFAR country program and implementing partner levels to assess process, progress, and outcomes as well as periodic impact evaluations at the country, multi-country, and HQ levels. o  GAC evaluation guidance should provide information about O prioritizing areas for evaluation, the types of evaluation ques- tions, methodological guidance, potential study designs, tem- plate evaluation plans, examples of key outcomes, and how evaluation results should be used and disseminated. PEPFAR should support a range of appropriate methodologies for pro- gram evaluation, including mixed qualitative and quantitative methods, and should shift emphasis from probability designs to plausibility designs that provide valid evidence of impact. o o allow for some comparability across countries and programs, T OGAC and HQ technical working groups should, with input from country teams, strategically plan and coordinate a subset of evaluations within programmatic areas that include (but are not limited to) a minimum set of centrally identified and defined outcome measures and methodologies. o  ithin PEPFAR-supported evaluation activities there should W be an emphasis on the use of in-country local expertise to en-

OCR for page 609
PEPFAR’S KNOWLEDGE MANAGEMENT 711 hance capacity building for program evaluation and contribute to country ownership. • For both program monitoring and evaluation OGAC should continue its work on defining and developing measures to assess progress in the currently under-measured areas of country owner- ship, sustainability, gender, policy, capacity building, and technical assistance. Further considerations for implementation of Recommendation 11-1B: Research • OGAC should clearly define which activities and methodologies will be included under the umbrella of PEPFAR-supported research as distinguished from program evaluation. • OGAC should draw on input from implementing agencies, mis- sion teams, partner countries, implementing partners, the SAB, and other experts to identify and articulate research priorities and appropriate research methodologies. The research proposals and funding mechanisms should be designed to ensure that these pri- orities are met and that methodologies are applied through RFAs and other investigator-driven research proposals as well as through targeted solicitations of research in gap areas not met through open requests. • Given PEPFAR’s legislative and programmatic objectives to sup- port research that assesses program quality, effectiveness, and population-based impact; optimizes service delivery; and contrib- utes to the global evidence base on HIV/AIDS interventions and program implementation, at the time of this evaluation the com- mittee identified the following gaps in PEPFAR’s research activities: o ehavioral and structural interventions, especially in areas such B as prevention, gender, nonclinical and OVC care and support, and treatment retention and adherence. These research activities should employ appropriate methodologies and study designs without being unduly limited to random assignment designs. o  osts, benefits, and feasibility of integrating gender-focused pro- C grams with clinical and community-based activities. o  ealth systems strengthening interventions across the WHO H building blocks, with a prioritized goal of determining setting- and system-specific feasibility, effectiveness, quality of services, and costs for innovative models.

OCR for page 609
712 EVALUATION OF PEPFAR o o contribute to country ownership, PEPFAR should facilitate T in-country local participation and research capacity building through simplified, streamlined, and transparent application and review processes that encourage submissions from country- based implementing partners and researchers. Further considerations for implementation of Recommendation 11-1C: Knowledge transfer and dissemination • The knowledge created within PEPFAR that should be more widely documented and disseminated includes program monitoring data, financial data, research results, evaluation outcomes, best practices, and informal knowledge such as implementation experience and lessons learned. • To institutionalize internal and external knowledge transfer and learning, PEPFAR should develop appropriate systems and pro- cesses for the most needed types and scale of knowledge transfer. To achieve this, PEPFAR should draw on broad stakeholder input to assess the strengths and weaknesses in current processes and to identify needs and opportunities for improved knowledge transfer. • PEPFAR should invest in innovative mechanisms and technol- ogy to facilitate knowledge transfer across partner countries and implementing partners. Mechanisms currently used successfully on a small scale and an ad hoc basis could be formally scaled up across PEPFAR. OGAC should also look to other organizations with wide geographic reach and organizational complexity, such as multi-country PEPFAR implementing partners, other large global health initiatives, and global corporations, for models of successful knowledge transfer systems. • OGAC should develop a policy for data sharing and transparency that facilitates timely access to PEPFAR-created knowledge for analysis and evaluation. The purpose of this policy would be to ensure that, within a purposefully and reasonably defined scope, specified program monitoring data and financial data, evaluation outcomes, and research data and results generated with PEPFAR support by contractors, grantees, mission teams, and USG agencies be made available to the public, research community, and other external stakeholders. OGAC and the PEPFAR implementing agen- cies should consult with both internal and external parties who would be affected by this policy to help identify the data that are most critical for external access and that can be reasonably subject

OCR for page 609
PEPFAR’S KNOWLEDGE MANAGEMENT 713 to data-sharing requirements, as well as to help develop feasible mechanisms to implement a data-sharing policy. o or routinely collected financial and program monitoring data, a F limited set of essential data should be identified and made avail- able for external use in a timely way. o valuation and research reports and publications using data col- E lected through PEPFAR-supported programs should be tracked and made available in a publicly accessible central repository. USG agencies with similar repositories can be considered as models. o or research data and other information that is expressly gener- F ated for new knowledge, the policy should respect time-bound exclusivity for the right to engage in the publication process, yet also ensure the timely availability of data, regardless of publica- tion, for access and use by external evaluators and researchers. OGAC should look to USG agencies with similar research data policies as models. o  developing the policy and specifying the scope of data to In be included, several key factors and potential constraints that can affect the implementation of the policy will need to be ad- dressed. These include patient and client information confiden- tiality; the financial resources, personnel, and time needed to make data available; and issues of data ownership, especially in the context of increasing responsibility in partner countries and the provision of PEPFAR support through country systems or through activities and programs supported by multiple funding streams. REFERENCES AIDStar-Two. 2012. OVCsupport.Net—a global hub on children and HIV. http://www. ovcsupport.net/s (accessed October 5, 2012). Bergmann, H. 2011. Field driven learning meeting: Linkages to and retention in HIV care and support programs. Arlington, VA: USAID’s AIDS Support and Technical Assistance Resources, (AIDStar-One), Task Order 1. Bouey, P., and N. Padian. 2011. PEPFAR-funded evaluations presentation. Presentation at the PEPFAR Scientific Advisory Board Meeting, Washington, DC. Bryant, M., J. Beard, L. Sabin, M. I. Brooks, N. Scott, B. A. Larson, G. Biemba, and C. Miller. 2012. PEPFAR’s support for orphans and vulnerable children: Some beneficial effects, but too little data, and programs spread thin. Health Affairs 31(7):1508-1518. Dentzer, S. 2012. Assessing the President’s Emergency Plan for AIDS Relief. Health Affairs 31(7).

OCR for page 609
714 EVALUATION OF PEPFAR Donnelly, J. 2012. The President’s Emergency Plan for AIDS Relief: How George W. Bush and aides came to “think big” on battling HIV. Health Affairs 31(7):1389-1396. Fain, J. 2005. Editorial: Is there a difference between evaluation and research? Diabetes Educator 31(2):150-155. Fullem, A., M. Levy, and M. Sharer. 2012. Meeting the HIV; maternal, newborn, and child health; and social support needs of mothers and their young children. Field driven learn- ing meeting, Addis Ababa, Ethiopia, November 8 to 10, 2011. Arlington, VA: AIDStar- One, Task Order 1. GAO (U.S. Government Accountability Office). 2011a. PEPFAR program planning and re- porting. GAO-11-785. Washington, DC: GAO. GAO. 2011b. Performance measurement and evaluation: Definitions and relationships. Wash- ington, DC: GAO. GAO. 2012. President’s Emergency Plan for AIDS Relief—agencies can enhance evaluation quality, planning, and dissemination: Report to congressional committees. Washington, DC: GAO. Garvin, D. A. 1993. Building a learning organization. Harvard Business Review 71(4):78-91. Gay, J., M. Croce-Galis, and K. Hardee. 2012. What works for women and girls: Evidence for HIV/AIDS interventions. 2nd edition. www.whatworksforwomen.org (accessed October 5, 2012). Global Fund (Global Fund to Fight AIDS, Tuberculosis, and Malaria). 2011. Monitoring and evaluation toolkit: HIV, tuberculosis, malaria and health and community systems strengthening. Part 2: HIV. Geneva: Global Fund. Goosby, E. 2012. The President’s Emergency Plan for AIDS Relief: Marshalling all tools at our disposal toward an AIDS-free generation. Health Affairs 31(7):1593-1598. Grosso, A. L., K. Hoan Tram, O. Ryan, and S. Baral. 2012. Countries where HIV is con- centrated among most-at-risk populations get disproportionally lower funding from PEPFAR. Health Affairs 31(7):1519-1528. Habicht, J. P., C.G. Victoria, J.P. Vaughan. 1999. Evaluation designs for adequacy, plausibil- ity and probability of public health programme performance and impact. International Journal of Epidemiology (28):10-18. HHS (U.S. Department of Health and Human Services). 2012. Operations research (imple- mentation science) for strengthening program implementation through the President’s Emergency Plan for AIDS Relief (PEPFAR). http://www.grants.gov/search/search. do?mode=VIEW&oppId=136553 (accessed October 5, 2012). Ho, D. D., P. A. Volberding, and W. A. Blattner. 2012. The United States President’s Emergency Plan for AIDS Relief (PEPFAR): Its vision, achievements, and new directions. Journal of Acquired Immune Deficiency Syndromes 60(Suppl 3). Holmes, C. 2012. Presentation to SAB: Implementation science updates. Washington, DC: OGAC. Holmes, C. B., J. M. Blandford, N. Sangrujee, S. R. Stewart, A. DuBois, T. R. Smith, J. C. Martin, A. Gavaghan, C. A. Ryan, and E. P. Goosby. 2012. PEPFAR’s past and future efforts to cut costs, improve efficiency, and increase the impact of global HIV programs. Health Affairs 31(7):1553-1560. Holzscheiter, A., G. Walt, and R. Brugha. 2012. Monitoring and evaluation in global HIV/ AIDS control-weighing incentives and disincentives for coordination among global and local actors. Journal of International Development 24(1):61-76. IHME (Institute for Health Metrics and Evaluation). 2011. Financing global health 2011: Continued growth as MDG deadline approaches. Seattle: IHME. IOM (Institute of Medicine). 2007a. PEPFAR implementation: Progress and promise. Wash- ington, DC: The National Academies Press.

OCR for page 609
PEPFAR’S KNOWLEDGE MANAGEMENT 715 IOM. 2007b. Preventing HIV infection among injecting drug users in high risk countries: An assessment of the evidence. Washington, DC: The National Academies Press. IOM. 2011. IOM staff internal review of PEPFAR phase I evaluation—list of targeted evalu- ations document. Washington, DC: IOM. IOM and NRC (National Research Council). 2010. Strategic approach to the evaluation of programs implemented under the Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008. Washing- ton, DC: The National Academies Press. JSI (John Snow International). 2012. AIDStar-One: AIDS support and technical assistance resources. http://www.aidstar-one.com (accessed October 5, 2012). Levin-Rozalis, M. 2003. Evaluation and research: Differences and similarities. Canadian Journal of Program Evaluation 18(2):1-31. Loermans, J. 2002. Synergizing the learning organization and knowledge management. Journal of Knowledge Management 6(3):285-294. Lyerla, R., C. S. Murrill, P. D. Ghys, J. M. Calleja-Garcia, and K. M. Decock. 2012. The use of epidemiological data to inform the PEPFAR response. Journal of Acquired Immune Deficiency Syndromes 60(Suppl 3):S57-S62. Needle, R., J. Fu, C. Beyrer, V. Loo, A. S. Abdul-Quader, J. A. McIntyre, Z. Li, J. Mbwambo, M. Muthui, and B. Pick. 2012. PEPFAR’s evolving HIV prevention approaches for key populations—people who inject drugs, men who have sex with men, and sex workers: Progress, challenges, and opportunities. Journal of Acquired Immune Deficiency Syn- dromes 60(Suppl 3):S145-S151. NIH (National Institutes of Health). 2010. Limited competition: Administrative supplements for HIV/AIDS implementation science in PEPFAR settings. http://grants.nih.gov/grants/ guide/notice-files/NOT-AI-10-023.html (accessed 2012). NIH. 2011. NIH/PEPFAR collaboration for implementation science and impact evaluation (R01). http://grants.nih.gov/grants/guide/rfa-files/RFA-AI-11-003.html (accessed October 5, 2012). Obama, B. 2011. Remarks by the President on World AIDS Day, December 1, Washington, DC. Office of Learning, Evaluation and Research. 2012. USAID evaluation policy: Year one; first annual report and plan for 2012 and 2013. Washington, DC: USAID. OGAC (Office of the U.S. Global AIDS Coordinator). 2003. PEPFAR country operational plan guidelines for FY04. Washington, DC: OGAC. OGAC. 2004. The President’s Emergency Plan for AIDS Relief: U.S. five-year global HIV/ AIDS strategy. Washington, DC: OGAC. OGAC. 2005a. Emergency Plan for AIDS Relief Fiscal Year 2005 operational plan: June 2005 update. Washington, DC: OGAC. OGAC. 2005b. President’s Emergency Plan for AIDS Relief: FY06 country operational plan final guidance. Washington, DC: OGAC. OGAC. 2005c. The President’s Emergency Plan for AIDS Relief: Indicators, reporting require- ments, and guidelines for focus countries. Washington, DC: OGAC. OGAC. 2006a. A blueprint for public health evaluations in the President’s Emergency Plan for AIDS Relief. Washington, DC: OGAC. OGAC. 2006b. The President’s Emergency Plan for AIDS Relief. FY2007 supplemental COP guidance resource guide. Washington, DC: OGAC. OGAC. 2006c. The President’s Emergency Plan for AIDS Relief: FY2007 country operational plan guidance. Washington, DC: OGAC. OGAC. 2006d. The U.S. President’s Emergency Plan for AIDS Relief fiscal year 2006: Op- erational plan. 2006 August update. Washington, DC: OGAC.

OCR for page 609
716 EVALUATION OF PEPFAR OGAC. 2007a. 2007 HIV/AIDS implementers meeting: Scaling up through partnerships; program and abstract book.Washington, DC, OGAC. OGAC. 2007b. Data quality assurance tool for program-level indicators. Washington, DC, OGAC. OGAC. 2007c. Factsheet: PEPFAR extranet (PEPFAR.net), edited by OGAC. Washington, DC: U.S. DoS, USAID, DoD, Department of Commerce, DoL, HHS, Peace Corps. OGAC. 2007d. The power of partnerships: The President’s Emergency Plan for AIDS Relief, third annual report to Congress. Washington, DC: OGAC. OGAC. 2007e. The President’s Emergency Plan for AIDS Relief: FY2008 country operational plan guidance. Washington, DC: OGAC. OGAC. 2007f. The President’s Emergency Plan for AIDS Relief: Indicators, reporting require- ments, and guidelines. Indicators reference guide: FY2007 reporting/FY2008 planning. Washington, DC: OGAC. OGAC. 2007g. The U.S. President’s Emergency Plan for AIDS Relief fiscal year 2007: Op- erational plan. 2007 June update. Washington, DC: OGAC. OGAC. 2008a. 2008 HIV/AIDS implementers’ meeting: Scaling up through partnerships— overcoming obstacles to implementation, program and abstract book. Washington, DC: OGAC. OGAC. 2008b. The President’s Emergency Plan for AIDS Relief: FY2009 country operational plan guidance. Washington, DC: OGAC. OGAC. 2008c. The U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2008: PEPFAR operational plan. Washington, DC: OGAC. OGAC. 2009a. 2009 HIV/AIDS implementers’ meeting program and abstract book: Optimiz- ing the response, partnerships for sustainability. Washington, DC: OGAC. OGAC. 2009b. Guidance for PEPFAR partnership frameworks and partnership framework implementation plans. Version 2.0. Washington, DC: OGAC. OGAC. 2009c. The President’s Emergency Plan for AIDS Relief: FY2010 country operational plan guidance. Washington, DC: OGAC. OGAC. 2009d. The President’s Emergency Plan for AIDS Relief: FY2010 country operational plan guidance: Programmatic considerations. Washington, DC: OGAC. OGAC. 2009e. The President’s Emergency Plan for AIDS Relief: Next generation indicators reference guide. Version 1.1. Washington, DC: OGAC. OGAC. 2009f. The U.S. President’s Emergency Plan for AIDS Relief: Five-year strategy. Washington, DC: OGAC. OGAC. 2010a. Comprehensive HIV prevention for people who inject drugs, revised guidance. Washington, DC: OGAC. OGAC. 2010b. The President’s Emergency Plan for AIDS Relief: Public health evaluation (PHE) concept submission guidance. Washington, DC: OGAC. OGAC. 2010c. The President’s Emergency Plan for AIDS Relief: FY2011 country operational plan guidance. Washington, DC: OGAC. OGAC. 2010d. The U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2009: PEPFAR operational plan. November 2010. Washington, DC: OGAC. OGAC. 2011a. Charter of the President’s Emergency Plan for AIDS Relief (PEPFAR) Scien- tific Advisory Board. Washington, DC: OGAC. OGAC. 2011b. E-mail communication between OGAC staff and IOM: “Request from IOM outcome and impact evaluation of PEPFAR team.” Washington, DC: OGAC. OGAC. 2011c. Guidance for the prevention of sexually trasmitted HIV infections. Washing- ton, DC: OGAC. OGAC. 2011d. List of continuing PHEs for FY 2012. Washington, DC: OGAC. OGAC. 2011e. OGAC technical review of table—“Evolution of PEPFAR supported research activities.” Washington, DC: OGAC.

OCR for page 609
PEPFAR’S KNOWLEDGE MANAGEMENT 717 OGAC. 2011f. PEPFAR response to data request from IOM for the PEPFAR II evaluation. Washington, DC: OGAC. OGAC. 2011g. The President’s Emergency Plan for AIDS Relief: FY2012 country operational plan guidance Washington, DC: OGAC. OGAC. 2011h. The President’s Emergency Plan for AIDS Relief: FY2012 country operational plan guidance appendices. Washington, DC: OGAC. OGAC. 2011i. The U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2010: PEPFAR operational plan. Washington, DC: OGAC. OGAC. 2011j. The U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2011: PEPFAR operational plan. Washington, DC: OGAC. OGAC. 2011k. The U.S. President’s Emergency Plan for AIDS Relief: Seventh annual report to Congress. Washington, DC: OGAC. OGAC. 2012a. Annual reports to congress on the President’s Emergency Plan for AIDS Relief. http://www.pepfar.gov/press/c19573.htm (accessed October 5, 2012). OGAC. 2012b. E-mail communication between OGAC staff and IOM: “Clarification regard- ing number of NGIs.” Washington, DC: OGAC. OGAC. 2012c. E-mail communication between OGAC staff and IOM: “Follow-up queries for IOM PEPFAR evaluation.” Washington, DC: OGAC. OGAC. 2012d. Email communication between OGAC staff and IOM: “RE: Follow-up re- quest: Partnership frameworks and implementation plans.” Washington, DC: OGAC. OGAC. 2012e. E-mail communication between OGAC staff and IOM: “re: Request from IOM outcome and impact evaluation of PEPFAR team.” Washington, DC: OGAC. OGAC. 2012f. PEPFAR Scientific Advisory Board. http://www.pepfar.gov/sab (accessed Oc- tober, 2012). OGAC. 2012g. PEPFAR Scientific Advisory Board membership list. http://www.pepfar.gov/ sab/160067.htm (accessed October 30, 2012). OGAC. 2012h. PEPFAR Scientific Advisory Board working groups. http://www.pepfar.gov/ sab/workinggroups/index.htm (accessed October 30, 2012). OGAC. 2012i. Reports: Annual reports to Congress and other reports to Congress. http:// www.pepfar.gov/reports/progress/index.htm (accessed October 30, 2012). OGAC. 2012j. The United State’s President’s Emergency Plan for AIDS Relief. http://www. pepfar.gov (accessed October 30, 2012). OGAC SAB Data Working Group. 2011. SAB Data Working Group (DWG) conference call notes. http://www.pepfar.gov/documents/organization/166969.pdf (accessed October 30, 2012). Padian, N. S., C. B. Holmes, S. I. McCoy, R. Lyerla, P. D. Bouey, and E. P. Goosby. 2011. Im- plementation science for the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). Journal of Acquired Immune Deficiency Syndromes 56(3):199-203. Porter, L. E., P. D. Bouey, S. Curtis, M. Hochgesang, P. Idele, B. Jefferson, W. Lemma, R. Myrick, H. Nuwagaba-Biribonwoha, D. Prybylski, Y. Souteyrand, and T. Tulli. 2012. Be- yond indicators: Advances in global HIV monitoring and evaluation during the PEPFAR era. Journal of Acquired Immune Deficiency Syndromes 60(Suppl 3):S120-S126. Reyes, M. 2009. Summary of PEPFAR State of the Program Area (SOPA): Care and sup- port. http://caps.ucsf.edu/uploads/pubs/presentations/pdf/PEPFARCareandSupportSOPA Summary.pdf (accessed January 25, 2013) Simonds, R. J., C. A. Carrino, and M. Moloney-Kitts. 2012. Lessons from the President’s Emergency Plan for AIDS relief: From quick ramp-up to the role of strategic partnership. Health Affairs 31(7):1397-1405. Small, P. 2012. Four differences between research and program evaluation. Blog: NonProfit Capacity Building, Authenticity Consulting, LLC.

OCR for page 609
718 EVALUATION OF PEPFAR Spratt, K., and Heather Bergmann. 2011. Strengthening gender programming in PEPFAR: Technical exchange of best practices, program models and resources. Arlington, VA: AIDStar-One, Task Order 1. Stufflebeam, D. 2007. CIPP evaluation model checklist: A tool for applying the CIPP model to assess long-term enterprises. Kalamazoo, MI: Western Michigan University. Swan, J., H. Scarbrough, and J. Preston. 1999. Knowledge management: The next fad to forget people. Paper read at 7th European Conference on Information Systems, Copenhagen. Thomas, J. C., S. Curtis, and J. B. Smith. 2011. The broader context of implementation sci- ence. Journal of Acquired Immune Deficiency Syndromes 58(1):e19-e21. UNAIDS (Joint United Nations Programme on HIV/AIDS). 2004. Consultation on harmoniza- tion of international AIDS funding, end-of-meeting agreement, Washington consulation of April 25, 2004. Geneva: UNAIDS. UNAIDS. 2007. Practical guidelines for intensifying HIV prevention: Towards universal ac- cess. Geneva: UNAIDS. UNAIDS. 2009. Monitoring the declaration of commitment on HIV/AIDS: Guidelines on construction of core indicators: 2010 reporting. Geneva: UNAIDS. USAID (U.S. Agency for International Development). 2011a. Implementation science research to support programs under the President’s Emergency Plan for AIDS Relief (PEPFAR). http://www.grants.gov/search/search.do;jsessionid=w6lxTG8QGvxLLXk0BZbXcSl2 FrTyCGQLLPcZV3qXrvyyvsCLFnkC!-25395513?oppId=114673&mode=VIEW (ac- cessed October 30, 2012). USAID. 2011b. USAID’s implementation science investment: Improving HIV/AIDS program- ming through the translation of research to practice. Washington, DC: USAID. USAID. 2012. Development experience clearinghouse. https://dec.usaid.gov/dec/home/Default. aspx (accessed October 5, 2012). Walport, M. and P. Breast. 2011. Sharing research data to improve public health. Lancet 377(9765):537-539. WHO (World Health Organization). 2005. World Health Organization knowledge manage- ment strategy. Geneva: WHO. WHO. 2012. Voluntary medical male circumcision for HIV prevention. http://www.who. int/hiv/topics/malecircumcision/fact_sheet/en/index.html (accessed September 20, 2012). World Bank. 2003. Sharing knowledge: Innovations and remaining challenges. Washington, DC: World Bank.