Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 489
11
PEPFAR’s Knowledge Management
Main Messages
Informing Priorities for PEPFAR-Supported Programs
Despite some exceptions, PEPFAR has implemented evidence-informed programs that have
been modified as new knowledge and scientific evidence emerged. Target setting has been
used to focus PEPFAR activities, program planning, and accountability. PEPFAR has utilized
epidemiological and intervention effectiveness data to drive program activities.
PEPFAR has provided financial and technical support for collecting epidemiological
information in partner countries. This was widely seen as a positive contribution to inform
decisions and priorities in planning the HIV/AIDS response and implementing HIV programs,
encouraging and facilitating responsiveness to the epidemic and the needs in partner
countries.
Program Monitoring Data
PEPFAR’s program monitoring indicator system has faced technological challenges limiting
the ability of both PEPFAR and external stakeholders to utilize and access both current and
historical trend data; resolving these challenges is critical for successful program monitoring.
PEPFAR’s program monitoring has evolved over time: the number of centrally reported
indicators was reduced, indicators to monitor new program activities were introduced, and
indicators identified as problematic, removed. PEPFAR needs a program monitoring strategy
which can adapt over time to respond to feedback, reflect emerging program priorities, and
accurately capture program activities and outcomes. However, this needs to be balanced
with the reality that changes in indicators place a burden on partner country programs and
limit comparability of data, hampering the ability to monitor trends.
PEPFAR’s current indicators do not capture sufficient information on its stated prioritized
goals and activities and are focused primarily on input and outputs. As a result, the program
monitoring system has limited utility for determining the effectiveness of PEPFAR’s efforts.
The need to quickly measure results at the onset of PEPFAR contributed to the development
of PEPFAR-specific data collection systems, which has limited harmonization with partner
countries and the global HIV/AIDS community. More recently, OGAC has worked with other
global actors to harmonize indicators and validate reporting. OGAC has modified the
PEPFAR monitoring system to reduce reporting burden and improve alignment with partner
country programs, however, further modifications could be made by eliminating PEPFAR-
specific language in the indicator guidance; further reducing the reporting burden; improving
indicator harmonization with global indicators; and advancing alignment with partner country
data collection at the program level.
There are some good examples of PEPFAR data use at the implementing partner, Mission
Team, and headquarters levels but, the preponderance of data collected does not seem to
be routinely utilized. PEPFAR’s requirement for collection and reporting of a large amount of
program monitoring data places a large burden on implementing partners and Mission Teams
that has limited the ability to analyze and use data.
PEPFAR has invested in building the capacity of partner countries to plan for, collect,
manage and use HIV data, which has implications for the larger health system. As a result,
PEPFAR has contributed to fostering a culture of evidence among partner countries.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 490
PEPFAR Supported Evaluation and Research
The manner in which PEPFAR initially approached research activities was a missed
opportunity to establish, from its inception, mechanisms to evaluate programs, assess
impact, contribute to the global knowledge base, and develop in-country research capacity.
PEPFAR has made progress in carrying out evaluation and research activities over time:
moving from an early proscription against research, to using TEs and PHEs to work within
research restrictions, to the recent creation of what holds promise as a more useful
processes for establishing priorities, managing activities, documenting “what works,”
expanding PEPFAR’s technical leadership, disseminating findings, and continually improving
the effectiveness and impact of PEPFAR. Defining appropriate and allowable research
activities within PEPFAR, however, was and remains a challenge; specifically clarity around
the activities and aims for evaluation and research within PEPFAR.
Knowledge Transfer and Learning within PEPFAR
PEPFAR has successfully established and used a variety of mechanisms to transfer
knowledge throughout PEPFAR; however, more progress is needed to address limitations in
current systems and to establish formal mechanisms to systematically transfer experiences
across countries, implementing partners, and sites. Without this, there will be missed
opportunities to capitalize on best practices and internal lessons learned.
Knowledge Dissemination External to PEPFAR
OGAC would benefit from developing a formal system to track and manage PEPFAR-funded
dissemination products (publications, reports, abstracts, guidelines and tools) from which to
measure contribution to the global knowledge base, and the global HIV/AIDS community
would benefit from a publically available central repository of these products from which to
share, collaborate, and accelerate knowledge creation.
PEPFAR has had some success in external dissemination of PEPFAR knowledge including:
establishing formal and informal mechanisms share knowledge externally and contributing
vast amounts of evidence and publications to the Global Knowledge Base. Despite this, more
progress is needed to: develop routine formal mechanisms for knowledge exchange with
partner country governments and other partners, increase the amount PEPFAR data that is
publically available for use by researchers and evaluators, and track and measure PEPFAR’s
contribution to the global knowledge base.
Overall Conclusion
PEPFAR has made progress in managing knowledge by developing systems for data
creation and collection, streamlining program monitoring data, advancing PEPFAR’s role and
approach to evaluation and research, and utilizing a wide variety of mechanisms to transfer
knowledge. Yet, like other entities involved in the global HIV/AIDS response, it struggles with
creating, acquiring and transferring the right knowledge, at the appropriate scale, and in a
manner that facilitates use. PEPFAR has the potential to lead the global HIV/AIDS
community in knowledge management by adopting a conceptual framework that articulates
the vision, purposes, intended audiences, and goals of knowledge; how knowledge will be
acquired, created, transferred, used, and disseminated to achieve these goals; and the
complementary roles of program monitoring, evaluation, and research. PEPFAR has the
opportunity to optimize program efficiency and effectiveness through an improved strategy
that 1) streamlines and focuses knowledge creation within PEPFAR; 2) increases acquisition
of knowledge external to PEPFAR; 3) improves the efficiency and effectiveness of knowledge
transfer within and external to PEPFAR; and 4) institutionalizes the use of knowledge to
improve the way work is accomplished.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 491
Recommendations Presented in this Chapter
Recommendation 11-1: OGAC should develop a comprehensive knowledge management
framework, including a program monitoring and evaluation strategy, a prioritized and
targeted research portfolio, and systems for knowledge dissemination. This framework
should adapt to emerging needs to assess PEPFAR’s models of implementation and
contribution to sustainable management of the HIV response in partner countries. This
knowledge management framework will require that PEPFAR implement and strategically
allocate resources for the following :
A. To better document PEPFAR’s progress and effectiveness, OGAC should refine its
program monitoring & evaluation strategy to streamline reporting and to strategically
coordinate a complementary portfolio of evaluation activities to assess outcomes and
effects that are not captured well by program monitoring indicators. Efforts should
support innovation in methodologies and measures where needed. Both monitoring
and evaluation should be specifically matched to clearly articulated data sources,
methods, and uses at each level of PEPFAR’s implementation and oversight.
B. To contribute to filling critical knowledge gaps that impede effective and sustainable
HIV programs, OGAC should continue to redefine permitted research within PEPFAR
by developing a prioritized portfolio with articulated activities and methods. The
planning and implementation process at the country and program level should inform
and be informed by the research portfolio, which should focus on research that will
optimize the effectiveness, quality, and efficiency of PEPFAR-supported activities and
will also contribute to the global knowledge base on implementation of HIV/AIDS
programs.
C. To maximize the use of knowledge created within PEPFAR, OGAC should develop
systems and processes for routine, active transfer and dissemination of knowledge
both within and external to PEPFAR. As one component, OGAC should institute a data
sharing policy, developed through a consultative process. The policy should identify
the data to be included and ensure that these stipulated data and results generated by
PEPFAR or through PEPFAR-supported activities are made available in a timely
manner to PEPFAR stakeholders, external evaluators, the research community, and
other interested parties.
Further considerations for implementation of Recommendation 11-1A: Program
monitoring and evaluation
OGAC’s current tiered program monitoring indicator reporting structure (illustrated in
Figure 11-10) should be further streamlined to report upward only those indicators
essential at each PEPFAR level:
Tier 1: A small set of core indicators, fewer than the current 25, to be reported to
central HQ level. These data should be used to monitor performance across
PEPFAR as a whole, for congressional reporting, and to document trends; as
such these indicators should remain consistent over time. Whenever possible
and appropriate, these indicators should be harmonized with existing global
indicators and/or national indicators; therefore some centrally reported indicators
will reflect PEPFAR’s contribution rather than aiming to measure direct
attribution.
Tier 2: A larger menu of indicators defined in OGAC guidance, from which a
subset are selected for their applicability to country programs to be reported by
implementing partners to the U.S. mission teams but not routinely reported to
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 492
HQ. These data should be used to monitor the effectiveness of the in-country
response and support mutual accountability with partner countries and their
citizens. These data could be considered for occasional centralized use to inform
special studies or respond to congressional requests but aggregation and
comparability across countries may be limited in this tier as all mission teams
may not collect the same data.
Tier 3: Indicators selected by implementing partners to monitor and manage
program implementation and effectiveness that are not routinely reported to
mission teams. Implementing partners should select appropriate indicators
defined in OGAC guidance and augment this with other indicators as needed for
their programs. Implementing partners should work with mission teams in
developing their program monitoring plans with selected indicators. Mission
teams should provide oversight and technical assistance to ensure
implementation of these plans and to promote local quality data collection, use,
and mutual accountability. Although not routinely reported, some of these data
could be considered for occasional country-level and centralized use.
OGAC should create mechanisms for implementing partners, mission teams, and
agency headquarters to mutually contribute to a periodic review across all tiers of
indicator development, applicability, and utility and to make modifications if
necessary.
Tier 1 indicators should be harmonized whenever possible and appropriate with
existing global indicators and/or national indicators. For indicators that are not
routinely reported centrally (Tiers 2 and 3), country program planning should
facilitate alignment of indicator selection and data collection with partner country
HIV monitoring and health information systems.
OGAC should complement program monitoring with a unified evaluation portfolio that
includes periodic program evaluation at the PEPFAR country program and
implementing partner levels to assess process, progress, and outcomes as well as
periodic impact evaluations at the country, multi-country, and headquarters levels.
OGAC evaluation guidance should provide information about prioritizing areas for
evaluation, the types of evaluation questions; methodologies guidance, potential
study designs, template evaluation plans, examples of key outcomes, and how
evaluation results should be used and disseminated. PEPFAR should support a
range of appropriate methodologies for program evaluation, including mixed
qualitative and quantitative methods, and should shift emphasis from probability
designs to plausibility designs that provide valid evidence of impact.
To allow for some comparability across countries and programs, OGAC and
HQ technical working groups should, with input from country teams,
strategically plan and coordinate a subset of evaluations within programmatic
areas that include (but are not limited to) a minimum of centrally identified
and defined outcome measures and methodologies.
Within PEPFAR-supported evaluation activities there should be an emphasis
on the use of in-country local expertise to enhance capacity building for
program evaluation and contribute to country ownership.
For both program monitoring and evaluation OGAC should continue its work on
defining and developing measures to assess progress in the currently under-
measured areas of country ownership, sustainability, gender, policy, capacity
building and technical assistance.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 493
Further considerations for implementation of Recommendation 11-1B: Research
OGAC should clearly define what activities and methodologies will be included under
the umbrella of PEPFAR-supported research, as distinguished from program
evaluation.
OGAC should draw on input from implementing agencies, mission teams, partner
countries, implementing partners, the Scientific Advisory Board, and other expert
consultations to identify and articulate research priorities and appropriate research
methodologies. The research proposals and funding mechanisms should be
designed to ensure that these priorities are met and methodologies are applied
through RFAs and other investigator-driven research proposals as well as through
targeted solicitations of research in gap areas not met through open requests.
Given PEPFAR’s legislative and programmatic objectives to support research that
assesses program quality, effectiveness, and population-based impact; optimizes
service delivery; and contributes to the global evidence base on HIV/AIDS
interventions and program implementation, at the time of this evaluation the
committee identified the following gaps in PEPFAR’s research activities:
Behavioral and structural interventions, especially in areas such as prevention,
gender, nonclinical and OVC care and support, and treatment retention and
adherence. These research activities should employ appropriate methodologies and
study designs, without being unduly limited to random assignment designs.
Costs, benefits, and feasibility of integrating gender-focused programs with clinical and
community-based activities.
Health systems strengthening interventions across the WHO building blocks, with a
prioritized goal of determining setting- and system-specific feasibility, effectiveness,
quality of services, and costs for innovative models.
To contribute to country ownership, PEPFAR should facilitate in-country local
participation and research capacity building through simplified, streamlined, and
transparent application and review processes that encourage submissions from
country-based implementing partners and researchers.
Further considerations for implementation of recommendation 11-1C: Knowledge
transfer and dissemination
The knowledge created within PEPFAR that should be more widely documented and
disseminated includes program monitoring data, financial data, research results,
evaluation outcomes, best practices, and informal knowledge such as
implementation experience, lessons learned, and other information.
To institutionalize internal and external knowledge transfer and learning, PEPFAR
should develop appropriate systems and processes for the most needed types and
scale of knowledge transfer. To achieve this, PEPFAR should draw on broad
stakeholder input to assess the strengths and weaknesses in current processes and
to identify needs and opportunities for improved knowledge transfer.
PEPFAR should invest in innovative mechanisms and technology to facilitate
knowledge transfer across partner countries and implementing partners.
Mechanisms currently used successfully on a small scale and an ad hoc basis could
be more formally scaled up across PEPFAR. OGAC should also look to other
organizations with wide geographic reach and organizational complexity, such as
multi-country PEPFAR implementing partners, other large global health initiatives,
and global corporations, for models of successful knowledge transfer systems.
OGAC should develop a policy for data sharing and transparency that facilitates
timely access to PEPFAR-created knowledge for analysis and evaluation. The
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 494
purpose of this policy would be to ensure that, within a purposefully and reasonably
defined scope, specified program monitoring data and financial data, evaluation
outcomes, and research data and results generated with PEPFAR support by
contractors, grantees, mission teams, and USG agencies be made available to the
public, research community, and other external stakeholders. OGAC and the
PEPFAR implementing agencies should consult with both internal and external
parties who would be affected by this policy to help identify the data that are most
critical for external access and that can be reasonably subject to data sharing
requirements, as well as to help develop feasible mechanisms to implement a data
sharing policy.
For routinely collected financial and program monitoring data, a limited set of
essential data should be identified and made available for external use in a timely
way.
Evaluation and research reports and publications using data collected through
PEPFAR-supported programs should be tracked and made available in a
publically accessible central repository. U.S. government agencies with similar
repositories can be considered as models.
For research data and other information that is expressly generated for new
knowledge, the policy should respect time-bound exclusivity for the right to
engage in the publication process, yet also ensure the timely availability of data,
regardless of publication, for access and use by external evaluators and
researchers. OGAC should look to U.S. government agencies with similar
research data policies as models.
In developing the policy and specifying the scope of data to be included, several
key factors and potential constraints that can affect the implementation of the
policy will need to be addressed. In particular, these include patient and client
information confidentiality; the financial resources, personnel, and time needed to
make data available; and issues of data ownership, especially in the context of
increasing responsibility in partner countries and provision of PEPFAR support
through country systems or through activities and programs supported by
multiple funding streams.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 495
11
PEPFAR’s Knowledge Management
INTRODUCTION
Knowledge is generated across all levels of PEPFAR from implementing partners,
partner country mission teams, PEPFAR implementing agencies, and OGAC headquarters. This
knowledge, if appropriately synthesized, transferred, disseminated, shared, and used, has the
potential not only to contribute to program improvement and sustainability of PEPFAR’s efforts,
but also to help the global community in its response to the HIV/AIDS epidemic. As the largest
donor currently addressing the global HIV/AIDS epidemic, PEPFAR has both the ability and the
responsibility to play a significant leadership role in this realm.
Knowledge management has been defined by Swan as, “any process or practice of
creating, acquiring, capturing, sharing and using knowledge wherever it resides, to enhance
learning and performance in organizations” (Swan, Scarbrough, & Preston, 1999). Knowledge
management is a strategy used by many organizations to harness and respond to both existing
and created knowledge and has been adopted by organizations such as The World Bank and the
World Health Organization (Loermans, 2002; The World Bank, 2003; WHO, 2005).
An organization that is skilled in knowledge management is able to efficiently and
effectively manage knowledge that has been created (Loermans, 2002). Examples of the types of
knowledge PEPFAR has created and utilized include developing a system for collecting
extensive program monitoring data, supporting epidemiologic and surveillance activities in
partner countries, strengthening partner country health information systems, implementing
various program evaluation approaches, supporting research, and the creation of both tacit and
experiential knowledge as a result of program implementation.
Evaluating PEPFAR’s knowledge management was not an explicit part of the
committee’s congressional mandate, but because availability and access to information was key
to every aspect of this evaluation, the committee felt strongly that to help guide PEPFAR’s future
efforts, examining and making recommendations regarding PEPFAR’s knowledge management
approach was critical. PEPFAR has made strong efforts in generating knowledge, often at a level
not seen in other development programs. Yet, as reflected in prior chapters of this report, there
are key areas where the information needed to assess efforts and guide future activities are
unavailable or insufficient. Significant gaps remain in PEPFAR’s knowledge management
approach, especially in the realms of knowledge creation, dissemination, and utilization, and to
date, OGAC has not articulated a clear and comprehensive strategy for managing knowledge to
optimize PEPFAR’s performance and effectiveness.
PREPUBLICATION COPY: UNCORRECTED PROOFS
11-1
OCR for page 496
11-2 EVALUATION OF PEPFAR
This chapter shifts its focus away from assessing and addressing the limitations in the
available information that affected the committee’s ability to respond to the specific charge
mandated by Congress; these were discussed in Chapter 2. Rather, the aim of this chapter is to
offer an assessment to guide PEPFAR to more strategically and efficiently meet its information
needs going forward. This chapter will review and assess PEPFAR’s current approach to
knowledge management, culminating with recommendations from the committee for future
directions to address current gaps and to strengthen PEPFAR’s ability to generate, share, and
utilize knowledge more effectively.
Strategic Information
PEPFAR articulated a goal of having evidence-based programs from the outset. To meet
this goal, the OGAC Office of Strategic Information (SI), which is responsible for using SI to
guide and coordinate PEPFAR performance planning and reporting, was established (GAO,
2011a). The first Five-Year Strategy defined strategic information as “the systematic collection,
analysis, and dissemination of information about reaching the Emergency Plan’s objectives, as
well as the related programmatic activities funded to reach these goals” (OGAC, 2004, p. 73).
Strategic information was used as an organizing concept because ‘WHO was just starting to use
the term strategic information, and that resonated with us—the use of information for program
improvement and operations—so, we decided to use that name—gathering of information—it
had to be strategic and it had to be used’ (NCV-3). 1,2
Initially, the OGAC SI office had an annual budget of around $33 million for centrally-
funded SI activities but over time, funding has been reduced to less than $10 million annually
(NCV-2-USG). At the partner country level, from FY 2006 to FY 2011, approximately four to five
percent of total funding for PEPFAR partner country activities was budgeted for SI activities,
excluding staff salaries (see Figure 11-1) (OGAC, 2005a, 2006d, 2007f, 2008c, 2010d, 2011f,
2011g). These activities have included monitoring and reporting partner results, as well as
surveillance, surveys, and efforts to strengthen partner country health information systems
(OGAC, 2008a, 2009c, 2010c). Reflecting an increased focus on country ownership, FY 2012
Country Operational Plan (COP) guidance advised mission teams that activities planned under
the SI budget code should aim “to build individual, institutional, and organizational capacity in
country” for strategic information activities (OGAC, 2011e, p. 68).
1
Single quotations denote an interviewee's perspective with wording extracted from transcribed notes written during
the interview. Double quotations denote an exact quote from an interviewee either confirmed by listening to the
audio-recording of the interview or extracted from a full transcript of the audio-recording.
2
Country Visit Exit Synthesis Key: Country # + ES
Country Visit Interview Citation Key: Country # + Interview # + Organization Type
Noncountry Visit Interview Citation Key: “NCV” + Interview # + Organization Type
Organization Types: United States: USG = US Government; USNGO = US Nongovernmental Organization; USPS
= US Private Sector; USACA = US Academia; Partner Country: PCGOV = Partner Country Government;
PCNGO = Partner Country NGO; PCPS = Partner Country Private Sector; PCACA = Partner Country Academia;
Other: CCM = Country Coordinating Mechanism; ML = Multilateral Organization; OBL = Other (non-US and
non-Partner Country) Bilateral; OGOV = Other Government; ONGO = Other Country NGO
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 497
KNOWLEDGE MANAGEMENT 11-3
FIGURE 11-1 PEPFAR funding for country-level strategic information in constant 2010 dollars and as percent of
total PEPFAR funding.
NOTE: This figure represents funding for all PEPFAR countries as planned/approved through PEPFAR’s budget
codes for country-level Strategic Information activities. The budget codes are the only available source of funding
information disaggregated by type of activity, and are therefore used in this report as the most reasonable and
reliable approximation of PEPFAR investment by programmatic area. Data are presented in constant 2010 USD for
comparison over time. See Chapter 4 for a more detailed discussion of PEPFAR’s budget codes and the available
data for tracking PEPFAR funding.
SOURCE: (OGAC, 2005a, 2006d, 2007f, 2008c, 2010d, 2011f, 2011g).
PROGRAM TARGETS AND PRIORITIES
Setting Program Targets
Setting priorities and targets is one important aspect of planning and managing programs.
Subsequently monitoring and assessing progress and performance in meeting these targets is
critical for program management. When PEPFAR was authorized in 2003, it was established
with an emphasis on accountability by setting specific performance targets and recognizing the
necessity of monitoring and evaluation to assess the performance of the PPEFAR-supported
programs. 3 The initial 5-year goals for the 15 focus countries were to provide treatment to 2
million HIV-infected people; prevent 7 million new HIV infections; and provide care to 10
million people infected and affected by HIV/AIDS, including orphans and vulnerable children
(OGAC, 2004). The treatment and care 5-year targets were determined based on 50 percent of
the estimated need for the focus countries with input from economists based at UNAIDS and
NIH (Donnelly, 2012; IOM, 2007) (NCV-2-USG). The 5-year prevention targets were based on cost
3
United States Leadership against HIV/AIDS, Tuberculosis, and Malaria Act of 2003, P.L. 108-25, 108th Cong.,1st
Sess. (May 27, 2003).
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 498
11-4 EVALUATION OF PEPFAR
estimates from UNAIDS and approximately half of the expected new infections in the focus
countries (Donnelly, 2012; IOM, 2007). With reauthorization under the 2008 Lantos-Hyde Act
and ongoing PEPFAR activities, the main cumulative targets for treatment, prevention and care
have increased steadily (see Table 11-1). In December 2011, on World AIDS Day, President
Obama announced an increase in PEPFAR’s target number of people on treatment from 4 to 6
million by the end of 2013 (The White House, 2011).
TABLE 11-1 Key PEPFAR Targets Under Legislation and Strategy Mandates
United States Lantos-Hyde PEPFAR 5-Year Presidential
Leadership Against Reauthorization Strategy Declaration,
HIV/AIDS, Act World AIDS Day,
Tuberculosis, and 2011
Malaria Act of
2003
Date Target Set 2003 2008 2011 December, 2011
Target FY 2004–2008 Through FY 2013 Through FY 2014 End of 2013
Timeframe
Targets Treatment of 2 Treatment of at least Treatment of more Treatment of more
million 3 million than 4 million than 6 million
Prevention of 7 Prevention of 12 Prevention of more
million new million new than 12 million new
infections infections infections
Provision of care to Provision of care to Provision of care to
10 million, including 12 million, including more than 12 million,
OVCs 5 million OVCs including 5 million
OVCs
Training & retention Training & retention of
of 140,000 new more than 140,000
health care workers new health care
workers
SOURCES: (Government; Kaiser Family Foundation, 2012; The White House, 2011).
To accomplish the overall PEPFAR I targets, each partner country mission team was
assigned a target to achieve during the initial 5-year implementation period (OGAC, 2003).
Starting in FY 2009, under PEPFAR II, targets were determined at the partner country level by
PEPFAR mission teams (OGAC, 2008a, 2009c, 2010c, 2011e).
To inform the targets for each upcoming fiscal year, which are determined as part of the
Country Operational Plan (COP) process, mission teams look at programmatic results from
previous years (240-33-USG; 636-1-USG). Ideally, targets should be set based on data, including
estimated need, and in at least one partner country, there appears to have been an evolution
toward increased use of data by mission teams to determine program targets (240-33-USG). However,
the epidemiological data needed to support rational targeting are not always available and there
is variability in the reliability of data that are available (461-16-USG; 461-18-USG). Mission teams
described working closely with implementing partners to set program targets (116-1-USG; 461-16-USG;
461-18-USG). One mission team described the target setting process as
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 499
KNOWLEDGE MANAGEMENT 11-5
‘[We] work with implementing partners to set targets based on the
partners’ budget, disease burden, and previous performance. [We] then
aggregate implementing partner’s targets and adjust for overlap to get the
overall PEPFAR target. Always need to ensure that their target doesn’t
exceed the national number.’ (461-16-USG)
However, the targets are not always realistic and achievable; as one implementing partner
interviewee stated about the organization’s program targets, “[It] feels like being asked to make
an elephant fly” (166-10-USNGO).
Use of Program Targets
Interviewees described using targets for program accountability and planning. At the
headquarters level, OGAC interviewees described comparing data reported by mission teams to
the targets set in the COP (NCV-2-USG; NCV-7-USG). At the partner country level, mission teams used
targets for COP planning and to assess whether implementing partners met their goals. Program
targets were seen as having limited utility for program management by some mission teams (461-
16-USG; 196-1-USG). Other mission teams, however, found the information more useful for program
planning (116-1-USG; 636-1-USG):
‘In particular for the PMTCT and treatment indicators, the
PEPFAR team has had a process to look back at programmatic
results from previous years to inform the targets for the upcoming
fiscal year. These programmatic results are useful when
developing consensus around the targets and planning of the
activities to be implemented in the next year. So, indicator data are
used programmatically to inform the managers on how to
implement the program especially when trying to scale up.’ (636-1-
USG)
OGAC is working toward more closely linking program monitoring targets to financial
information. Initially, targets were set using best guess estimates of what the money could buy
given the costs at the time, without knowing the real costs or knowing what the partner country
health system could absorb, particularly in the areas of treatment and care (NCV-11-USG). In 2012,
OGAC began an expenditure analysis in ten countries to better understand the range of unit
costing for PEPFAR’s core services in order to help mission teams build budgets and more
accurately estimate costs (NCV-11-USG) (C. B. Holmes et al., 2012). This type of expenditure
analysis will become a routine process after this initial study (C. B. Holmes et al., 2012). The
increased emphasis on tying targets to financial cost may be due to the fact that, as one
interviewee put it, ‘the budget now provides constraints and [we] have to really think about how
to leverage resources’ (NCV-2-USG). The targets are ‘more useful and more realistic now’ (NCV-2-
USG).
Interviewees described an inherent tension between trying to meet program targets while
also trying to implement interventions such as investing in quality programs, health systems
strengthening efforts, building capacity, and focusing on prevention. These types of activities
contribute to PEPFAR goals but could result in lower numbers reached for the program targets
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 576
11-82 EVALUATION OF PEPFAR
OGAC’s current tiered program monitoring indicator reporting structure
(illustrated in Figure 11-10) should be further streamlined to report upward only
those indicators essential at each PEPFAR level:
o Tier 1: A small set of core indicators, fewer than the current 25, to be reported
to central HQ level. These data should be used to monitor performance across
PEPFAR as a whole, for congressional reporting, and to document trends; as
such these indicators should remain consistent over time. Whenever possible
and appropriate, these indicators should be harmonized with existing global
indicators and/or national indicators; therefore some centrally reported
indicators will reflect PEPFAR’s contribution rather than aiming to measure
direct attribution.
o Tier 2: A larger menu of indicators defined in OGAC guidance, from which a
subset are selected for their applicability to country programs to be reported
by implementing partners to the U.S. mission teams but not routinely reported
to HQ. These data should be used to monitor the effectiveness of the in-
country response and support mutual accountability with partner countries and
their citizens. These data could be considered for occasional centralized use to
inform special studies or respond to congressional requests but aggregation
and comparability across countries may be limited in this tier as all mission
teams may not collect the same data.
o Tier 3: Indicators selected by implementing partners to monitor and manage
program implementation and effectiveness that are not routinely reported to
mission teams. Implementing partners should select appropriate indicators
defined in OGAC guidance and augment this with other indicators as needed
for their programs. Implementing partners should work with mission teams in
developing their program monitoring plans with selected indicators. Mission
teams should provide oversight and technical assistance to ensure
implementation of these plans and to promote local quality data collection,
use, and mutual accountability. Although not routinely reported, some of these
data could be considered for occasional country-level and centralized use.
o OGAC should create mechanisms for implementing partners, mission teams,
and agency headquarters to mutually contribute to a periodic review across all
tiers of indicator development, applicability, and utility and to make
modifications if necessary.
o Tier 1 indicators should be harmonized whenever possible and appropriate
with existing global indicators and/or national indicators. For indicators that
are not routinely reported centrally (Tiers 2 and 3), country program planning
should facilitate alignment of indicator selection and data collection with
partner country HIV monitoring and health information systems.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 577
KNOWLEDGE MANAGEMENT 11-83
FIGURE 11-10 Recommended PEPFAR Tiered Reporting in the Context of Partner Country and Global Reporting Systems.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 578
11-84 EVALUATION OF PEPFAR
OGAC should complement program monitoring with a unified evaluation
portfolio that includes periodic program evaluation at the PEPFAR country
program and implementing partner levels to assess process, progress, and
outcomes as well as periodic impact evaluations at the country, multi-country, and
headquarters levels.
o OGAC evaluation guidance should provide information about prioritizing
areas for evaluation, the types of evaluation questions, methodologies
guidance, potential study designs, template evaluation plans, examples of key
outcomes, and how evaluation results should be used and disseminated.
PEPFAR should support a range of appropriate methodologies for program
evaluation, including mixed qualitative and quantitative methods, and should
shift emphasis from probability designs to plausibility designs that provide
valid evidence of impact.
o To allow for some comparability across countries and programs, OGAC and
HQ technical working groups should, with input from country teams,
strategically plan and coordinate a subset of evaluations within programmatic
areas that include (but are not limited to) a minimum of centrally-identified
and -defined outcome measures and methodologies.
o Within PEPFAR-supported evaluation activities there should be an emphasis
on the use of in-country local expertise to enhance capacity building for
program evaluation and contribute to country ownership.
For both program monitoring and evaluation OGAC should continue its work on
defining and developing measures to assess progress in the currently under-
measured areas of country ownership, sustainability, gender, policy, capacity
building, and technical assistance.
Further considerations for implementation of Recommendation 11-1B (Research):
OGAC should clearly define what activities and methodologies will be included
under the umbrella of PEPFAR-supported research as distinguished from program
evaluation.
OGAC should draw on input from implementing agencies, mission teams, partner
countries, implementing partners, the Scientific Advisory Board, and other expert
consultations to identify and articulate research priorities and appropriate research
methodologies. The research proposals and funding mechanisms should be
designed to ensure that these priorities are met and methodologies are applied
through RFAs and other investigator-driven research proposals as well as through
targeted solicitations of research in gap areas not met through open requests.
Given PEPFAR’s legislative and programmatic objectives to support research that
assesses program quality, effectiveness, and population-based impact; optimizes
service delivery; and contributes to the global evidence base on HIV/AIDS
interventions and program implementation, at the time of this evaluation the
committee identified the following gaps in PEPFAR’s research activities:
o Behavioral and structural interventions, especially in areas such as prevention,
gender, nonclinical and OVC care and support, and treatment retention and
adherence. These research activities should employ appropriate methodologies
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 579
KNOWLEDGE MANAGEMENT 11-85
and study designs without being unduly limited to random assignment
designs.
o Costs, benefits, and feasibility of integrating gender-focused programs with
clinical and community-based activities.
o Health systems strengthening interventions across the WHO building blocks,
with a prioritized goal of determining setting- and system-specific feasibility,
effectiveness, quality of services, and costs for innovative models.
To contribute to country ownership, PEPFAR should facilitate in-country local
participation and research capacity building through simplified, streamlined, and
transparent application and review processes that encourage submissions from
country-based implementing partners and researchers.
Further considerations for implementation of Recommendation 11-1C (Knowledge
transfer and dissemination):
The knowledge created within PEPFAR that should be more widely documented
and disseminated includes program monitoring data, financial data, research
results, evaluation outcomes, best practices, and informal knowledge such as
implementation experience, lessons learned, and other information.
To institutionalize internal and external knowledge transfer and learning,
PEPFAR should develop appropriate systems and processes for the most needed
types and scale of knowledge transfer. To achieve this, PEPFAR should draw on
broad stakeholder input to assess the strengths and weaknesses in current
processes and to identify needs and opportunities for improved knowledge
transfer.
PEPFAR should invest in innovative mechanisms and technology to facilitate
knowledge transfer across partner countries and implementing partners.
Mechanisms currently used successfully on a small scale and an ad hoc basis
could be more formally scaled up across PEPFAR. OGAC should also look to
other organizations with wide geographic reach and organizational complexity,
such as multi-country PEPFAR implementing partners, other large global health
initiatives, and global corporations, for models of successful knowledge transfer
systems.
OGAC should develop a policy for data sharing and transparency that facilitates
timely access to PEPFAR-created knowledge for analysis and evaluation. The
purpose of this policy would be to ensure that, within a purposefully and
reasonably defined scope, specified program monitoring data and financial data,
evaluation outcomes, and research data and results generated with PEPFAR
support by contractors, grantees, mission teams, and USG agencies be made
available to the public, research community, and other external stakeholders.
OGAC and the PEPFAR implementing agencies should consult with both internal
and external parties who would be affected by this policy to help identify the data
that are most critical for external access and that can be reasonably subject to data
sharing requirements, as well as to help develop feasible mechanisms to
implement a data sharing policy.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 580
11-86 EVALUATION OF PEPFAR
o For routinely collected financial and program monitoring data, a limited set of
essential data should be identified and made available for external use in a
timely way.
o Evaluation and research reports and publications using data collected through
PEPFAR-supported programs should be tracked and made available in a
publically accessible central repository. U.S. government agencies with
similar repositories can be considered as models.
o For research data and other information that is expressly generated for new
knowledge, the policy should respect time-bound exclusivity for the right to
engage in the publication process, yet also ensure the timely availability of
data, regardless of publication, for access and use by external evaluators and
researchers. OGAC should look to U.S. government agencies with similar
research data policies as models.
o In developing the policy and specifying the scope of data to be included,
several key factors and potential constraints that can affect the implementation
of the policy will need to be addressed. In particular, these include patient and
client information confidentiality; the financial resources, personnel, and time
needed to make data available; and issues of data ownership, especially in the
context of increasing responsibility in partner countries and provision of
PEPFAR support through country systems or through activities and programs
supported by multiple funding streams.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 581
KNOWLEDGE MANAGEMENT 11-87
REFERENCES
(IOM), Institute of Medicine. (2007). Preventing HIV Infection among Injecting Drug Users in High Risk
Countries: An Assessment of the Evidence. Washington, DC: National Academies Press.
AIDStar-Two. (2012). Website: OVCSupport.net- A Global Hub on Children and HIV. Retrieved
October 5 2012, 2012, from http://www.ovcsupport.net/s/
Bergmann, Heather. (2011). Field Driven Learning Meeting: Linkages to and Retention in HIV Care and
Support Programs. Arlington, VA: USAID's AIDS Support and Technical Assistance Resources,
AIDSTAR-One, Task Order 1.
Bouey, P; Padian, N. (2011). PEPFAR-funded Evaluations Presentation. Paper presented at the Scientific
Advisory Board Meeting, Washington, DC.
Bryant, M.;, Beard, J.;, Sabin, L;, Brooks, M.I.;, Scott, N.;, Larson, B.A;, . . . Miller, C. (2012).
PEPFAR's Support for Orphans and Vulnerable Children: Some Beneficial Effects, but too Little
Data, and Programs Spread Thin. Health Affairs,, 31(7), 1508-1518.
Donnelly, John. (2012). The President's Emergency Plan for AIDS Relief: How George W. Bush and
Aides Came to 'Think Big' on Battling HIV. Health Affairs,, 31(7), 1389-1396.
Fain, James. (2005). Editorial: Is There a Differene Between Evaluation and Research? The Diabetes
Educator, 31(2), 150-155.
Fullem, Andrew, Marcy Levy, and Melissa Sharer. (2012). Meeting the HIV; Maternal, Newborn, and
Child Health; and Social Support Needs of Mothers and their Young Children. Field Driven
Learning Meeting, Addis Ababa, Ethiopia, November 8 to 10, 2011. Arlington, VA: USAID's
AIDS Support and Technical Assistance Resources, AIDSTAR-One, Task Order 1.
GAO. (2011a). PEPFAR program planning and reporting. GAO-11-785 (pp. 1 online resource (iii, 38 p.) :
ill.). Washington, D.C.: U.S. Government Accountability Office.
GAO. (2011b). Performance Measurement and Evaluation: Definitions and Relationships (Vol. GAO-11-
646SP). Washington, DC: United States Government Accountability Office.
GAO. (2012). President's Emergency Plan For AIDS Relief-Agencies Can Enhance Evaluation Quality,
Planning, and Dissemination: Report to Congressional Committees. Washington, DC: GAO.
Garvin, D. A. (1993). Building a learning organization. Harv Bus Rev, 71(4), 78-91.
Gay, J., Croce-Galis, M., Hardee, K. . (2012). What Works for Women and Girls: Evidence for
HIV/AIDS Interventions. 2nd edition. Retrieved October 5, 2012, from
www.whatworksforwomen.org
Goosby, E. (2012). The President's Emergency Plan For AIDS Relief: Marshalling All Tools At Our
Disposal Toward An AIDS-Free Generation. Health Aff (Millwood), 31(7), 1593-1598. doi:
10.1377/hlthaff.2012.0241
Government, United States. The United States Goverment Global Health Initiative Strategy Document.
Grosso, Ashley L., Hoan Tram, Khai, Ryan, Owen, & Baral, Stefan. (2012). Countries Where HIV Is
Concentrated Among Most-At-Risk Populations Get Disproportionally Lower Funding From
PEPFAR. Health Aff (Millwood), 31(7), 1519-1528. doi: 10.1377/hlthaff.2012.0216
Group, OGAC SAB Data Working. (2011, March 17). SAB Data Working Group (DWG) Conference
Call Notes. from http://www.pepfar.gov/documents/organization/166969.pdf
Habicht, JP, CG Victoria, JP Vaughan. (1999). Evaluation designs for adequacy, plausibility and
probability of public health programme performance and impact. International Journal of
Epidemiology(28).
HHS. (2012). Operations Research (Implementation Science) for Strengthening Program Implementation
through the President's Emergency Plan for AIDS Relief (PEPFAR). Retrieved October, 2012,
from http://www.grants.gov/search/search.do?mode=VIEW&oppId=136553
Holmes, Charles. (2012). Presentation to SAB: Implementation Science Updates. Washington, DC:
Office of U.S. Global AIDS Coordinator.
Holmes, Charles B., Blandford, John M., Sangrujee, Nalinee, Stewart, Scott R., DuBois, Amy, Smith,
Tyler R., . . . Goosby, Eric P. (2012). PEPFAR’S Past And Future Efforts To Cut Costs, Improve
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 582
11-88 EVALUATION OF PEPFAR
Efficiency, And Increase The Impact Of Global HIV Programs. Health Aff (Millwood), 31(7),
1553-1560. doi: 10.1377/hlthaff.2012.0562
Holzscheiter, A;, Walt, G.;, & Brugha, R. (2012). Monitoring and Evaluation in Global HIV/AIDS
Control-Weighing Incentives and Disincentives for Coordination among Global and Local
Actors. Journal of International Development, 24(1), 61-76.
IOM. (2007). PEPFAR Implementation: Progress and Promise. Washington, DC: The National
Academies Press.
IOM (2011, March). [IOM Staff Internal Review of PEPFAR phase I Evaluation-List of Targeted
Evaluations Document].
IOM and NRC. (2010). Strategic Approach to the Evaluation of Programs Implemented under The Tom
Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria
Reauthorization Act of 2008. Washington, DC: The National Academies Press.
JSI. (2012). Website: AIDSTAR-One: AIDS Support and Technical Assistance Resources. Retrieved
October 5, 2012, from http://www.aidstar-one.com/
Kaiser Family Foundation. (2012, June, 2012). FACT SHEET: The U.S. President’s Emergency Plan for
AIDS Relief (PEPFAR). Retrieved September 4, 2012, from
http://www.kff.org/globalhealth/upload/8002-04.pdf
Levin-Rozalis, Miri. (2003). Evaluation and Research: Differences and Similarities. The Canadian
Journal of Program Evaluation, 18(2), 1-31.
Loermans, J. (2002). Synergizing the Learning Organization and Knowledge Management. Journal of
Knowledge Management, 6(3), 285-294.
Lyerla, Rob;, Murrill, Christopher S.;, Ghys, Peter D.;, Calleja-Garcia, Jesus M.;, & Decock, Kevin M.;.
(2012). The Use of Epidemiological Data to Inform the PEPFAR Response. Journal of acquired
immune deficiency syndromes (1999), 60 Suppl 3, S57-62.
Needle, Richard PhD M. P. H., Fu, Joe B. S., Beyrer, Chris M. D. M. P. H., Loo, Virginia PhD, Abdul-
Quader, Abu S., McIntyre, James A. MBChB Frcog, . . . Pick, Billy J. D. (2012). PEPFAR's
Evolving HIV Prevention Approaches for Key Populations-People Who Inject Drugs, Men Who
Have Sex With Men, and Sex Workers: Progress, Challenges, and Opportunities. JAIDS Journal
of Acquired Immune Deficiency Syndromes, 60 Supplement(3), S145-S151.
NIH. (2010). Limited Competition: Administrative Supplements for HIV/AIDS Implementation Science
in PEPFAR Settings. 2012, from http://grants.nih.gov/grants/guide/notice-files/NOT-AI-10-
023.html
NIH. (2011). NIH/PEPFAR Collaboration for Implementation Science and Impact Evaluation (R01).
Retrieved October, 2012, from http://grants.nih.gov/grants/guide/rfa-files/RFA-AI-11-003.html
Office of Learning, Evaluation and Research. (2012). USAID Evaluation Policy: Year One; First Annual
Report and Plan For 2012 and 2013. Washington, DC: USAID.
OGAC. (2003). PEPFAR Country Operational Plan Guidelines for FY04. Washington, DC.
OGAC. (2004). The President's Emergency Plan for AIDS Relief: U.S. Five-Year Global HIV/AIDS
Strategy. Washington, DC: OGAC.
OGAC. (2005a). Emergency Plan for AIDS Relief Fiscal Year 2005 Operational Plan: June 2005 Update.
Washington, DC.
OGAC. (2005b). President's Emergency Plan for AIDS Relief: FY06 Country Operational Plan Final
Guidance. Washington DC: OGAC.
OGAC. (2005c). The President’s Emergency Plan for AIDS Relief: Indicators, Reporting Requirements,
and Guidelines for Focus Countries.
OGAC. (2006a). A Blueprint for Public Health Evaluations in the President's Emergency Plan for AIDS
Relief. Washington, DC: OGAC.
OGAC. (2006b). The President's Emergency Plan for AIDS Relief. FY2007 Supplemental COP Guidance
Resource Guide. Washington, DC: OGAC.
OGAC. (2006c). The President’s Emergency Plan for AIDS Relief: FY2007 Country Operational Plan
Guidance.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 583
KNOWLEDGE MANAGEMENT 11-89
OGAC. (2006d). The U.S. President's Emergency Plan for AIDS Relief Fiscal Year 2006: Operational
Plan. 2006 August Update. Washington, DC.
OGAC. (2007a). Data Quality Assuarance Tool for Program-Level Indicators.
OGAC. (2007b). Factsheet: PEPFAR Extranet (PEPFAR.net). In OGAC (Ed.). Washington DC: US
Department of State, USAID, DoD, DoC, DoL, HHS, Peace Corps.
OGAC. (2007c). The Power of Partnerships: The President's Emergency Plan for AIDS Relief, Third
Annual Report to Congress. Washington, DC.
OGAC. (2007d). The President's Emergency Plan for AIDS Relief: FY2008 Country Operational Plan
Guidance. Washington DC: OGAC.
OGAC. (2007e). The President’s Emergency Plan for AIDS Relief: Indicators, Reporting Requirements,
and Guidelines. Indicators Reference Guide: FY2007 Reporting/FY2008 Planning. Washington,
DC: OGAC.
OGAC. (2007f). The U.S. President's Emergency Plan for AIDS Relief Fiscal Year 2007: Operational
Plan. 2007 June Update. Washington, DC.
OGAC. (2008a). The President’s Emergency Plan for AIDS Relief: FY2009 Country Operational Plan
Guidance Washington, DC: OGAC.
OGAC. (2008b). The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) Fiscal Year 2008:
PEPFAR Operational Plan.
OGAC. (2008c). The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) Fiscal Year 2008:
PEPFAR Operational Plan. June 2008. Washington, DC.
OGAC. (2009a). 2009 HIV/AIDS Implementers' Meeting Program and Abstract Book: Optimizing the
Response, Partnerships for Sustainability. Washington DC.
OGAC. (2009b). Guidance for PEPFAR partnership frameworks and partnership framework
implementation plans. Version 2.0. . Washington, DC: OGAC.
OGAC. (2009c). The President’s Emergency Plan for AIDS Relief: FY2010 Country Operational Plan
Guidance: Programmatic Considerations. Washington, DC: OGAC.
OGAC. (2009d). The President’s Emergency Plan for AIDS Relief: Next Generation Indicators Reference
Guide. Version 1.1. OGAC: Washington, DC.
OGAC. (2009e). The U.S. President's Emergency Plan for AIDS Relief: Five-Year Strategy. Washington,
DC.
OGAC. (2010a). Comprehensive HIV Prevention for People Who Inject Drugs, Revised Guidance.
Washington, D.C.: Office of the U.S. Global AIDS Coordinator.
OGAC. (2010b). The Presiden't Emergency Plan for AIDS Relief: Public Health Evaluation (PHE)
Concept Submission Guidance. Washington, DC: OGAC.
OGAC. (2010c). The President’s Emergency Plan for AIDS Relief: FY2011 Country Operational Plan
Guidance. Washington, DC: OGAC.
OGAC. (2010d). The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) Fiscal Year 2009:
PEPFAR Operational Plan. November 2010. Washington, DC.
OGAC. (2011a). Charter of the President's Emergency Plan for AIDS Relief (PEPFAR) Scientific
Advisory Board. Washington DC: OGAC.
OGAC. (2011b). Email from OGAC to IOM. "Request from IOM Outcome and Impact Evaluation of
PEPFAR Team". Washington, DC.
OGAC. (2011c). List of Continuing PHEs for FY2012. Washington DC: OGAC.
OGAC. (2011d). OGAC Technical Review of Table- "Evolution of PEPFAR Supported Research
Activities". Washington DC.
OGAC. (2011e). The President’s Emergency Plan for AIDS Relief: FY2012 Country Operational Plan
Guidance Appendices. Washington, DC: OGAC.
OGAC. (2011f). The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) Fiscal Year 2010:
PEPFAR Operational Plan. . Washington, DC: OGAC.
OGAC. (2011g). The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) Fiscal Year 2011:
PEPFAR Operational Plan. . Washington, DC: OGAC.
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 584
11-90 EVALUATION OF PEPFAR
OGAC. (2011h). The U.S. President’s Emergency Plan for AIDS Relief: Seventh Annual Report to
Congress. Washington, DC: OGAC.
OGAC (2012a, November 15). [Email Between OGAC Staff and IOM: "Follow-up Queries for IOM
PEPFAR Evaluation"].
OGAC (2012b, August 14, 2012). [Email communication with OGAC staff: "Clarification Regarding
Number of NGIs."].
OGAC (2012c, November 15, 2012). [Email communication with OGAC staff: "Follow-Up Queries for
IOM PEPFAR Evaluation"].
OGAC (2012d, May 2). [Email from OGAC Staff-Technical Review of Evolution of PEPFAR Research
Activities "Re: Request from IOM Outcome and Impact Evaluation of PEPFAR Team"].
OGAC. (2012e). PEPFAR Scientific Advisory Board Working Groups. Retrieved October, 2012, from
http://www.pepfar.gov/sab/workinggroups/index.htm
OGAC. (2012f). Website: PEPFAR Scientific Advisory Board. Retrieved October, 2012, from
http://www.pepfar.gov/sab/
OGAC. (2012g). Website: PEPFAR Scientific Advisory Board Membership List. Retrieved October,
2012, from http://www.pepfar.gov/sab/160067.htm
OGAC. (2012h). Website: The United State's President's Emergency Plan For AIDS Relief. Retrieved
October, 2012, from http://www.pepfar.gov/
OGAC. (n.d.). Partnership Frameworks. Retrieved November 30, 2012, from
http://www.pepfar.gov/countries/frameworks/index.htm
Padian, N. S., Holmes, C. B., McCoy, S. I., Lyerla, R., Bouey, P. D., & Goosby, E. P. (2011).
Implementation science for the US President's Emergency Plan for AIDS Relief (PEPFAR). J
Acquir Immune Defic Syndr, 56(3), 199-203. doi: 10.1097/QAI.0b013e31820bb448
Porter, Laura E. PhD, Bouey, Paul D. PhD, Curtis, Sian PhD, Hochgesang, Mindy M. P. H., Idele,
Priscilla PhD, Jefferson, Bobby B. S., . . . Tulli, Tuhuma M. D. M. P. H. PGdMEM. (2012).
Beyond Indicators: Advances in Global HIV Monitoring and Evaluation During the PEPFAR
Era. JAIDS Journal of Acquired Immune Deficiency Syndromes, 60 Supplement(3), S120-S126.
Reyes, Michael. (2009). Summary of PEPFAR State of the Program Area (SOPA): Care and Support.
Simonds, R.J., Carrino, Constance A., & Moloney-Kitts, Michele. (2012). Lessons From The President’s
Emergency Plan For AIDS Relief: From Quick Ramp-Up To The Role Of Strategic Partnership.
Health Aff (Millwood), 31(7), 1397-1405. doi: 10.1377/hlthaff.2012.0193
Small, Priya. (2012). Four Differences Between Research and Program Evaluation. Retrieved from
http://managementhelp.org/blogs/nonprofit-capacity-building/2012/01/08/four-differences-
between-research-and-program-evaluation/
Spratt, Kai, and Heather Bergmann. (2011). Strengthening Gender Programming in PEPFAR: Technical
Exchange of Best Practices, Program Models and Resources. Arlington, VA: USAID's AIDS
Support and Technical Assistance Resources, AIDSTAR-One, Task Order 1.
Stufflebeam, Daniel. (2007). CIPP Evaluation Model Checklist: A tool for applying the CIPP Model to
assess long-term enterprises (Second Edition ed.). Michigan: Western Michigan University.
Swan, J, Scarbrough, H., & Preston, . (1999). Knowledge Management: The Next Fad to Forget People.
Paper presented at the 7th European Conference on Information Systems, Copenhagen.
The Global Fund to Fight AIDS, Tuberculosis and Malaria. (2011). Monitoring and Evaluation Toolkit:
HIV, Tuberculosis, Malariaand Health and Community Systems Strengthening. Part 2: HIV.
The White House. (2011). FACT SHEET: The Beginning of the End of AIDS. Retrieved September 4,
2012, from http://www.whitehouse.gov/the-press-office/2011/12/01/fact-sheet-beginning-end-
aids
The World Bank. (2003). Sharing Knowledge: Innovations and Remaining Challenges. Washington, DC:
The World Bank.
Thomas, J. C., Curtis, S., & Smith, J. B. (2011). The broader context of implementation science. J Acquir
Immune Defic Syndr, 58(1), e19-21; author reply e21-12. doi: 10.1097/QAI.0b013e31822103e4
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 585
KNOWLEDGE MANAGEMENT 11-91
UNAIDS. (2004). Consultation on Harmonization of International AIDS Funding, End-of-Meeting
Agreement, Washington consulation of April 25, 2004. Geneva.
UNAIDS. (2007). Practical Guidelines for Intensifying HIV Prevention: Towards Universal Access.
Geneva.
UNAIDS. (2009). Monitoring the Declaration of Commitment on HIV/AIDS: Guidelines on Construction
of Core Indicators: 2010 Reporting. Geneva.
USAID. (2011a). Implementation Science Research to Support Programs under the President's
Emergency Plan for AIDS Relief (PEPFAR). Retrieved October, 2012, from
http://www.grants.gov/search/search.do;jsessionid=w6lxTG8QGvxLLXk0BZbXcSl2FrTyCGQL
LPcZV3qXrvyyvsCLFnkC!-25395513?oppId=114673&mode=VIEW
USAID. (2011b). USAID's Implementation Science Investment: Improving HIV/AIDS Programming
through the Translation of Research to Practice. Washington, DC: USAID.
USAID. (2012). Website: Development Experience Clearinghouse. Retrieved October 5, 2012, from
https://dec.usaid.gov/dec/home/Default.aspx
Walport, M., & Brest, P. (2011). Sharing research data to improve public health. Lancet, 377(9765), 537-
539. doi: 10.1016/S0140-6736(10)62234-9
WHO. (2005). World Health Organization Knowledge Management Strategy. Geneva.
WHO. (2012). Voluntary medical male circumcision for HIV prevention. Retrieved September 20, 2012,
from http://www.who.int/hiv/topics/malecircumcision/fact_sheet/en/index.html
PREPUBLICATION COPY: UNCORRECTED PROOFS
OCR for page 586
