Informing Priorities for PEPFAR-Supported Programs
• Despite some exceptions, PEPFAR has implemented evidence-informed programs that have been modified as new knowledge and scientific evidence emerged. Target setting has been used to focus PEPFAR activities, program planning, and accountability. PEPFAR has utilized epidemiological and intervention effectiveness data to drive program activities.
• PEPFAR has provided financial and technical support for collecting epidemiological information in partner countries. This was widely seen as a positive contribution to inform decisions and priorities in planning the HIV/AIDS response and implementing HIV programs, encouraging and facilitating responsiveness to the epidemic and the needs in partner countries.
Program Monitoring Data
• PEPFAR’s program monitoring indicator system has faced technological challenges limiting the ability of both PEPFAR and external stakeholders to utilize and access both current and historical trend data; resolving these challenges is critical for successful program monitoring.
• PEPFAR’s program monitoring has evolved over time: the number of centrally reported indicators was reduced, indicators to monitor new program activities were introduced, and indicators identified as problematic, removed. PEPFAR needs a program monitoring strategy that can adapt over time to respond to feedback, reflect emerging program priorities, and accurately capture program activities and outcomes. However, this needs to be balanced with the reality that changes in indicators place a burden on partner country programs and limit comparability of data, hampering the ability to monitor trends.
• PEPFAR’s current indicators do not capture sufficient information on its stated prioritized goals and activities and are focused primarily on
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 601
11 PEPFAR’s Knowledge Management MAIN MESSAGES Informing Priorities for PEPFAR-Supported Programs • espite some exceptions, PEPFAR has implemented evidence- D informed programs that have been modified as new knowledge and scientific evidence emerged. Target setting has been used to focus PEPFAR activities, program planning, and accountability. PEPFAR has utilized epidemiological and intervention effectiveness data to drive program activities. • EPFAR has provided financial and technical support for collecting P epidemiological information in partner countries. This was widely seen as a positive contribution to inform decisions and priorities in planning the HIV/AIDS response and implementing HIV programs, encourag- ing and facilitating responsiveness to the epidemic and the needs in partner countries. Program Monitoring Data • EPFAR’s program monitoring indicator system has faced techno- P logical challenges limiting the ability of both PEPFAR and external stakeholders to utilize and access both current and historical trend data; resolving these challenges is critical for successful program monitoring. • EPFAR’s program monitoring has evolved over time: the number P of centrally reported indicators was reduced, indicators to monitor new program activities were introduced, and indicators identified as problematic, removed. PEPFAR needs a program monitoring strategy that can adapt over time to respond to feedback, reflect emerging program priorities, and accurately capture program activities and outcomes. However, this needs to be balanced with the reality that changes in indicators place a burden on partner country programs and limit comparability of data, hampering the ability to monitor trends. • EPFAR’s current indicators do not capture sufficient information on P its stated prioritized goals and activities and are focused primarily on 601
OCR for page 601
602 EVALUATION OF PEPFAR input and outputs. As a result, the program monitoring system has limited utility for determining the effectiveness of PEPFAR’s efforts. • he need to quickly measure results at the onset of PEPFAR contrib- T uted to the development of PEPFAR-specific data collection systems, which has limited harmonization with partner countries and the global HIV/AIDS community. More recently, the Office of the U.S. Global AIDS Coordinator (OGAC) has worked with other global actors to harmonize indicators and validate reporting. OGAC has modified the PEPFAR monitoring system to reduce reporting burden and improve alignment with partner country programs; however, further modifica- tions could be made by eliminating PEPFAR-specific language in the indicator guidance; further reducing the reporting burden; improving indicator harmonization with global indicators; and advancing align- ment with partner country data collection at the program level. • here are some good examples of PEPFAR data use at the implement- T ing partner, mission team, and headquarters (HQ) levels, but the pre- ponderance of data collected does not seem to be routinely utilized. PEPFAR’s requirement for collection and reporting of a large amount of program monitoring data places a large burden on implementing partners and mission teams that has limited the ability to analyze and use data. • EPFAR has invested in building the capacity of partner countries to P plan for, collect, manage, and use HIV data, which has implications for the larger health system. As a result, PEPFAR has contributed to fostering a culture of evidence among partner countries. PEPFAR-Supported Evaluation and Research • he manner in which PEPFAR initially approached research activities T was a missed opportunity to establish, from its inception, mechanisms to evaluate programs, assess impact, contribute to the global knowl- edge base, and develop in-country research capacity. • EPFAR has made progress in carrying out evaluation and research P activities over time: moving from an early proscription against re- search, to using Targeted Evaluations and Public Health Evaluations to work within research restrictions, to the recent creation of what holds promise as a more useful process for establishing priorities, managing activities, documenting “what works,” expanding PEPFAR’s technical leadership, disseminating findings, and continually improv- ing the effectiveness and impact of PEPFAR. Defining appropriate and allowable research activities within PEPFAR, however, was and remains a challenge, specifically clarity around the activities and aims for evaluation and research within PEPFAR.
OCR for page 601
PEPFAR’S KNOWLEDGE MANAGEMENT 603 Knowledge Transfer and Learning Within PEPFAR • EPFAR has successfully established and used a variety of mech- P anisms to transfer knowledge throughout PEPFAR; however, more progress is needed to address limitations in current systems and to establish formal mechanisms to systematically transfer experiences across countries, implementing partners, and sites. Without this, there will be missed opportunities to capitalize on best practices and inter- nal lessons learned. Knowledge Dissemination External to PEPFAR • GAC would benefit from developing a formal system to track and O manage PEPFAR-funded dissemination products (e.g., publications, reports, abstracts, guidelines, and tools) from which to measure con- tribution to the global knowledge base, and the global HIV/AIDS community would benefit from a publicly available central repository of these products from which to share, collaborate, and accelerate knowledge creation. • EPFAR has had some success in external dissemination of PEPFAR P knowledge, including establishing formal and informal mechanisms to share knowledge externally and contributing vast amounts of evi- dence and publications to the global knowledge base. Despite this, more progress is needed to develop routine formal mechanisms for knowledge exchange with partner country governments and other partners, increase the amount of PEPFAR data that is publicly avail- able for use by researchers and evaluators, and track and measure PEPFAR’s contribution to the global knowledge base. Overall Conclusion • EPFAR has made progress in managing knowledge by develop- P ing systems for data creation and collection, streamlining program monitoring data, advancing PEPFAR’s role and approach to evalu- ation and research, and utilizing a wide variety of mechanisms to transfer knowledge. Yet, like other entities involved in the global HIV/ AIDS response, it struggles with creating, acquiring, and transferring the right knowledge, at the appropriate scale, and in a manner that facilitates use. PEPFAR has the potential to lead the global HIV/AIDS community in knowledge management by adopting a conceptual framework that articulates the vision, purposes, intended audiences, and goals of knowledge; how knowledge will be acquired, created, transferred, used, and disseminated to achieve these goals; and the complementary roles of program monitoring, evaluation, and research. PEPFAR has the opportunity to optimize program efficiency and ef- fectiveness through an improved strategy that (1) streamlines and focuses knowledge creation within PEPFAR; (2) increases acquisition of knowledge external to PEPFAR; (3) improves the efficiency and ef-
OCR for page 601
604 EVALUATION OF PEPFAR fectiveness of knowledge transfer within and external to PEPFAR; and (4) institutionalizes the use of knowledge to improve the way work is accomplished. Recommendations Presented in This Chapter Recommendation 11-1: The Office of the U.S. Global AIDS Coordina- tor (OGAC) should develop a comprehensive knowledge management framework, including a program monitoring and evaluation strategy, a prioritized and targeted research portfolio, and systems for knowledge dissemination. This framework should adapt to emerging needs to as- sess PEPFAR’s models of implementation and contribution to sustain- able management of the HIV response in partner countries. This knowledge management framework will require that PEPFAR im- plement and strategically allocate resources for the following: A. o better document PEPFAR’s progress and effectiveness, OGAC T should refine its program monitoring and evaluation strategy to streamline reporting and to strategically coordinate a complemen- tary portfolio of evaluation activities to assess outcomes and ef- fects that are not captured well by program monitoring indicators. Efforts should support innovation in methodologies and measures where needed. Both monitoring and evaluation should be specifi- cally matched to clearly articulated data sources, methods, and uses at each level of PEPFAR’s implementation and oversight. B. o contribute to filling critical knowledge gaps that impede effec- T tive and sustainable HIV programs, OGAC should continue to rede- fine permitted research within PEPFAR by developing a prioritized portfolio with articulated activities and methods. The planning and implementation process at the country and program level should inform and be informed by the research portfolio, which should focus on research that will improve the effectiveness, quality, and efficiency of PEPFAR-supported activities and will also contrib- ute to the global knowledge base on implementation of HIV/AIDS programs. C. o maximize the use of knowledge created within PEPFAR, OGAC T should develop systems and processes for routine, active transfer and dissemination of knowledge both within and external to PEPFAR. As one component, OGAC should institute a data-sharing policy, de- veloped through a consultative process. The policy should identify the data to be included and ensure that these stipulated data and results generated by PEPFAR or through PEPFAR-supported activi- ties are made available in a timely manner to PEPFAR stakeholders, external evaluators, the research community, and other interested parties.
OCR for page 601
PEPFAR’S KNOWLEDGE MANAGEMENT 605 Further considerations for implementation of Recommendation 11-1A: Program monitoring and evaluation • GAC’s current tiered program monitoring indicator reporting struc- O ture (illustrated in Figure 11-10) should be further streamlined to report upward only those indicators essential at each PEPFAR level: o ier 1: A small set of core indicators, fewer than the current 25, T to be reported to central HQ level. These data should be used to monitor performance across PEPFAR as a whole, for congressional reporting, and to document trends; as such these indicators should remain consistent over time. Whenever possible and appropriate, these indicators should be harmonized with existing global indica- tors and national indicators; therefore, some centrally reported indicators will reflect PEPFAR’s contribution rather than aim to measure direct attribution. o ier 2: A larger menu of indicators defined in OGAC guidance, T from which a subset are selected for their applicability to country programs to be reported by implementing partners to the U.S. mis- sion teams but not routinely reported to HQ. These data should be used to monitor the effectiveness of the in-country response and to support mutual accountability with partner countries and their citizens. These data could be considered for occasional centralized use to inform special studies or respond to congressional requests but aggregation and comparability across countries may be limited in this tier as all mission teams may not collect the same data. o ier 3: Indicators selected by implementing partners to monitor T and manage program implementation and effectiveness that are not routinely reported to mission teams. Implementing partners should select appropriate indicators defined in OGAC guidance and augment these with other indicators as needed for their pro- grams. Implementing partners should work with mission teams in developing their program monitoring plans with selected in- dicators. Mission teams should provide oversight and technical assistance to ensure implementation of these plans and to pro- mote local quality data collection, use, and mutual accountability. Although not routinely reported, some of these data could be considered for occasional country-level and centralized use. o GAC should create mechanisms for implementing partners, mis- O sion teams, and agency headquarters to mutually contribute to a periodic review across all tiers of indicator development, applica- bility, and utility and to make modifications if necessary. o ier 1 indicators should be harmonized whenever possible and ap- T propriate with existing global indicators and national indicators. For indicators that are not routinely reported centrally (Tiers 2 and 3), country program planning should facilitate alignment of
OCR for page 601
606 EVALUATION OF PEPFAR indicator selection and data collection with partner country HIV monitoring and health information systems. • GAC should complement program monitoring with a unified evalua- O tion portfolio that includes periodic program evaluation at the PEPFAR country program and implementing partner levels to assess process, progress, and outcomes as well as periodic impact evaluations at the country, multi-country, and headquarters levels. o GAC evaluation guidance should provide information about pri- O oritizing areas for evaluation, the types of evaluation questions, methodological guidance, potential study designs, template evalu- ation plans, examples of key outcomes, and how evaluation results should be used and disseminated. PEPFAR should support a range of appropriate methodologies for program evaluation, including mixed qualitative and quantitative methods, and should shift em- phasis from probability designs to plausibility designs that provide valid evidence of impact. n o allow for some comparability across countries and programs, T OGAC and HQ technical working groups should, with input from country teams, strategically plan and coordinate a subset of evaluations within programmatic areas that include (but are not limited to) a minimum set of centrally identified and defined outcome measures and methodologies. n ithin PEPFAR-supported evaluation activities there should W be an emphasis on the use of in-country local expertise to en- hance capacity building for program evaluation and contribute to country ownership. • or both program monitoring and evaluation OGAC should continue F its work on defining and developing measures to assess progress in the currently under-measured areas of country ownership, sustain- ability, gender, policy, capacity building and technical assistance. Further considerations for implementation of Recommendation 11-1B: Research • GAC should clearly define which activities and methodologies will O be included under the umbrella of PEPFAR-supported research, as distinguished from program evaluation. • GAC should draw on input from implementing agencies, mission O teams, partner countries, implementing partners, the Scientific Ad- visory Board, and other experts to identify and articulate research priorities and appropriate research methodologies. The research pro- posals and funding mechanisms should be designed to ensure that these priorities are met and that methodologies are applied through requests for applications and other investigator-driven research pro-
OCR for page 601
PEPFAR’S KNOWLEDGE MANAGEMENT 607 posals as well as through targeted solicitations of research in gap areas not met through open requests. • iven PEPFAR’s legislative and programmatic objectives to support G research that assesses program quality, effectiveness, and population- based impact; optimizes service delivery; and contributes to the global evidence base on HIV/AIDS interventions and program imple- mentation, at the time of this evaluation the committee identified the following gaps in PEPFAR’s research activities: o ehavioral and structural interventions, especially in areas such as B prevention, gender, nonclinical care and support, care and support for orphans and vulnerable children, and treatment retention and adherence. These research activities should employ appropriate methodologies and study designs, without being unduly limited to random assignment designs. o osts, benefits, and feasibility of integrating gender-focused pro- C grams with clinical and community-based activities. o ealth systems strengthening interventions across the World H Health Organization building blocks, with a prioritized goal of de- termining setting- and system-specific feasibility, effectiveness, quality of services, and costs for innovative models. • o contribute to country ownership, PEPFAR should facilitate in- T country local participation and research capacity building through simplified, streamlined, and transparent application and review pro- cesses that encourage submissions from country-based implementing partners and researchers. Further considerations for implementation of Recommendation 11-1C: Knowledge transfer and dissemination • he knowledge created within PEPFAR that should be more widely T documented and disseminated includes program monitoring data, financial data, research results, evaluation outcomes, best practices, and informal knowledge such as implementation experience, and les- sons learned. • o institutionalize internal and external knowledge transfer and learn- T ing, PEPFAR should develop appropriate systems and processes for the most needed types and scale of knowledge transfer. To achieve this, PEPFAR should draw on broad stakeholder input to assess the strengths and weaknesses in current processes and to identify needs and opportunities for improved knowledge transfer. • EPFAR should invest in innovative mechanisms and technology to P facilitate knowledge transfer across partner countries and implement- ing partners. Mechanisms currently used successfully on a small scale and an ad hoc basis could be formally scaled up across PEPFAR.
OCR for page 601
608 EVALUATION OF PEPFAR OGAC should also look to other organizations with wide geographic reach and organizational complexity, such as multi-country PEPFAR implementing partners, other large global health initiatives, and global corporations, for models of successful knowledge transfer systems. • GAC should develop a policy for data sharing and transparency that O facilitates timely access to PEPFAR-created knowledge for analysis and evaluation. The purpose of this policy would be to ensure that, within a purposefully and reasonably defined scope, specified pro- gram monitoring data and financial data, evaluation outcomes, and research data and results generated with PEPFAR support by contrac- tors, grantees, mission teams, and U.S. Government (USG) agencies be made available to the public, research community, and other external stakeholders. OGAC and the PEPFAR implementing agencies should consult with both internal and external parties who would be affected by this policy to help identify the data that are most critical for exter- nal access and that can be reasonably subject to data-sharing require- ments, as well as to help develop feasible mechanisms to implement a data- sharing policy. o or routinely collected financial and program monitoring data, a F limited set of essential data should be identified and made avail- able for external use in a timely way. o valuation and research reports and publications using data col- E lected through PEPFAR-supported programs should be tracked and made available in a publicly accessible central repository. USG agencies with similar repositories can be considered as models. o or research data and other information that is expressly gener- F ated for new knowledge, the policy should respect time-bound exclusivity for the right to engage in the publication process, yet also ensure the timely availability of data, regardless of publication, for access and use by external evaluators and researchers. OGAC should look to USG agencies with similar research data policies as models. o n developing the policy and specifying the scope of data to be in- I cluded, several key factors and potential constraints that can affect the implementation of the policy will need to be addressed. These include patient and client information confidentiality; the financial resources, personnel, and time needed to make data available; and issues of data ownership, especially in the context of increasing re- sponsibility in partner countries and the provision of PEPFAR sup- port through country systems or through activities and programs supported by multiple funding streams.