Strategic Approach to the Evaluation of Programs Implemented Under the Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008 (2010) / Chapter Skim
Currently Skimming:
PART II Proposed Evaluation Approach
PART II Proposed Evaluation Approach
Pages 17-48
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 17... ...
PART II Proposed Evaluation Approach
|
From page 19... ...
Consistent with the clarified congressional intent, U.S. contributions to the Global Fund that are not a part of activities jointly funded or implemented by PEPFAR will not be the focus of the evaluation, and the evaluation will not compare the performance of bilateral PEPFAR programs to that of Global Fund programs (Bressler, 2009; Marsh, 2009)
|
From page 20... ...
During this operational planning phase, IOM staff, planning committee members, and consultants will carry out activities to further develop and refine the plan described here. These activities, which will inform the implementation of the evaluation, are described as part of the work plan later in this section.
|
From page 21... ...
The subsequent sections in Part III of this report address specific components of the evaluation in greater detail. Evaluation Goals and Assumptions The planning committee understood the mandate from Congress as a charge to develop a plan to assess the program with two primary goals.
|
From page 22... ...
Conceptual Framework for Evaluation Design The planning committee developed an overall conceptual framework for the evaluation, which calls for the use of a program impact pathway to guide an assessment of the contribution of PEPFAR to changes in health impact within the context of multiple international and national funding streams. This program impact pathway, described in more detail below, illustrates the committee's understanding of how PEPFAR programs are currently structured and intended to ultimately translate into health impacts, laying out a plausible pathway for causal effects.
|
From page 23... ...
These are described in Part III of this report, along with illustrative evaluation questions based on the committee's interpretation of its charge to assess PEPFAR's performance and impact. All of the specific program impact pathways are oriented to describe outcomes that contribute to the HIV-related health impacts shown in Figure 3, which represent the stated overall goals of PEPFAR.
|
From page 24... ...
24 FIGURE 3 Program impact pathway for evaluation of PEPFAR's effects on HIV-related health impact for children and adults. In the case of joint PEPFAR and Global Fund programs, some inputs may be provided by the Global Fund.
|
From page 25... ...
Therefore, although the ideal goal in a program impact assessment may be to determine to what extent a desired outcome can be attributed directly to the program or policy investment, the realities of a large-scale program such as PEPFAR can make it difficult to determine the extent to which successes or failures in achieving the intended effect can be attributed directly to the program. Thus, the aim of this proposed evaluation approach is not to attempt to determine the direct attribution of PEPFAR funds to health outcomes.
|
From page 26... ...
26 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS FIGURE 4 Context for PEPFAR program implementation.
|
From page 27... ...
In the mixed methods approach described in this evaluation plan some limitations with readily obtained impact data may be overcome by using other proposed comparison methodologies, ancillary studies, key informant interviews, and site visits. These methods are described in more detail in the subsequent sections on evaluation methodologies and data sources.
|
From page 28... ...
This is especially true in some critical areas where consensus indicators have yet to be developed, such as health systems strengthening, integration of services, and country ownership. Finally, there are also evaluation questions the committee will consider that will simply not be possible to answer in the forthcoming evaluation period.
|
From page 29... ...
. Behavioral interventions for HIV prevention have also been evaluated using randomized trials conducted in a variety of venues with a diversity of populations in sub-Saharan Africa (Cornman et al., 2008; Jemmott et al., in press; Jewkes et al., 2008; Kalichman et al., 2008; Stanton et al., 1998)
|
From page 30... ...
However, this method requires that data collection be repeated reliably over time, which is often not the case for many data sources. The challenge for evaluating a program such as PEPFAR, which is widely implemented at the national level across many countries, is that it can be very difficult to identify an appropriate comparison or control to use in the kinds of comparison approaches described previously.
|
From page 31... ...
. In addition to modeling of costs and resource needs, mathematical modeling to estimate infections averted is of particular relevance because HIV incidence is difficult to measure directly.
|
From page 32... ...
follows a disease-modeling approach in which expected or baseline HIV incidence estimates are developed with data prior to 2005 and compared to re-estimated trends in HIV incidence from new surveillance data available after 2004. The difference in the number of new infections, based on this comparison approach, is used as the number of infections averted.
|
From page 33... ...
Country studies will also provide the necessary data collection to make the cross-country comparisons described later. The main component of these country studies will be country data sets that will be compiled for each of the current PEPFAR partner countries using key indicators gathered from OGAC and other available data sources as well as document review (see Boxes 2a–d)
|
From page 34... ...
Therefore, the committee will seek to maximize engagement within the bounds of what will realistically be feasible. Pilot country visits during the operational planning phase will provide an opportunity to explore and assess the feasibility of engaging country experts in the evaluation process.
|
From page 35... ...
A primary purpose of the country visits will be to obtain qualitative data, including semi-structured interviews with key informants and observational data. This is particularly critical for the committee to collect information on process questions related to the implementation of PEPFAR programs, including barriers to implementation, harmonization with national plans, and indirect or unintended effects as observed by local authorities and implementing partners.
|
From page 36... ...
The goal of this comparison approach would be to determine if changes in key indicators in a PEPFAR country are associated with variables such as the timing of the introduction of PEPFAR into that country; the scale of the PEPFAR presence in that country (as measured, for example, by the extent of funding and activities) ; or the operational infrastructure (including, for example, how PEPFAR funding is distributed among implementing partners or the extent to which PEPFAR activities are parallel to or integrated within public sector health services)
|
From page 37... ...
Adjusted analyses could then be performed that correlate those changes in key indicators with explanatory variables such as the duration of time PEPFAR had been present in the country, the cumulative PEPFAR investment in the country up to that point in time, and differing PEPFAR implementation strategies. The committee recognizes that there are critical differences among PEPFAR countries with respect to demographics, social and economic factors, and the epidemiology of the epidemic that must be taken into account in these analyses.
|
From page 38... ...
The before-after percentage changes in PEPFAR countries will be compared to non-PEPFAR countries, adjusting for differences in baseline variables and taking into account HIV activities supported from other sources. As with the comparisons among countries within PEPFAR, a major limitation is that there are important uncertainties in some of the key benchmark indicators used as the dependent variables, such as HIV prevalence and numbers of HIV-related deaths.
|
From page 39... ...
Some of these data sources will also be mapped for non-PEPFAR countries to inform the feasibility of the comparison approaches described earlier -- these approaches would rely on the availability of data from these countries, and on the willingness and capacity of stakeholders in non-PEPFAR countries to participate in country visits and other data-gathering requests. This mapping of available data will also include an assessment of the feasibility of collecting data from each source, taking into consideration the burden that additional data requests would place on each source's resources and staff time.
|
From page 40... ...
Guidance for field programs includes both formal guidance documents and other communications from headquarters to implementing partners and country teams. 5-year strategic plans Country Operational Plan guidance PEPFAR indicators reference guide (including the Next Generation Indicators Reference Guide)
|
From page 41... ...
Government publicly available information and PEPFAR's website (www.pepfar.gov)
|
From page 42... ...
National Health Accounts Progress reports including technical support grants from PEPFAR Key performance indicators Global Fund five-year evaluation: Study area 3 reporta Global Fund evaluation country case studies
|
From page 43... ...
43 PROPOSED EVALUATION PLAN The World Bank Public expenditure reviews Project documents Analytic work/research Health and HIV/AIDS project evaluations Evaluation of HIV/AIDS support Bank-wideb Country assistance evaluations Organisation for Economic Co-operation and Development (OECD) HIV-related funding data Country surveys Evaluation studies Other Multilateral or International Data Sources: aids2031 reports and working papers Committee on the Rights of the Child, States reports Millennium Development Goals reports UNITAID reports European HIV/AIDS Funders Group Interagency Group for Mortality Estimation Funders Concerned about AIDS SOURCE: Compiled from the Global Fund, OECD, UNAIDS, UNICEF, WHO, and World Bank publicly available information and personal communications with individuals at these organizations.
|
From page 44... ...
HIV Programs costing data Other communications among country teams and implementing partners National Policy Documents and other National AIDS Response Information: Relevant national policy documents, strategies, and plans of action supporting PEPFAR activities and/or beneficiaries of PEPFAR-funded activities. National AIDS Coordinating Authority's strategy and framework Agencies or departments policy documents and plans (e.g., Ministries of Health, Finance)
|
From page 45... ...
Existing Modeling Data Sources for HIV Infections Averted: Since the numbers of HIV infections averted due to the implementation of a specific intervention(s) cannot be measured directly, modeling approaches provide a proxy to measure impact (e.g., models that estimate the efficacy of the intervention on incident HIV infection, models that describe how HIV infections are mediated by behavior, and models that compare incidence trends with the expected or baseline HIV incidence trends)
|
From page 46... ...
The operational planning will focus on data mapping (sources and availability of relevant data) ; mapping of methods and data sources, including key indicators, to the mandated tasks and illustrative questions in order to refine and prioritize key evaluation questions and identify key indicators; developing procedures for data requests; initiating data requests; designing and initiating data quality review methods for data received; refining and testing country visit selection criteria; preparing country timelines and other background materials for PEPFAR countries, and developing country study frameworks
|
From page 47... ...
In light of this, OGAC agreed to partner with the IOM to help facilitate, to the extent they are able, access to these data by making introductions to field, headquarters, and agency staff and by disseminating information about the purpose of the evaluation. An initial introduction was sent in a News-to-the-Field posting from OGAC on June 4, 2010, which explained the mandate for the study, the progress of the planning committee as of the posting date, the proposed data-mapping activities and pilot country visits during the operational planning phase, and that the IOM data requests, country visits, site visits, and interview would be entirely independent of the relationship of implementing partners and other country-level stakeholders with OGAC and other USG implementing agencies.
|
From page 48... ...
A summary schematic of the proposed work plan and timeline for the evaluation can be found in Appendix F Adjustments may be needed to the timeline and work plan due to any delay in the start time of the evaluation phase or to uncontrollable external shocks such as man-made or natural disasters (e.g., Haitian earthquake)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.