Strategic Approach to the Evaluation of Programs Implemented Under the Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008 (2010) / Chapter Skim
Currently Skimming:
PART III Illustrative Evaluation Details for Assessment of PEPFAR's Performance and Impact
PART III Illustrative Evaluation Details for Assessment of PEPFAR's Performance and Impact
Pages 49-132
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 49... ...
PART III Illustrative Evaluation Details for Assessment of PEPFAR's Performance and Impact
|
From page 51... ...
The guiding framework of the program impact pathway is applied to each of these areas, reflecting the committee's understanding of the rationale for how PEPFAR's specific inputs and activities can be plausibly linked to PEPFAR's contribution to effects on HIV-specific health outcomes and impacts. This part of the report illustrates the types of questions that will guide the evaluation of PEPFAR's activities in prevention, adult and pediatric treatment, care and support, child and adolescent wellbeing (including orphans and vulnerable children)
|
From page 53... ...
. During PEPFAR II, the SI goals also include improved harmonization of USG reporting needs with country-driven M&E efforts through not only strengthening country capacity and alignment with national data collection, but also through better alignment with global reporting requirements to lessen the burden on implementing partners and partner governments (OGAC, 2009h)
|
From page 54... ...
Within headquarters there is also a SI technical working group. These efforts at the headquarters level also support M&E activities at the country level through the development of resources such as an M&E Systems Strengthening Tool, which is designed to help partner countries prioritize their M&E needs and encourage alignment with a national M&E strategy, and a Data Quality Assessment Tool (MEASURE Evaluation, 2007; PEPFAR, 2008a)
|
From page 55... ...
. Some research activities are already occurring in individual partner countries.
|
From page 56... ...
. Illustrative Questions The evaluation of PEPFAR's SI activities will be carried out at the level of headquarters and in those partner countries where PEPFAR has made major investment in M&E.
|
From page 57... ...
The following are examples of illustrative questions that the committee may consider in the evaluation. These questions related to knowledge management reflect a fundamental activity of the program and as such are intended to contribute to addressing all of the areas for consideration in the congressional mandate, as described in the Statement of Task (see Appendix A)
|
From page 58... ...
Congress) on continuation of PEPFAR funding and on decision making for priority investments as well as program management and improvement?
|
From page 59... ...
What mechanisms are used by PEPFAR for M&E capacity building and to ensure effective partnerships for technical cooperation and technology transfer? What are the effects of the PEPFAR M&E capacity building activities on national M&E system strengthening and data use for decision making?
|
From page 61... ...
Figure 7 represents the committee's initial understanding of the landscape of PEPFAR funding flows, which may change as more information is accumulated throughout the evaluation.
|
From page 62... ...
government. SOURCE: Committee assessment based on a review of documents from OGAC, including PEPFAR operational plans and partner information, as well as other readily available sources.
|
From page 63... ...
It will include other donor governments, multilateral institutions, partner country governments and households, and the private sector.
|
From page 64... ...
Multilateral Bilateral Country Global Fund World Bank CIDA DFID Angola X X X Botswana Cambodia X X X China X X Congo, Democratic Republic of the X X X Côte d'Ivoire X X X Dominican Republic X X Ethiopia X X X X Ghana X X X X Guyana X X Haiti X X India X X X X Indonesia X X Kenya X X X X Lesotho X X X Malawi X X X X Mozambique X X X X Namibia X X Nigeria X X X Russia X X Rwanda X X X X South Africa X X X Sudan X X X Swaziland X X X Tanzania X X X X Thailand X Uganda X X X X Ukraine X X Vietnam X X X Zambia X X X Zimbabwe X X X NOTE: An X denotes that a country receives or has recently received approval for an HIV/AIDS grant from the Global Fund, or has at least one currently operational HIV/AIDS program funded by the World Bank (excluding IBRD-financed loans) , the Canadian International Development Agency (CIDA)
|
From page 65... ...
Illustrative Questions The committee will consult a variety of data sources including PEPFAR and USG agencies (e.g., OGAC databases, COPs, partnership frameworks, prime and sub-prime partner reports, etc.) , other donors and international stakeholders (UNAIDS, the World Bank, the Global Fund, Organisation for Economic Co-operation and Development [OECD]
|
From page 66... ...
Who are the recipients of PEPFAR funding at each stage of program? As depicted in Figure 8, PEPFAR channels funding through governments, NGOs, academic institutions, and private contractors that may be based in the United States, in partner countries, or in other countries.
|
From page 67... ...
How are PEPFAR funds distributed among different programmatic areas and interventions? How are funds from PEPFAR and the Global Fund meshed together in countries?
|
From page 68... ...
The number and types of populations supported by PEPFAR funding will be measured through quantitative analysis of PEPFAR indicators. Aggregated data reported to OGAC provide limited insight about the types of populations supported, so the committee will seek disaggregated data from other sources.
|
From page 69... ...
The are several targets for PEPFAR II: to support the prevention of more than 12 million infections over the course of the entire program (since 2003) , 18 to ensure that every partner country with a generalized epidemic has both 80 percent coverage of testing for pregnant women at the national level19 and 85 percent coverage of ARV drug prophylaxis and treatment of women found to be HIV-infected (OGAC, 2009g)
|
From page 70... ...
. In addition to the performance targets, PEPFAR II has also established additional overarching goals for prevention, such as strengthening partner country capacity to generate and use timely, accurate, and up-to-date epidemiological information (emphasizing prevention efforts targeted toward most vulnerable populations)
|
From page 71... ...
. OGAC reports that this lifted ban creates an opportunity for PEPFAR's MARP TWG to explore ways to support NSPs as part of a comprehensive package of services for IDUs and to develop a revised set of comprehensive guidance for PEPFAR programs and teams so that NSPs can augment prevention, treatment, and care activities already provided to this population (Personal communication from OGAC, January 8, 2010)
|
From page 72... ...
. Program Impact Pathway PEPFAR's prevention efforts will be assessed in accordance with a program impact pathway framework, as shown in Figure 9.
|
From page 73... ...
FIGURE 9 Program impact pathway for evaluation of PEPFAR's prevention programs. NOTES: ARVs = antiretroviral drugs, IDU = injecting drug user, PMTCT = prevention of mother-to-child transmission.
|
From page 74... ...
This will be done by relying on publicly available data sets from DHS and other population surveys as well as qualitative data gathered through site visit interviews. PEPFAR indicators and programmatic guidance related to prevention efforts will be coupled with data from other sources on the nature of the national epidemics and on internationally-accepted best practices for HIV prevention.
|
From page 75... ...
. Within each country, have HIV prevention programs reached the performance targets for prevention for both PEPFAR I and II?
|
From page 76... ...
Implementation and tracking of biomedical prevention programs varies significantly and targets a broad range of individuals at all ages. Data for the evaluation of biomedical prevention interventions will be obtained from aggregated OGAC level indicators and other sources at the country and local levels, as there are not reported PEPFAR indicators for blood safety, PEP, and male circumcision.
|
From page 77... ...
The committee may also attempt to triangulate, where available, the information collected with indicators on PMTCT services coverage and ART coverage for women in specific regions, in order to gain insight into PEPFAR's progress toward its goal of 85 percent prophylaxis and treatment of pregnant women found to be HIVpositive. Has PEPFAR implemented effective prevention programs for injecting and other drug users (behavioral, biomedical, and structural interventions and approaches)
|
From page 79... ...
. As of September 30, 2009, PEPFAR has contributed to the support of treatment for 2,485,300 people, of which 8 percent are children under 15 years of age, through bilateral programs in countries (OGAC, 2010)
|
From page 80... ...
. This overlap estimate also is included in the treatment results reported by the Global Fund (2.5 million individuals receiving ART with support from the Global Fund)
|
From page 81... ...
will depend on the ability to identify women and children routinely through maternal-child care service entry points. For example, scaling up optimal implementation of PMTCT programs not only can improve treatment for women by providing HIV-positive pregnant women with ART, but also has the potential to identify and increase access for children in need of HIV treatment services.
|
From page 82... ...
. Under PEPFAR's laboratory program, PEPFAR's investments have provided support for the monitoring of drug resistance, which included "establishing HIV drug resistance testing capacity" and training of people from PEPFAR countries "on monitoring the emergence of drug resistance mutations in antiretroviral drug treated populations" (OGAC, 2009c, p.
|
From page 83... ...
. Since the United States alone cannot sustain ART for the millions of PLWHA, PEPFAR's focus under its new strategy is to support countries in discussing what resources are needed to respond to the HIV/AIDS epidemic, how to prioritize the large unmet need for treatment of adults, adolescents, and children, and how to identify resources for the gap -- PEPFAR plans to support these efforts in a five-year plan through partnership frameworks with partner countries.
|
From page 84... ...
Given the timely availability of quality data, the committee has developed examples of illustrative questions that it could address during the evaluation of PEPFAR. The committee developed a separate impact pathway for evaluating pediatric treatment considering the differences with adult treatment, including different points of entry for pediatric treatment services and socio-cultural barriers associated with the effective delivery of pediatric treatment by clinicians and non-clinicians, as well as the availability of drug formulations for children or appropriate use of adult preparations when substitutions are necessary (see Figure 11)
|
From page 85... ...
FIGURE 10 Program impact pathway for evaluation of PEPFAR's adult treatment program. NOTE: CD4 = cluster of differentiation 4.
|
From page 86... ...
86 FIGURE 11 Program impact pathway for evaluation of PEPFAR's pediatric treatment programs. NOTES: ARV = antiretroviral drugs, ART = antiretroviral therapy, CD4 = cluster of differentiation 4, MCH = maternal and child health, mo = months old, yrs = years old.
|
From page 87... ...
What is the financial sustainability of existing and expanded coverage with treatment services, specifically ART? To what extent do PEPFAR activities or PEPFAR/Global Fund joint activities assist countries in determining and projecting costs for existing and expanded coverage with treatment services including ART?
|
From page 88... ...
How are these data used for policymaking decisions about selection of treatment regimens? PEPFFAR supports the WHO's strategy on HIV drug resistance surveillance and monitoring, and has provided support to build drug resistance testing capacity in PEPFAR countries (OGAC, 2009c)
|
From page 89... ...
. Components of PEPFAR Care and Support Services The latest PEPFAR-issued guidance for care and support services, "Guidance for United States Government In-Country Staff and Implementing Partners for a Preventive Care Package for Adults," was released in 2006 and describes a menu of preventive care services for adults (OGAC, 2006b)
|
From page 90... ...
NOTES: HPV = human papillomavirus; ITNs = insecticide-treated nets; PCP = Pneumocystis jiroveci pneumonia; STI = sexually transmitted infections; TB = tuberculosis.
|
From page 91... ...
. In order to identify people who have HIV infection, PEPFAR efforts have focused on the scale-up of same-day HIV testing at TB clinics.
|
From page 92... ...
. Food and Nutrition Support Services HIV infection may cause or intensify malnutrition by reducing appetites, increasing energy needs, and impairing nutrient absorption in PLWHA (OGAC, 2009c)
|
From page 93... ...
The components of the preventive care package recommended by PEPFAR address health promotion, and prevention efforts are discussed in this report's prevention section. PEPFAR is working with country governments to develop "policies that address the drivers of the epidemic in country and provide equitable access to quality services for marginalized populations" and expanding "linkages to multiple primary and specialty health services" which "increases community-level access to quality care and reduces the stigma associated with HIV" (OGAC, 2009i, pp.
|
From page 94... ...
Data on 5-year survival rates is very limited for any population; unless these data are being collected and can be accessed, it will not be possible to assess PEPFAR's impact on 5-year survival rates among people not yet eligible for ART. Program Impact Pathway To assess whether PEPFAR has achieved targets regarding care and support services, the committee will examine PEPFAR's care and support activities and use output and outcome indicators (e.g., number of people accessing care services, percent of HIV-positive patients provided CTX prophylaxis)
|
From page 95... ...
FIGURE 13 Program impact pathway for evaluation of PEPFAR's care and support programs. NOTES: ART = antiretroviral therapy; HBC = home-based care workers; HCW = heath care workers; ITNs = insecticide-treated nets; OIs = opportunistic infections; TB = tuberculosis.
|
From page 96... ...
28 What populations are accessing care and support services? The number and types of populations accessing care will be measured through quantitative analysis of PEPFAR indicators.
|
From page 97... ...
PEPFAR indicators measure the number of people receiving food or nutrition services and the percent of HIV-positive clinically malnourished patients receiving food. The committee will use aggregated data reported to OGAC and seek disaggregated data from organizations and initiatives to which PEPFAR links or refers HIV-positive patients for nutritional support.
|
From page 99... ...
. In 2007, the number of children and adolescents aged 0–17 years old who have lost one or both parents as a result of the AIDS epidemic 30 was estimated to be approximately 15 million worldwide -- in sub-Saharan Africa alone, the number was estimated to be approximately 12 million children (UNAIDS, 2008)
|
From page 100... ...
. Beyond biological susceptibility to HIV, socio-cultural factors that contribute to the vulnerability of young women to sexually transmitted HIV infection include entrenched gender roles, unbalanced power relations, sexual violence (such as coerced sex)
|
From page 101... ...
. In FY2009, PEPFAR supported care for 3,620,140 million children and adolescents through its OVC programming, and provided pediatric treatment for 201,500 individuals under 15 years of age (about 8 percent of the total number of people receiving ART with direct PEPFAR support)
|
From page 102... ...
Work with countries to increase support for family-based care by establishing and strengthening linkages between clinical and home- and community-based care. Scale up and ensure robust monitoring of existing high-impact OVC programs and support countries in developing, implementing, and evaluating innovative OVC pilot programming.
|
From page 103... ...
. Similar to other bilateral and multilateral stakeholders, such as the Global Fund, PEPFAR II strategic plans include expanding access to PMTCT services "as a mechanism to both prevent transmission of HIV to children and support expanded access to care and related services for pregnant women" (OGAC, 2009i, p.
|
From page 104... ...
. PEPFAR's "Guidance for United States Government In-Country Staff and Implementing Partners for a Preventive Care Package for Children Aged 0–14 Years Old Born to HIV-Infected Mothers - #1," which was released in April 2006, describes PEPFAR-supported activities for children born to HIV-infected mothers (HIV-exposed children)
|
From page 105... ...
PEPFAR also supports OVC programs "that link OVC services with HIV-affected families [through] linkages with PMTCT, palliative care, treatment" (OGAC, 2009e, p.
|
From page 106... ...
. Food and nutritional support OVC programs include nutritional assessments and counseling, therapeutic or supplementary feeding and micronutrient supplementation for HIV-infected children based on national and appropriate international guidelines, and replacement feeding and support for children born to HIV-infected mothers (OGAC, 2006e)
|
From page 107... ...
. PEPFAR programs provide health care to HIV-positive children, including HIV-exposed children, through direct access to health providers, or through referrals to prevention and treatment services.
|
From page 108... ...
Program Impact Pathway An impact pathway (Figure 14) summarizes how the IOM evaluation committee proposes to examine the strength of evidence establishing plausible causal links between PEPFAR programs for children and adolescents and their intended impacts.
|
From page 109... ...
FIGURE 14 Program impact pathway for evaluation of PEPFAR's services for children and adolescents.
|
From page 110... ...
. This will be an important issue not only for this evaluation but also during the implementation of PEPFAR II, in particular within the context of implementation plans for partnership frameworks, which under the reauthorization legislation need to contain age- and sex- disaggregated data for "an identification of the intended beneficiaries […]
|
From page 111... ...
The committee will rely on qualitative data to ensure that interventions for children are contextually relevant and that PEPFAR program interventions are responsive to variances in high and low HIV prevalence areas. What criteria are used by PEPFAR countries to establish priorities and determine the balance among prevention, treatment, and care interventions for HIV-infected infants, children, and adolescents including those orphaned or made vulnerable by HIV/AIDS?
|
From page 112... ...
PEPFAR programs also link children, adolescents, and their guardians to appropriate legal services. The committee will seek to assess the effects of PEPFAR activities on current or proposed legislative reforms that incorporate child and adolescent protection measures and support protection programs mainly through primary collection of qualitative data during country visits and a review of publicly available documents.
|
From page 113... ...
and conduct interviews to assess whether these strategies and/or planning activities are designed to engage different sectors or agencies within the national government dedicated to child and adolescent well-being. The committee will seek any available data from OGAC and governments such as partnership framework implementation plans.
|
From page 115... ...
However, it is important to note that unlike other aspects of addressing HIV and AIDS, issues associated with gender-related risk and vulnerability cut across all of PEPFAR's implementation, and do not currently comprise their own unique programmatic or funding category. In addition, while ongoing efforts to highlight the differential needs of men and women have been underway since the beginning of the program, addressing issues associated with gender norms, violence, and stigma are still in the early stages in many PEPFAR countries.
|
From page 116... ...
The committee will also assess the extent to which PEPFAR has increased its gender programming to include activities aimed at reducing gender-based violence, addressing male norms, empowering women through behavior change and income generating activities, and improving equity in access to prevention, treatment, and care services. Program Impact Pathway The following impact pathway (Figure 15)
|
From page 117... ...
FIGURE 15 Program impact pathway for evaluation of PEPFAR's gender efforts. 117 NOTES: PHE = public health evaluations; PMTCT = prevention of mother-to-child transmission.
|
From page 118... ...
As with assessing male norms, the committee may use structured interviews with key informants from the OGAC, program, and project levels to determine the incorporation of efforts to reduce women's sexual risk into program planning and implementation. When available at the country level, the committee will pursue access to a limited number of recommended PEPFAR indicators, including those related to cross-generational sex.
|
From page 119... ...
How and to what extent have they succeeded in reducing incidence of violence? There are no PEPFAR indicators specific to measuring gender-based violence, so the committee will use country data and project level data where it is available.
|
From page 121... ...
. This section therefore outlines how the evaluation committee will explore the role of PEPFAR programs in facilitating improvements to health systems in partner countries and the readiness of PEPFAR and partner countries to increasingly share responsibility for managing the response to the epidemic and move to greater sustainability and country ownership.
|
From page 122... ...
. Although the reauthorization legislation did not define sustainability per se, PEPFAR did define how to promote sustainability in the Partnership Frameworks and Partnership Framework Implementation Plans guidance issued in September 2009 -- based on the principles of the Three Ones, the Paris Declaration, and the Monterey Accord of 2002 (activities that facilitate financing for development)
|
From page 123... ...
As such, the Partnership Frameworks are expected to identify indicators to assess progress toward meeting the goals, objectives, and programmatic and financial commitments -- with an eye toward international efforts to harmonize indicators. PEPFARspecific reporting systems are expected to be transitioned to nationally- and country-owned systems.
|
From page 124... ...
funding, including multilateral and bilateral programs involving joint operations. Further discussions among IOM staff, OGAC, and congressional staff 48 clarified "multilateral and bilateral programs involving joint operations" to mean programs operated in conjunction with bilateral funding through PEPFAR and the Global Fund (Bressler, 2009; Marsh, 2009)
|
From page 125... ...
, to permit the committee to assess performance of current or past PEPFAR activities. The committee will also examine Partnership Frameworks and Implementation Plans to assess what the countries and OGAC are responsible for, how it is being measured or tracked, and how the processes evolve for PEPFAR-related responsibilities and activities to be transitioned to country leadership for sustainable programs and positive impacts on individual and population health.
|
From page 126... ...
. FIGURE 16 Representation of WHO's six building blocks for effective health systems.
|
From page 127... ...
The pathway can also help assess the same block elements for technical assistance and other activities related to country readiness for assuming increased to total responsibility for their HIV/AIDS response. Although a useful evaluation tool, the linear nature of an impact pathway is a simplified representation of the reality of PEPFAR programs and their impact, and the committee found this to be particularly challenging in the area of systems-level activities.
|
From page 128... ...
All of the elements listed in the pathway can be categorized under one of the six building blocks, but due to space limitations, select elements were chosen to illustrate blocks within the pathway. ARV = antiretroviral drugs; ART = antiretroviral therapy; CD4 = cluster of differentiation 4; EQA/PT = external quality assurance/proficiency testing; M&E = monitoring and evaluation.
|
From page 129... ...
Service Delivery: What elements of more efficient, equitable, and effective service delivery should be expected to result in improved population health over time with better integration of HIV- and non-HIV health care and use of continuous quality improvement methods? Leadership and Governance: To what extent will partnership frameworks, jointly funded Global Fund activities, and other expressions of country ownership lead to improved and accountable governance as well as transfer of oversight, management, guidance, and financing for HIV-related services in health systems?
|
From page 130... ...
Do the Partnership Frameworks represent an "agreement" on the definition between the country and PEPFAR? To what extent have PEPFAR capacity-building, technical assistance, and financing activities to countries contributed to country readiness for transitioning the knowledge management, decision making responsibility, financing, and accountability/oversight of the PEPFAR-funded HIV/AIDS response to the national government?
|
From page 131... ...
PEPFAR activities will be evaluated relative to their contribution of inputs and support of processes, and the resulting outputs and outcomes. These should ultimately result in the desired outcome of stronger health systems that can more adequately respond not only to HIV but to other serious causes of morbidity and mortality, as well as any emerging challenges to the health and safety of their respective populations.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.