National Academies Press: OpenBook

Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary (2008)

Chapter: 2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting

« Previous: 1 Introduction to Impact Evaluation for PEPFAR
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 37
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 38
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 39
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 40
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 41
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 42
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 43
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 44
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 45
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 46
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 47
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 48
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 49
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 50
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 51
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 52
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 53
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 54
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 55
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 56
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 57
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 58
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 59
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 60
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 61
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 62
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 63
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 64
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 65
Suggested Citation:"2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 66

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting This chapter distills the results of a visioning exercise conducted by workshop participants to identify, frame, and prioritize the questions that matter for the evaluation of HIV/AIDS interventions by the President’s Emergency Plan for AIDS Relief (PEPFAR). Workshop participants as di- verse as congressional representatives, PEPFAR representatives, global part- ners, implementers, and country partners converged on some of the same impact evaluation questions. Workshop participants stressed the importance of identifying meaning- ful and appropriate impact indicators and moving beyond quantitative out- puts. Workshop speakers Jonathan Mwiindi of the Kijabe HIV/AIDS Relief Program, Kenya, and Mary Lyn Field-Nguer of John Snow, Inc., observed that the use of inappropriate indicators for evaluating program impact can sometimes tell an incomplete story, mislead, or mask other problems. For example, large numbers of individuals receiving first-line HIV treatment, Mwiindi noted, may mean that clinicians are not well trained to recog- nize when patients are in need of second-line drugs, not that the need for second-line drugs does not exist. Speakers Mwiindi and Kent Glenzer of the Cooperative for Assistance and Relief Everywhere, Inc. (CARE), described some of the drawbacks of and an overemphasis on quantitative measures. Mwiindi noted that an emphasis on counting sometimes drives inflated re- porting as programs compete for limited funds. Glenzer observed that the 37

38 EVALUATING THE IMPACT OF PEPFAR rush to produce quantitative outputs for goals like 2–7–10 can sometimes jeopardize sustainability. He called for a reduction in the number of indica- tors and a focus on a more strategic set of results. Such prioritization could be accomplished using a criteria-based approach, noted one participant. The generation of improved indicators, noted discussant Caroline Ryan of the Office of the U.S. Global AIDS Coordinator (OGAC), will ultimately provide better information for making programmatic course corrections. Many participants argued in favor of a broader interpretation of im- pact as opposed to a narrower, or more pure, notion of impact. Many participants also believed that priorities and opportunities for evaluation include not only those describing the results of program implementation, but also the process of implementation—for example, coordination and capacity building—because the means of implementation are inseparable from the ends. Speaker Julia Compton of the UK Department for Interna- tional Development (DFID) noted that defining the boundaries of what to measure—around what is AIDS and what is AIDS spending—is a challenge, and that defining the boundaries too widely or too narrowly involves risks. Defining boundaries includes answering the question of whether to measure direct AIDS impacts very narrowly or to measure broader impacts such as gender empowerment or land rights or health systems. Workshop participants identified questions for evaluating impact that can be clustered into the following nine broad categories: cost-effectiveness, conceptual approach, health impacts, impacts beyond health, capacity building and health systems strengthening, coordination and harmoniza- tion, sustainability, equity and fairness, and unintended impacts. Cost-Effectiveness Workshop participants described the importance of developing indica- tors that track the cost-effectiveness of PEPFAR. “Are we getting the biggest bang for our buck?” asked workshop speaker Christos Tsentas from the of- fice of Representative Barbara Lee. Discussant Mead Over of the Center for Global Development emphasized that cost-effectiveness of interventions— and whether costs can be afforded in particular countries once donor fund- ing is discontinued—has important implications for program sustainability. Cost-effectiveness measures can be used to evaluate different approaches, strategies, and interventions. Speaker Stefano Bertozzi of the National Institute of Public Health, Mexico, urged that cost-effectiveness measures be designed to assess types of prevention services delivered. He described a   PEPFAR’s 5-year goals—known as 2–7–10—are to support treatment for 2 million people, prevention of 7 million new infections, and care for 10 million people, including orphans and vulnerable children.

ENVISIONING A MEANINGFUL IMPACT EVALUATION 39 study of prevention service delivery in five countries in which a difference of three orders of magnitude in cost per service delivered was found: there were voluntary counseling and testing (VCT) programs that cost $7, $70, and $700 per client. Cost-effectiveness measures can help to ensure that VCT clinics not be established in contexts where there are not going to be any clients for them. Discussant Over recommended that cost-effectiveness measures also be used to test AIDS treatment interventions. Such analysis can determine the overall balance of adverse biological effects of treatment (that is, the spread of resistance or the increased opportunity of infection of longer living HIV-positive individuals), adverse behavioral effects of treatment (that is, increased risk behaviors), treatment effects on nontarget populations, and treatment effects on orphans. He added that cost-effectiveness evaluation could be used to test the assumption that PEPFAR should focus support on government-financed, free, high-quality therapy as compared to high- quality private-sector therapy. He noted that the support of public health care may be a problem in countries where government is weak and where private health care is abundant, but of poor quality. Workshop participants debated whether cost-effectiveness should be expressed in terms of results per donor investment versus results per ap- proach investment (also see Chapter 1, “Meaning and Uses of Impact Evaluation,” and Chapter 4, “Methodological Challenges and Opportuni- ties in Evaluating Impact”). Even if attribution cannot be shown, observed workshop moderator Ruth Levine of the Center for Global Development, it is useful to show what programs or approaches have the greatest impact. Conceptual Approach A number of proposed impact evaluation questions centered on the conceptual approach of PEPFAR and the assumptions that had been made in the program’s design: the countries and populations targeted, the budget allocations for different types of interventions, and the management and financing of the program. Such impact evaluation questions can be help- ful, noted workshop speaker Compton, in thinking through the conceptual model of how PEPFAR inputs lead to outputs and outcomes. Countries Targeted Workshop speakers Allen Moore of the Center for Strategic and Inter- national Studies and Compton urged PEPFAR decision makers to evaluate the impact and usefulness of working with the 15 “focus countries.” Is PEPFAR targeting focus countries appropriately? Compton added that im- pacts of regional factors should be evaluated; for example, focus-country

40 EVALUATING THE IMPACT OF PEPFAR programs should track the impact of migration from a nonfocus country that is a major source of infection. Populations Targeted Several speakers urged the development of impact indicators that track whether the appropriate populations are being targeted. Speaker Compton cited work by David Wilson of the World Bank showing that although 76 percent of adult male infections in Ghana are linked to commercial sex workers, only 1 percent of World Bank prevention funding supports pre- vention programs for sex workers (MOH and ORC Macro, 2006). She and speaker Theresa Diaz of the U.S. Centers for Disease Control and Preven- tion emphasized the importance of evaluating the targeting of high-risk and marginalized groups—such as commercial sex workers and injection drug users—who are the main drivers of epidemics in many countries. Speaker Bertozzi further observed that because patterns of risk behavior may vary by country, PEPFAR may want to evaluate how effectively it targets not only infected individuals and orphans, but also populations where new infections are expected to occur. Budget Allocations for Types of Interventions Considerable discussion focused on the value of assessing the impact of the budget allocations for prevention, treatment, and care on PEPFAR’s programming. Many participants cited the Institute of Medicine (IOM) study recommendations urging the removal of constraining congressional budget allocations (IOM, 2007). Observing that the current allocations came about as a result of highly effective advocacy by HIV treatment in- terests, speaker Field-Nguer called for an evidence-based assessment of the effectiveness of these allocations. Speaker Moore added that the usefulness of budget allocations and the appropriateness of funding priorities should be evaluated. Discussant Ambassador Jimmy Kolker, OGAC, suggested that evaluative information would be useful in informing future targets for scal- ing up prevention, care, and treatment interventions. Citing the imbalance in enrolling 90,000 people in treatment programs while 135,000 people were newly infected each year over the first phase of PEPFAR, speaker Compton urged the development of impact indicators to assess the appro- priate balance among prevention, treatment, and care interventions. She also suggested an evaluation of the balance among interventions within prevention, noting that a significant proportion of the prevention funding is required to support abstinence and faithfulness (AB) programs. Speaker David Gootnick of the U.S. Government Accountability Of- fice suggested that the assessment of prevention interventions provides an

ENVISIONING A MEANINGFUL IMPACT EVALUATION 41 opportunity for a deeper level of analysis that could inform the dialogue about earmarks. Citing the Futures Group GOALS model (Stover and Bollinger, 2006), he suggested that specific interventions (that is, partner reduction, male circumcision, opt-out testing, abstinence programs, and sex worker peer education programs) can be evaluated for their effectiveness in averting infections. Another workshop participant suggested that impact evaluation be used to assess the degree to which earmarks and pipeline programming may have translated into a lack of integration of programs on the ground. Several workshop participants stressed the importance of distinguish- ing between the evaluation of implementation of a particular policy on the ground and the evaluation of a policy as it is written. In many countries, noted speaker Jessica Price of Family Health International, there is still ambiguity in interpreting PEPFAR policies and how money can be used. For example, those receiving AB money interpreted PEPFAR policies to mean they could not educate youth about sexually transmitted infections. Simi- larly, many countries interpret differently whether or not PEPFAR funds can be used for family planning, and this is reflected in differential implementa- tion on the ground. Speaker Tom Kenyon of OGAC added that often there has been misinterpretation or overinterpretation of what could not be done, particularly with regard to implementation of condom programming. He noted that PEPFAR has had to modify guidance because there was a retreat from programming using condoms and other prevention approaches. Workshop participants stated that because earmarks have a political basis, as opposed to an empirical or a scientific basis, there may be less value in evaluating earmarks or in combining their evaluation with a more rigorous scientific evaluation. Speaker Jim Sherry of George Washington University noted that room is needed for political decision making about what is most important in terms of the evaluation agenda. Workshop par- ticipant Naomi Seiler from the U.S. House of Representatives Oversight Committee warned that it may be risky to mix scientific and objective evaluation with evaluation of more difficult political questions, such as evaluating a policy that prevents a certain type of intervention from taking place. As speaker Compton noted, “An evaluation cannot answer what goals are most important; that is a political decision.” Management and Financing Workshop participants also encouraged the design of impact evaluation measures for the management and financing of the PEPFAR program across a variety of levels. One participant urged the evaluation of the impact of PEPFAR’s practice of awarding hundreds of small grants to community or- ganizations. Speaker Sara Pacqué-Margolis of the Elizabeth Glaser Pediatric

42 EVALUATING THE IMPACT OF PEPFAR AIDS Foundation encouraged the development of impact indicators for performance-based financing systems. Speaker Agnes Binagwaho of the Rwanda National AIDS Control Commission called for a development of metrics to assess the effectiveness and results of program comanagement, such as that piloted between PEPFAR and the government of Rwanda. Speaker Bertozzi remarked that evaluation could be used to test the effec- tiveness of different technical assistance approaches in different contexts. He noted that PEPFAR, the World Bank, and The Global Fund have very different approaches in terms of levels of technical assistance, with PEPFAR providing intensive technical assistance and The Global Fund providing countries with funds to purchase their own technical assistance. Evaluation may reveal that some approaches may work more efficiently for certain contexts; for example, where there is high local capacity, The Global Fund approach might be more efficient, and in a country with lower capacity, the PEPFAR approach might be more efficient. Health Impacts Workshop participants suggested the development of impact evalua- tion measures for tracking change in health, including indicators specific to HIV/AIDS as well as more general indicators. HIV/AIDS-Specific Health Impacts Prevalence, Incidence, Morbidity, Mortality, Longevity Many workshop participants suggested that future impact evaluation of HIV-specific factors move beyond measurements of those accessing care and treatment to measurement of longer term health status. Because antiretrovi- ral (ARV) treatment may achieve viral suppression in a small percentage of patients receiving drugs, observed speaker Bertozzi, measuring the number of people on antiretroviral therapy (ART) may not be enough to determine the lifesaving effectiveness of a program. Speaker Jaime Sepúlveda of the University of California–San Francisco urged the use of state-of-the-art methods to measure prevalence and incidence of HIV infection. Speaker Field-Nguer spoke of the importance of measuring HIV/AIDS mortality and morbidity, and speaker Savannah Lengsfelder from the U.S. Senate Committee on Foreign Relations called for the development of measures of PEPFAR impact on longevity.

ENVISIONING A MEANINGFUL IMPACT EVALUATION 43 Prevention of HIV Transmission The full range of impacts of treatment on HIV transmission—both biological and behavioral—should be evaluated, urged discussant Over (Table 2-1). In terms of biological effects, ARV treatment can reduce viral loads, but it can have a negative impact on prevention because of resistance that comes about through imperfect adherence to drug regimens and a greater period of infectivity of longer living, HIV-positive individuals. He and speaker Mwiindi urged the development of indicators to measure the completeness of viral suppression, the level of drug adherence, and the level of resistance created by PEPFAR-supported treatment programs. In terms of behavioral effects, treatment programs can on one hand encourage health- seeking behaviors, including prevention behaviors, but those receiving ART as well as HIV-positive and HIV-negative individuals in the community may on the other hand engage in more risky or other adverse behaviors TABLE 2-1  Possible Effects of ART on HIV Transmission Direction of Effect Beneficial Adverse Type of Effect (Slow transmission) (Speedy transmission) Biological Reduce infectiousness. ART may Select for resistance. lower viral loads and may therefore Imperfect adherence to ART lower the risk of transmission per selects for resistant strains of sexual contact. the virus, which can then be transmitted. Longer duration of infectivity. The greater longevity of HIV-infected people taking ART has the unintended negative consequence of increasing the period in which the patient can transmit the virus. Behavioral Encourage prevention, especially Offsetting behavior. People diagnostic testing. ART may increase receiving ART, and HIV- the uptake rates of prevention positive and -negative activities, particularly voluntary people in the surrounding counseling and testing. community, may engage in more risky behaviors than they would if ART were unavailable. SOURCE: Over et al., 2007.

44 EVALUATING THE IMPACT OF PEPFAR to preserve eligibility for programs than they would have if ART were not available. The extent to which treatment has contributed, through such behavioral channels, to widening the gap between the number of people on treatment and the number of people newly infected can be measured through evaluation. Speakers Mwiindi and Moore, along with moderator Levine, emphasized the importance of evaluating PEPFAR’s success in pre- vention of mother-to-child transmission of HIV (PMTCT) and in preventing HIV infection in children, adolescent girls, and women. Quality of Life Many speakers identified an important need to broaden HIV-specific in- dicators to include those measuring quality of life, which speaker Sepúlveda described as an evaluation of overall physical, mental, and social well- being of people affected by HIV/AIDS. According to speakers Field-Nguer, Price, and William Holzemer of the University of California–San Francisco, other attributes of quality of life that could be measured through impact evaluation include restored productivity; effective management of chronic, secondary symptoms of ART patients; palliative care and use of analgesics to support dying with dignity; and family grieving. Behavioral Change Another major area of HIV/AIDS-specific health indicators that could be tracked is human behavioral change, which, according to workshop speaker Sepúlveda, includes the rates of both risky and protective behav- iors. Speaker Tsentas enumerated several measures that could be developed to track changes in human behavior, including retention of patients on treatment, protected sexual intercourse, increased circumcision of males, and participation in needle-exchange programs. Discussant Over empha- sized the importance of evaluating the impacts of treatment on prevention behaviors, including VCT. He argued that treatment may itself encourage or “disinhibit” risky behavior because it creates a perception that HIV/AIDS is now curable. He noted that such an effect has already been observed, for example, as decreases in condom use among sex workers in Kenya follow- ing announcements of “false cures” (Jha et al., 2001). This phenomenon is further discussed later in this chapter under “Unintended Impacts.” Speaker Martha Ainsworth of the World Bank suggested that impact evaluation could be used to assess the effect of knowledge on behavioral change. She cited an example from Thailand in which the mere dissemination of knowl- edge that 44 percent of sex workers in Chiang Mai were infected with HIV had a dramatic effect on sexual behavior. Speaker Compton suggested that evaluation could be used to assess the impacts of a lack of investment in

ENVISIONING A MEANINGFUL IMPACT EVALUATION 45 specific behavioral change interventions, such as clean-needle distribution programs for intravenous drug users, which currently cannot be supported using PEPFAR funds. Another workshop participant wondered if evaluation could assess the “dose response” of interventions required to maintain a behavioral change over time. Speaker Shannon Hader of OGAC asserted that measuring behavioral outcomes is much more difficult than measuring treatment out- comes. Tools and methods to measure and understand behavioral change outcomes are described further in Chapter 4, “Methodological Challenges and Opportunities in Evaluating Impact.” Stigma and Discrimination A final area of HIV-specific assessment, noted speakers Sepúlveda and Field-Nguer, is stigma and discrimination. To what degree have PEPFAR’s interventions contributed to promoting the acceptance of people living with HIV/AIDS in the community and to reducing stigma? Monitoring and evaluation (M&E) should consider the impact of stigma on a variety of factors among both people living with HIV/AIDS and health care workers, suggested speaker Holzemer and discussant Timothy Fowler of the U.S. Bureau of the Census. These include participation in HIV testing, use of services (that is, antenatal care), poor treatment by health care providers, adherence to medications, health status, and quality of life, such as loss of social support, isolation, violence, and limiting social interactions because of fear. Other Health Impacts Future impact evaluation of PEPFAR could also consider other health indicators in addition to those related to HIV/AIDS. Evidence from the Rwanda context, noted speaker Allen Moore, suggests that many health indicators beyond those specific to HIV/AIDS have improved as a direct outcome of PEPFAR investments. (This case study is described in greater de- tail in Chapter 4, “Measuring Impacts of Health Systems Strengthening.”) Speaker Savannah Lengsfelder also spoke of the importance of evaluating any negative impacts of PEPFAR investments on other health conditions. Speakers Sepúlveda, Binagwaho, and Field-Nguer emphasized the impor- tance of examining the effects of PEPFAR programs on mortality by all causes, overall survival, and lives saved from other diseases. Understand- ing the infant and child mortality contribution to overall mortality is an important piece of this. Speaker Pacqué-Margolis pointed out that attri- butes of reproductive health should also be monitored, including fertility, unintended pregnancy, and intended pregnancy. As for HIV/AIDS patients,

46 EVALUATING THE IMPACT OF PEPFAR quality-of-life indicators should be tracked, noted speaker Tsentas, as well as the effectiveness of interventions in protecting, educating, and equipping people with tools they need to take care of themselves. Speaker Lengsfelder suggested that data on other health indicators be disaggregated to show if differential effects are occurring in specific popula- tions and in focus countries versus nonfocus countries. Impacts Beyond Health What have been the effects of PEPFAR interventions in the areas of gender equality, child welfare, security and development, and institutional change? Workshop participants discussed a number of measures of HIV/ AIDS interventions beyond health that could be tracked through impact evaluation. Gender-Focused Activities Numerous speakers called for an evaluation of gender-focused ac- tivities, particularly those aimed at empowering women and girls. Gender- related dynamics occur in both men and women and influence their risk of contracting HIV, noted speaker Julie Pulerwitz of the Population Council. For men, gender norms encourage multiple sexual partners and early sexual debut. For women and girls, power imbalances result in an increased vul- nerability to HIV/AIDS. As speakers Glenzer and Tsentas noted, power imbalances may be reflected in factors such as early and child marriage, weaker ability to negotiate sexual relations, susceptibility to pressure to engage in transactional and intergenerational sex, vulnerability to sexual violence, poor education, lack of economic opportunities and exclusion from control of strategic economic assets, exclusion from decision-making processes and from patronage networks, lack of property and inheritance rights, and lack of legal and enforceable rights. Indicators for tracking such factors could be developed, suggested Glenzer, as well as indicators to measure the effectiveness of PEPFAR in addressing the underlying causes of women’s and girls’ vulnerability to HIV/AIDS. Glenzer also noted that building the analytical skills and com- petencies of men and women themselves around what is happening in their societies is a way to accelerate change in power structures and accelerate improvements, and the extent to which interventions build such capacities is something that can be measured through evaluation. Speaker Rachel Glennerster of the Abdul Latif Jameel Poverty Action Laboratory suggested that impact evaluation be used to identify the most effective messages in persuading teenage girls to change their behavior, and workshop modera- tor Levine suggested that programs be evaluated to assess their effective-

ENVISIONING A MEANINGFUL IMPACT EVALUATION 47 ness in reducing the spread of HIV infection to women and girls. Speaker Mwiindi introduced a nuance that the targeting and packaging of gender interventions should also be assessed through impact evaluation. Interven- tions aimed at sensitizing men may be perceived more positively and may be more effective in some cultures, he noted, than interventions targeting women. Child Welfare Several speakers highlighted the importance of evaluating parameters related to child welfare. Evaluation could measure the effectiveness of PEPFAR in raising the ability of adults to parent, suggested speaker Paul De Lay of the Joint United Nations Programme on HIV/AIDS (UNAIDS). Another participant observed that evaluation measures could track whether PEPFAR interventions had resulted in healthier, better nourished, and bet- ter educated orphans and vulnerable children who end up leading more productive lives as adults. Security, Development, and Poverty Alleviation Parameters of a country’s security and development could be tracked using impact evaluation. Speakers Binagwaho and De Lay suggested that impact evaluation be designed to track the effects of PEPFAR interventions on peace and security in a nation, poverty alleviation and economic growth, and general national development. Institutional and Societal Changes Speakers De Lay and Field-Nguer suggested that impact evaluation of PEPFAR also focus on tracking how societies and institutions are changing and improving. Policy changes such as property rights, inheritance laws, and human rights could be monitored, as well as changes in the contextual environment more indicative of supportive systems, such as political will, community ownership, food and water security, engagement of vulnerable populations, destigmatization, and antidiscrimination measures. Speaker Compton raised the question of what technical assistance is doing to help change political views and priorities in a country. Impacts on Sustainability, Capacity Building, and Health Systems Strengthening Workshop participants emphasized the importance of monitoring both the process and results of building local capacity, particularly in the area

48 EVALUATING THE IMPACT OF PEPFAR of strengthening health systems. Discussant Ambassador Kolker asserted that building local capacity is a key to scaling up programs and that an ideal for PEPFAR is to have countries with plans and capacity, for which all that is needed for national scale-up is funding. Building local capacity is an important prerequisite for handing off important aspects of the PEPFAR program, he noted. Speakers Lengsfelder and Tsentas raised the questions of whether PEPFAR’s interventions were motivating and empowering part- ner countries to develop their own aggressive treatment and prevention strategies and contribute to the development of capable and self-sustaining national health systems and whether the process of building capacity was responsive to local needs. Beyond measuring the effectiveness of the capacity-building process, evaluation can help to assess the results of capacity building. Evaluation can help to determine the impact of improved personnel staffing, training, and additional equipment on reinforcing the health system in general, noted speaker Binagwaho. Evaluation can also help define the extent to which overall improvements in the health system are attributable to PEPFAR’s capacity-building inputs as opposed to other external changes in the envi- ronment and the populations served, suggested speaker Pacqué-Margolis. She added that evaluation can also inform us about any worsening of out- comes in health systems as a result of interventions. Needs for tracking changes in the capacity of the health care system were identified in the areas of health care workforce, infrastructure, insti- tutions, quality of care, the knowledge base, and the national-level health agenda. Health Care Workforce Measuring the Effects of PEPFAR on Workforce Shifts The HIV/AIDS epidemic and interventions have had a dramatic effect on the health care workforce in many countries, noted speaker Sepúlveda. Health care workers themselves are among the populations that have be- come infected with and succumbed to the virus. In addition, “brain drain” has attracted health care workers both externally to higher salaries in de- veloped countries and internally to generously funded vertical health pro- grams in developing countries. Evaluation of PEPFAR should be designed to include an examination of the impacts of such shifts on the health care system, urged Sepúlveda.

ENVISIONING A MEANINGFUL IMPACT EVALUATION 49 Impact of PEPFAR’s Training Approaches Speaker Holzemer recommended that evaluative strategies be used to measure the impacts of PEPFAR’s investments in different types of health care workforce training approaches. Distinguishing between two types of PEPFAR training strategies—in-service education, or developing skill sets across health provider groups in the existing workforce (that is, physi- cians, advanced-practice nurses, nurses, nurse assistants, community-based workers), and preservice education, or incorporating more people into the workforce—Holzemer suggested that evaluation be used to assess the quali- fications, skills, competencies, and quality of health care workers trained using each approach. Impact evaluation could be used to test the assump- tions behind the in-service training approach, in which skills are assigned to the lowest level worker possible. He noted that an in-service, or skill trans- fer, training approach assumes that it takes too long to train more nurses and other health care workers, and this urgency may not be justified. He added that the in-service training strategy also devalues clinical judgment and may result in a generation of poorly prepared health care workers who are insufficiently supervised and trained. Citing a lack of culture of continu- ous medical education in many African countries, speaker Mwiindi stressed the importance of evaluating the degree to which in-service training links new research and new findings to training programs. Holzemer suggested that evaluation also be used to assess the effective- ness of PEPFAR’s numerous “twinning”-based training programs, which involve the participation of partnering individuals and institutions from the United States. These include programs such as volunteer-based programs, institution-based partnerships, peer-to-peer collaborative relationships, professional exchanges and mentoring, and nonprescriptive demand- and process-driven partnerships. Holzemer pointed out that the model for such volunteer programs assumes that participants can afford financially to miss work for 3 weeks to serve and to train; however, many potential volunteer nurses are single parents, have families, and cannot afford to take time off from work. The assumptions and logic of such programs should be evalu- ated, Holzemer noted. Speaker Field-Nguer suggested that it would be worthwhile to also assess the effectiveness of other workforce strengthening strategies, such as postgraduation fellowships with organizations that can mentor and give technical assistance, as well as recruitment of retired nurses back into the workforce. The effectiveness of training HIV-positive patients to serve as community health care workers could be evaluated, suggested speaker Mwiindi. Possible measurements of workforce training programs should go be-

50 EVALUATING THE IMPACT OF PEPFAR yond the number of persons trained, noted Holzemer, and could include the following: • Length of time to train • Degree to which clinical facilities are strengthened • Impact of the least prepared worker on clients and patients • Degree to which the capacity of qualified workers is developed • The effectiveness of training and supervising the lowest level workers • The degree to which the strategy provides opportunities for ad- vanced professional development • The degree to which the training program addresses gender in- equality in the work environment • Unintended consequences of the training, such as in-service training diverting workers from their jobs because higher pay is offered • Knowledge, competencies, attitudes, and skills; types of positions taken; and quality of work environment Impact of PEPFAR’s Programs to Support Health Care Workers A number of PEPFAR programs exist to provide support to professional and community-based caregivers and their families, including programs that provide HIV testing, treatment, and care for infected health care work- ers. Speaker Holzemer suggested that the effectiveness of such programs be evaluated. Metrics for evaluation of the effectiveness of these interventions might include, for example, whether services for health professionals are provided at different times of day from regular treatment programs, given professionals’ desire not to wait in line and mix with clients. Sustaining Local Workforce Development Systems Speaker Holzemer noted the importance of evaluating how workforce development strategies complement and contribute to the sustainability of those systems in the country that controls the workforce. For example, there is strong potential for PEPFAR interventions to build the capacity of national regulatory bodies, professional associations (nursing and physi- cians’ associations), educational accrediting associations, and government ministries. Through collaboration with such institutions, Holzemer noted, the careers of health care workers over time, from registration through retirement, could be tracked.

ENVISIONING A MEANINGFUL IMPACT EVALUATION 51 Institutions Impact evaluation should be designed to assess the effectiveness of PEPFAR’s institution-building efforts, said speaker Sepúlveda. He added that tracking change in the ability of community-based organizations to re- spond to the epidemic is an important consideration for impact evaluation design. Speaker Mwiindi added that evaluation should also consider the degree to which nontraditional institutions have been engaged or integrated in capacity-building efforts. He noted that there is underused potential for strengthening the capacity of the health system by involving institutions such as the church sector, which plays a substantial role in health provision in Africa, but is not traditionally recognized as a health institution. Mwiindi cited an example of a PEPFAR pilot site in Gezabi, Kenya, which did not have enough patients to receive drug treatment until religious leaders were engaged and trained to help identify and recruit patients. Infrastructure Speakers Sepúlveda and Mwiindi spoke of the need to evaluate the improvement of public health infrastructure, giving as an example the development of local supply chain management and drug delivery systems. Indicators could be developed, suggested Mwiindi, to measure the extent to which drug forecasting procedures and structures are in place to match resources to the need. Quality of Care and Service Delivery Quality of care and service delivery was highlighted as another aspect of health systems strengthening that could be evaluated. Speakers Mwiindi and Field-Nguer suggested a variety of indicators that could be used to track the effectiveness of PEPFAR interventions in the area of quality improvement. These include the quality and appropriateness of service delivery, and existing gaps, in the areas of prevention, care, treatment, support, and mitigation; ARV retention rates; levels of client satisfaction; appropriateness of referrals; and improved community attitudes toward people living with HIV/AIDS. Both speakers emphasized the importance of rational prescription behavior as a metric of quality of care. Field-Nguer commented that even within the same clinic, it is a challenge for health care workers to prescribe the right common antibiotic for the same syndrome for many sexually transmitted diseases. The more complex ARVs represent a further challenge, she said. Speakers Hader and Compton addressed a need for the development of broader metrics of quality. Metrics should be devel- oped for assessing the quality of methodologies and of data, noted Hader.

52 EVALUATING THE IMPACT OF PEPFAR In addition, noted Compton, information about the process of achieving quality should be collected through evaluation, including factors such as timeliness, access, follow-up, leadership, and management. Knowledge Base An expanded knowledge base is a critical determinant of PEPFAR’s success, noted speaker Sepúlveda. Impact evaluation measures should be designed to track how PEPFAR has contributed to the development of an evidence base that includes information on what works best and how pro- grams should be implemented, scaled up, and sustained. Moderator Levine asserted, however, that tensions may exist between building knowledge and responding to a serious public health problem. The appropriateness of the balance between implementation and strengthening the knowledge domain is an element that can be assessed through impact evaluation. National-Level Health Agenda Speakers Nils Daulaire of the Global Health Council and Sepúlveda broadened the discussion of evaluating PEPFAR’s impacts on health systems strengthening to include assessment of how well PEPFAR has integrated with other health issues and has contributed to the change in national-level health agendas and priorities over time. Sepúlveda posited that a possible outcome of explicit PEPFAR interventions is to drive desired improvements into the health system, including incremental incorporation of other health priorities into country agendas, through what he termed a “diagonal” approach, a juxtaposition of both horizontal and vertical approaches. Daulaire added that PEPFAR support for broad maternal and child health systems should be monitored because such systems are key to the founda- tion for effective health systems overall. Coordination and Harmonization How integrated are PEPFAR systems with existing systems and pro- grams, and how well is PEPFAR aligned with country priorities and plans? These are among the questions raised by workshop participants, who em- phasized the importance of measuring the coordination and harmonization of program implementation. Participants also emphasized that the design and implementation of the impact evaluation itself should be coordinated and harmonized; this is described further in Chapter 3. Valuable synergy can be achieved through coordination and harmoni- zation. Moderator Levine observed that coordination and harmonization can show which programs or approaches have impact on specific outcomes.

ENVISIONING A MEANINGFUL IMPACT EVALUATION 53 Coordination and harmonization also bring together the different strengths of diverse actors involved in a program. Discussant Ambassador Kolker hoped that U.S. involvement and leadership in the HIV/AIDS response would motivate other donors to fill gaps in areas in which they have a comparative advantage relative to the United States. He observed that The Global Fund to Fight AIDS, Malaria, and Tuberculosis (The Global Fund), UNAIDS, and the World Bank have different advantages, histories, strengths, and implementation approaches that can be brought to the table in the response to HIV/AIDS. Although the United States is delighted to be in the lead, he noted, it is also delighted not to be the only player. Other workshop participants echoed this sentiment. For example, in comparing the U.S. response to that of DFID, Julia Compton observed that DFID’s use of a more process- and sustainability-orientated approach, as opposed to an emergency-orientated approach, might create difficulties in achieving results quickly. Speaker Jody Kusek of the World Bank noted that her institution has a comparative advantage in helping countries build monitoring systems. Discussant Kolker and speaker De Lay cautioned that although harmoniza- tion, alignment, and process are important means to an end, they are not the exclusive determinants of a successful program. A useful framework for considering coordination and harmonization across multiple levels, suggested by speaker Tsentas, is outlined in the fol- lowing three sections on coordination among implementing agencies of the U.S. government, harmonization and alignment with partner countries, and coordination among program-implementing organizations such as donors, humanitarian assistance programs, and country-level actors. Coordination Among Implementing Agencies of the U.S. Government The HIV/AIDS epidemic has stimulated an unprecedented interagency response by the U.S. government (see Box 1-1), but this cooperative re- sponse has required a harmonization of approaches within the various implementing agencies, observed speaker Kenyon. For example, stated speaker Kathy Marconi of OGAC, within those U.S. agencies, there was initially a lack of consistent targets across focus countries because each government agency had its own reporting system; such systems have had to be coordinated and harmonized through the PEPFAR program. Among the main needs identified by workshop participants for evalu- ation of coordination and harmonization among U.S. government agencies are the impacts of complementary interventions that are essential for man- aging HIV/AIDS specifically and health generally in developing countries but that go beyond narrowly focused HIV/AIDS services. These so-called wraparound programs include investments in areas such as malaria, tuber- culosis, nutrition, food security, social security, education, child survival,

54 EVALUATING THE IMPACT OF PEPFAR family planning, reproductive health, medical training, health systems, and potable water. Speakers Compton and Pacqué-Margolis suggested that evaluation could be helpful in assessing the effectiveness and constraints of the wraparound model, and speaker Lengsfelder suggested that the ef- fectiveness of integration between HIV/AIDS services and complementary services also be measured. Finally, speakers Ainsworth of the World Bank and Pacqué-Margolis suggested that impact evaluation could be used to measure negative impacts of such programs, such as those created by dis- torted incentives. If provision of a service is contingent on a patient being infected with HIV, noted Ainsworth, there is a risk that someone who needs the service equally or even more may not receive it. If nutritional support programs are discontinued at a site, inquired Pacqué-Margolis, will patients stop seeking HIV services? A similarly dramatic example was described by speaker Binagwaho, in which a Rwandan child interviewed on television said he wished his mother were HIV positive so he could go to school. Potential negative impacts of complementary interventions are further de- scribed later in this chapter under “Unintended Impacts.” Harmonization and Alignment with Partner Countries A second level of harmonization identified for future evaluation by workshop participants is harmonization with partner countries. Workshop participants value harmonization and alignment of PEPFAR with country priorities and plans because they believe this leads to greater success of the overall program. For example, ownership and local engagement, said speaker Daulaire, are critical to both financial and operational sustainabil- ity. However, several participants spoke of challenges and constraints to harmonization with partner countries. Compton pointed out that in some cases, depending on national systems for data collection can be problem- atic if those systems are weak. Discussant Ambassador Kolker added that where national leadership and national ownership exist, donor alignment can work, implying that the absence of national leadership and owner- ship makes alignment with partner countries more challenging. Speaker Binagwaho summarized that it is unclear what level of alignment is optimal; she and speaker De Lay observed that impact evaluation could help assess the effectiveness of varying degrees of alignment of PEPFAR with national development plans. Speaker Mwiindi suggested that the extent to which interventions are culturally contextualized is an important component of evaluating harmo- nization with partner countries. For example, he noted that interventions for orphans and vulnerable children should take into account that child care is a community-based effort in many cultures. Similarly, how messages about male circumcision are communicated should consider that the prac-

ENVISIONING A MEANINGFUL IMPACT EVALUATION 55 BOX 2-1 Comanagement of PEPFAR in Rwanda: A Case Study In Rwanda, agreements have been created to align PEPFAR, The Global Fund, and the World Bank to Rwanda’s national HIV/AIDS plan. For example, all donors must align to the national format for quantification, designed for the district level. Realignment of PEPFAR’s 2–7–10 plan to the Rwandan indicators took about one and a half years. The majority of PEPFAR investments are also coman- aged with the Rwandan government. Committees that make decisions about the use of funds are chaired by a partner country national. At the technical level, U.S. government institutions, Rwandan government institutions, other donors, and civil society work together. For example, field visits to assess progress are conducted jointly by representatives of PEPFAR, The Global Fund, and the government of Rwanda. Local institutions also set policies and standards for implementation. Plans and projects are open and accessible at the community level, and the gov- ernment of Rwanda shares progress among donors in a transparent fashion. The comanagement approach is a key to ownership and success because the host country is part of the decision and leads the decision-making process. SOURCE: Binagwaho, 2007. tice is not culturally accepted in certain areas. Citing the experience of co- management of PEPFAR between the U.S. government and the government of Rwanda (see Box 2-1), speaker Binagwaho described other measures that could be tracked to evaluate whether harmonization mechanisms have provided effective and demonstrable results. These include the openness and accessibility of donor plans and projects at the community level, the pres- ence of instruments and structures for joint decision making, and the degree of information sharing and joint implementation among partners. The alignment of PEPFAR with national priorities in Uganda was another example discussed. Citing a perceived criticism in the IOM com- mittee’s report that alignment had been difficult in the early years of the Uganda partnership because of the many funding earmarks and restrictions, discussant Ambassador Kolker acknowledged that although U.S. efforts were not a perfect match to local priorities, they were congruent with the national plan. The United States does not claim to be the only actor in the field, and PEPFAR does not claim responsibility for everything, he stated; in the Uganda case, the United States was able to match its unique capabilities and expertise to local needs and opportunities. Kolker added that about

56 EVALUATING THE IMPACT OF PEPFAR half of the focus countries have taken advantage of abstinence waivers, a mechanism that provides some flexibility in the coordination process. The existence and effectiveness of such mechanisms might be included in future evaluations of the impact of PEPFAR’s harmonization efforts with partner countries. Coordination Among Program Implementers A third area of coordination identified for prospective evaluation is the coordination among program implementers. Workshop discussant Ryan suggested that the development and effectiveness of a variety of coordi- nation tools, or mechanisms for fostering the exchange of learning, be evaluated. Speaker Field-Nguer gave an example of such a coordination tool: an implementers’ group that has been formed, through John Snow International and World Learning with Global Health Council, to foster learning about what is happening and what could be improved. This group has been using the experience of implementers in the field to influence the reauthorization process. Speaker Marconi described the development of two coordination tools whose effectiveness could be evaluated: (1) the PEPFAR extranet, a mechanism for information sharing accessible to all U.S. government employees, and (2) systematic literature reviews on the latest intervention research. The systematic reviews are conducted by the Cochrane Group and circulated to PEPFAR partners all over the world. Speakers Mwiindi and Compton suggested that harmonization of the drug procurement system can provide a useful case study for assessing co- ordination among program implementers. Mwiindi noted that PEPFAR’s entry into the drug supply chain—and the program’s exclusive sourcing of U.S. Food and Drug Administration (FDA)–approved drugs or branded drugs—has resulted in parallel streams of drug qualification, delays in drug delivery, and inconsistency with national protocols. Compton called for an impact evaluation of the policy requiring FDA approval of drugs distrib- uted through PEPFAR. Mwiindi added that the effectiveness of PEPFAR’s harmonization with existing drug procurement systems also be evaluated, which might include tracking measures such as drug quality, drug cost, reliability of the drug supply, and level of engagement with organizations on the ground, such as religious groups, that function in the existing health care delivery and drug supply system.   PEPFAR partner countries can apply for a waiver that allows them to reapportion their prevention funds among abstinence–faithfulness–condom use (ABC) interventions to reflect the nature of the local epidemic.

ENVISIONING A MEANINGFUL IMPACT EVALUATION 57 Sustainability Impacts Workshop participants appealed for the development of evaluation measures that would assess PEPFAR’s effectiveness in evolving into a long- term, sustained response. Moderator Levine, along with speakers Moore and Gootnick, emphasized that making the transition to sustainability will be a challenge for what Moore termed a “hurry-up effort” that was legis- lated with a sense of urgency and began as an emergency response. Other participants stressed the importance of program sustainability given the long-term nature of the HIV/AIDS disease. Discussant Phillip Nieburg at the Center for Strategic and International Studies noted the importance of acknowledging that HIV/AIDS is neither an epidemic nor a pandemic, but endemic, which implies a sustainable response. Speaker De Lay also under- scored the need for PEPFAR to ensure effective chronic care for infected persons into the future. Measuring the financial sustainability of PEPFAR was a major focus of discussions, and speaker Compton warned that the financial implications for PEPFAR not continuing into the future would be tremendous, given the substantial financial burden that would be passed on to countries. Speaker Mwiindi suggested that evaluation could help prompt the development of and measure the progress toward a clear exit strategy. Measures could be developed, for example, of the degree to which the PEPFAR program has leveraged additional resources by other donors and by national govern- ments, remarked workshop discussant Sherry. Workshop participants also emphasized the importance of evaluation for long-term learning about the program. Moderator Levine suggested that evaluation can help to generate a technical consensus and a stock of knowledge about what works. Speaker Ainsworth added that information on what combination of services is most effective—a potential product of evaluation—can help to improve the efficiency and sustainability of the program. Evaluation of PEPFAR’s impacts on sustainability needs to include mea- sures of capacity building and promotion of local independence, workshop participants said. Speaker Compton noted that evaluation should assess the degree to which PEPFAR is making a sustained contribution to institutions and systems that include research, M&E, policy, budgeting, planning, and programming systems, as well as the degree to which PEPFAR is sustaining public-sector and voluntary staffing. Speaker Pulerwitz advised that evalu- ation of sustainability also take into account the degree to which “least dependency” practices are in place that are conducive to local implementa- tion, ownership, and coordination with national systems and structures.

58 EVALUATING THE IMPACT OF PEPFAR Equity and Fairness Impacts Speaker Norman Daniels of the Harvard School of Public Health fo- cused his remarks on the importance of evaluating the fairness of the PEPFAR program. He introduced the concept of fairness and defined its subcomponents—equity, accountability, and efficiency. He provided ex- amples showing the trade-offs and tensions between the competing goals of equity and efficiency in decision making. Finally, he offered guidance on indicators that can be developed to examine issues of equity as PEPFAR programs are scaled up. Defining Fairness Aspects of fairness are central in decisions to scale up ART treatment and prevention, noted Daniels. In the context of domestic and international health policy, the goal is to improve population health, as measured in the aggregate, but also to distribute the benefits of health policy in a fair way and reduce disparities. These dual goals may conflict with each other, how- ever, due to conflicting priorities. A framework integrating concerns about equity, accountability, and efficiency can be used for evaluating the fairness of health interventions in developing countries (Box 2-2). Benchmarks can be developed for each component and indicators generated to measure the improvement that a particular intervention produces for each of these components. Trade-offs Among Competing Goals in Evaluating Fairness The evaluation of fairness involves not simply the maximization of a specific outcome, but the way in which resources are being used to achieve several goals, which may at times conflict with each other. Trade-offs can exist, when attempting to achieve the goals of equity and efficiency, be- tween getting the best outcomes with scarce resources and giving people fair chances at some benefits. Several examples of trade-offs in evaluating fairness were shared. World Health Organization 3 by 5 Initiative Issues with equity implications have arisen in the context of scale-up decisions of the World Health Organization’s (WHO’s) 3 by 5 Initiative, such as cost recovery, eligibility criteria for patient selection, site selection,   The WHO 3 by 5 Initiative was a global target to provide ART to 3 million people living with HIV/AIDS in low- and middle-income countries by the end of 2005.

ENVISIONING A MEANINGFUL IMPACT EVALUATION 59 BOX 2-2 The Elements of Fairness • Equity: A key aspect of equity is the absence of unjustifiable inequalities or disparities across demographic groups. Such inequalities might include gen- der inequality, urban–rural inequality in access, child versus adult inequality, inequality of vulnerable and stigmatized groups, and inequality of immigrant and migrant populations, which are large in some high-prevalence countries. Benchmarks focusing on various aspects of equity include the equity for exposures to risk through intersectoral public health issues, financial and nonfinancial barriers to access to care, different levels of coverage in different parts of the health system, and equity in financing. •  ccountability: The acceptance of responsibility for and readiness to justify A decisions, acts, or failures to act has both value as a means to achieving performance and intrinsic value, in that people want to know something about how decisions were made. Accountability works at three levels. “Upward” ac- countability to congressional leaders is important for justifying how taxpayer money was spent and whether the goals authorized by decision makers were achieved. “Horizontal” accountability among funders and donors providing technical assistance is important for cooperation. A “downward” accountability refers to the responsibility to populations affected by the decisions and imple- mentation of the program. Accountability to program beneficiaries reinforces ownership of goals within a population, transparency about how the goals were established, and commitment to sustaining achievement around those goals over time. Benchmarks for accountability include democratic accountability and empowerment, and patient and provider autonomy. •  fficiency: Efficiency is the use of scarce resources in a way that gives value E for money. Benchmarks for efficiency include both clinical- and administrative- level efficiencies. SOURCE: Daniels et al.,1996. and selection of practitioner groups or health workers (Daniels, 2005). In the case of cost recovery, there are concerns about barriers to access created by charging for drugs, but user fees are arguably critical for sustainability of a program, a key element of efficiency. The issue of site selection was similarly controversial. Although mobilization of resources and trained personnel in tertiary care centers may be technically the most efficient solu- tion for rapid scale-up, a concentration of service delivery in areas where the largest numbers of people can be reached most rapidly leaves people in rural areas without a fair chance of any benefit because of poor access to services.

60 EVALUATING THE IMPACT OF PEPFAR Integration of Treatment with the Health System Another example of the tension that exists between health maximiza- tion and equity concerns the integration of ART treatment services with the rest of the health system. Despite direct investments in health system im- provement, there is concern about unintended effects of PEPFAR implemen- tation that may undermine other health programs in the form of negatively affecting personnel distribution in countries or draining parts of the health system toward politically driven programs. Daniels said that while AIDS in- terventions could be steering resources disproportionately away from other health problems, a focus on health system strengthening from the AIDS effort could also broaden the assistance with regard to other diseases. Equal Access to Care Through Randomized Trials Speaker Glennerster raised some of the ethical tensions that exist be- tween the goals of accessing care and generating knowledge about what interventions are effective through randomized trials. Because randomized trials traditionally require a control group, there is an ethical dilemma of excluding people from access to a program that might save their lives. Prevention and Treatment Another workshop participant raised the example of tension between the prevention and treatment approaches of PEPFAR, that is, between the people living with the disease who will die without treatment and those who will benefit from effective prevention programs—the uninfected population plus future generations. Daniels observed that while the ethical argument for prevention was initially based on “best outcomes” because it assumed that infected people would not have any chance of benefit and that not as many people would be helped overall if treatment were provided, that context has changed by the decreasing cost of treatment and by increasing political pressures favoring treatment. Indicators for Evaluating Fairness Daniels provided guidance on two types of measures that could be developed to assess the fairness of PEPFAR: (1) evaluation of the process of goal setting and (2) implementation and evaluation of the level of the program’s integration within the health system. To better understand and address basic aspects of equity, Daniels stressed, it is important in any eval- uation to disaggregate data and to collect it uniformly. Disaggregated data can tell us much about inequality related to gender, urban–rural access, age,

ENVISIONING A MEANINGFUL IMPACT EVALUATION 61 employment status, vulnerable and stigmatized groups, and immigrant and migrant populations. For example, noted speaker Binagwaho, evaluation of disaggregated data can tell us about the degree to which all geographic re- gions of a country and marginalized groups—prostitutes and prisoners—are benefiting from services. In Rwanda, she mentioned, prisoners now have access to the same prevention, care, and treatment services as communities, thanks to a successful national plan. The effectiveness of such programs can be assessed only if data on populations served are disaggregated. A Fair Process in Goal Setting and Implementation Processes for airing disagreements, surveying stakeholders for their in- put, and finding fair and legitimate solutions are constructive mechanisms for resolving tensions and for setting and implementing goals (Daniels and Sabin, 2002). Such processes have been used by the National AIDS Commission in Malawi and by planners of Mexico’s national insurance program. In Malawi, the National AIDS Commission set one of the best examples of a public ethics discussion by holding public hearings, involving stakeholders as members of the AIDS Commission, and publishing reports on decisions taken, including reasons why minority positions were not ad- opted (Daniels, 2005). Speaker Binagwaho contributed yet another example of a public ethics discussion from the Rwandan experience, in which people living with HIV/AIDS can themselves have a voice through their election to district-, provincial-, and national-level decision-making bodies. It is pos- sible to develop indicators to measure whether these types of processes are occurring as decisions are made about program implementation. Integration with the Health System Assessment of unintended fairness impacts requires monitoring of the extent to which a particular scale-up program is integrated with the health care system. A key monitoring indicator is population-disaggregated, geo- referenced information about the health system level of the site at which people receive treatment. Such data enables identification of where the ben- efits of scale-up are being delivered and to what parts of the population. Daniels noted that the issue of brain drain of trained health workers can be an element of evaluating PEPFAR integration with the broader health system. Useful information would include data on health personnel in parts of the health system that are adjacent to the ones where scale-up sites are being established. Such information would show whether health workers are being pulled out of places where services are needed. How developed countries meet their health care needs with the training of health workers and the impact of not meeting health care needs in devel-

62 EVALUATING THE IMPACT OF PEPFAR oping countries can also be explored through evaluation. Daniels noted that the United Kingdom has now put in place a specific code to correct some of the drain on human resources from the developing countries, including making monetary contributions to Malawi to eliminate some factors that were driving health workers out of the country. The existence and effective- ness of such compensatory mechanisms could be assessed through impact evaluation. Unintended Impacts Workshop participants discussed the need for evaluation to assess whether PEPFAR has had unintended consequences. Speaker Compton pointed out that both positive and negative synergies can occur and that methodologies are needed for capturing contextual data that will allow detection of these unintended impacts. Workshop participants discussed potential unintended effects of PEPFAR on program integration, diversion of resources from other health areas, corruption, access to services, ad- verse and high-risk behavior, nutrition, and reproductive health and family planning. Program Integration One workshop participant observed that evaluation could be used to assess the unintended impacts of earmarking of prevention funds on pro- gram integration. The participant hypothesized that earmarking of preven- tion funds may have resulted in more pipelining of funds, which in turn may have prevented integration of programs on the ground as separate groups of contractors arose to conduct AB programs independently of condom use programs, but few integrated abstinence–faithfulness–condom use (ABC) programs have arisen. Diversion of Resources Speakers Compton and Lengsfelder suggested that evaluation be used to consider unintended effects of PEPFAR on other diseases or health care areas. If shifts in emphasis have occurred, asked Lengsfelder, which areas are receiving the least attention? Compton suggested that the budgets and outcomes of malaria and less popular diseases should be tracked. Speaker Daulaire brought up the specific example of monitoring the change in al- location of resources to family planning, which he believed should be more actively sought as part of PEPFAR’s HIV/AIDS prevention and control strategy. While HIV programming from the United States has increased from $120 million a year in the mid-1990s to $5.3 billion, he noted, fam-

ENVISIONING A MEANINGFUL IMPACT EVALUATION 63 ily planning funding in the same period has been reduced 14 percent, even though the number of women of childbearing age has increased by 30 percent. Workshop participants also discussed the importance of monitoring un- intended impacts of PEPFAR on aspects of the broader health care system, such as the workforce and infrastructure. Compton suggested that evalua- tion assess the macroeconomic impacts of PEPFAR funding on brain drain from essential services resulting from hiring and per diem practices. Speaker Mwiindi proposed that unintended effects of PEPFAR’s large, multicoun- try supply chain program on existing supply chains be evaluated. Possible outcomes such as trading volume imbalance or brain drain from existing supply chains to the new system could be assessed. Discussant Ambassador Kolker suggested that evaluation be used to assess the extent to which PEPFAR’s investments in other diseases, in the health system, and in complementary development services (antipoverty programs, family-planning services, nutrition, women’s rights) have served as a counterweight to the diversion of attention from other areas. Corruption Speaker Compton suggested that evaluation be used to assess the effects of PEPFAR on corruption. She asked whether corrupt practices, such as double counting of infrastructure, are occurring in the context of PEPFAR implementation. Access to Services Speaker Pacqué-Margolis expressed interest in measuring possible posi- tive or negative effects of PEPFAR on the access and use of other services. For example, are wait times becoming longer and decreasing access, or are access and use of other services in the context of HIV/AIDS programs improving? Adverse and High-Risk Behavior Discussant Over noted that HIV/AIDS treatment programs may be in- advertent incentives for adverse behaviors, such as lack of drug adherence to maintain low CD4 (cluster of differentiation antigen 4) counts in order to qualify for disability, or expression of desire to become HIV positive to   CD4 is the receptor for HIV predominantly found on the surface of T-lymphocytes. The CD4 count is the number of helper CD4 T-lymphocytes in a cubic millimeter of blood. The absolute CD4 count declines as HIV infection progresses.

64 EVALUATING THE IMPACT OF PEPFAR qualify for programs. Effects such as these should be monitored through impact evaluation. As Over described previously in this chapter in the “Behavioral Change” section of “HIV/AIDS-Specific Health Impacts,” the availability of treatment may also encourage high-risk behavior because of the perception that AIDS is now treatable. Speaker Compton reinforced this argument by drawing from the experience in Zambia, where there is anecdotal evidence that abstinence programs have been successful in in- creasing the age of first sexual relations but have simultaneously decreased the amount of condom use. Compton asked if the emphasis on abstinence and faithfulness approaches may be unintentionally increasing the stigma of condom use. Nutrition Several speakers raised the importance of evaluating unintentional im- pacts of complementary programs, such as nutrition and food aid. Speaker Binagwaho mentioned the example of a nutrition program that pushed some HIV-positive families to have a child in order to obtain food; evalu- ation may shed light on the relationship between nutrition programs and childbearing. Speaker Compton noted that the effects of food-aid distribu- tion by PEPFAR on local markets also could be evaluated. Reproductive Health and Family Planning Unintended impacts of PEPFAR on reproductive health and family planning was identified by workshop participants as an area of interest for evaluation. Several participants noted that PEPFAR’s reproductive health programs may have had some positive synergies with HIV prevention, treatment, and care programs. One participant noted that family planning was the most cost-effective way to prevent mother-to-child transmission and urged impact evaluation to look more closely at the links between reproductive health and changing sexual behavior for the prevention of in- fection. Speaker Kenyon observed that through HIV counseling and testing procedures, more women than ever are receiving needed attention during pregnancy. At the same time, Kenyon noted an example of the potential for negative synergies between treatment and reproductive health in that women on ART in one country have been getting pregnant at a high rate. He observed that family planning guidance had not been updated in that country since 1995 but needed to be revisited in light of HIV and avail- ability of ART. Discussant Over highlighted the phenomenon of continued or restored fertility during treatment, observing that treatment programs may thus create more orphans. Citing data from work in India and other countries, Over posited that while treatment programs can avert orphan-

ENVISIONING A MEANINGFUL IMPACT EVALUATION 65 hood years among a woman’s current children, new orphanhood years can also be created among children born during the extended life span of a woman on treatment. Speakers Daulaire and Kenyon suggested that evaluation could be used to assess positive or negative synergistic effects of PEPFAR’s policies not to procure family planning commodities and contraceptives. While Daulaire argued that nonprocurement of such commodities may undermine HIV/ AIDS prevention and control, Kenyon asserted that such investments may divert resources from, and thereby weaken, HIV-related activities.

Next: 3 Designing an Evaluation That Incorporates the Guiding Principles of Coordination, Harmonization, and Capacity Building »
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary Get This Book
×
Buy Paperback | $50.00 Buy Ebook | $39.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Design Considerations for Evaluating the Impact of PEPFAR is the summary of a 2-day workshop on methodological, policy, and practical design considerations for a future evaluation of human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) interventions carried out under the President's Emergency Plan for AIDS Relief (PEPFAR), which was convened by the Institute of Medicine (IOM) on April 30 and May 1, 2007. Participants at the workshop included staff of the U.S. Congress; PEPFAR officials and implementers; major multilateral organizations such as The Global Fund to Fight AIDS, Malaria, and Tuberculosis (The Global Fund), the Joint United Nations Programme on HIV/AIDS (UNAIDS), and the World Bank; representatives from international nongovernmental organizations; experienced evaluation experts; and representatives of partner countries, particularly the PEPFAR focus countries. The workshop represented a final element of the work of the congressionally mandated IOM Committee for the Evaluation of PEPFAR Implementation, which published a report of its findings in 2007 evaluating the first 2 years of implementation, but could not address longer term impact evaluation questions.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!