2
Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting

This chapter distills the results of a visioning exercise conducted by workshop participants to identify, frame, and prioritize the questions that matter for the evaluation of HIV/AIDS interventions by the President’s Emergency Plan for AIDS Relief (PEPFAR). Workshop participants as diverse as congressional representatives, PEPFAR representatives, global partners, implementers, and country partners converged on some of the same impact evaluation questions.

Workshop participants stressed the importance of identifying meaningful and appropriate impact indicators and moving beyond quantitative outputs. Workshop speakers Jonathan Mwiindi of the Kijabe HIV/AIDS Relief Program, Kenya, and Mary Lyn Field-Nguer of John Snow, Inc., observed that the use of inappropriate indicators for evaluating program impact can sometimes tell an incomplete story, mislead, or mask other problems. For example, large numbers of individuals receiving first-line HIV treatment, Mwiindi noted, may mean that clinicians are not well trained to recognize when patients are in need of second-line drugs, not that the need for second-line drugs does not exist. Speakers Mwiindi and Kent Glenzer of the Cooperative for Assistance and Relief Everywhere, Inc. (CARE), described some of the drawbacks of and an overemphasis on quantitative measures. Mwiindi noted that an emphasis on counting sometimes drives inflated reporting as programs compete for limited funds. Glenzer observed that the



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 37
2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting This chapter distills the results of a visioning exercise conducted by workshop participants to identify, frame, and prioritize the questions that matter for the evaluation of HIV/AIDS interventions by the President’s Emergency Plan for AIDS Relief (PEPFAR). Workshop participants as di- verse as congressional representatives, PEPFAR representatives, global part- ners, implementers, and country partners converged on some of the same impact evaluation questions. Workshop participants stressed the importance of identifying meaning- ful and appropriate impact indicators and moving beyond quantitative out- puts. Workshop speakers Jonathan Mwiindi of the Kijabe HIV/AIDS Relief Program, Kenya, and Mary Lyn Field-Nguer of John Snow, Inc., observed that the use of inappropriate indicators for evaluating program impact can sometimes tell an incomplete story, mislead, or mask other problems. For example, large numbers of individuals receiving first-line HIV treatment, Mwiindi noted, may mean that clinicians are not well trained to recog- nize when patients are in need of second-line drugs, not that the need for second-line drugs does not exist. Speakers Mwiindi and Kent Glenzer of the Cooperative for Assistance and Relief Everywhere, Inc. (CARE), described some of the drawbacks of and an overemphasis on quantitative measures. Mwiindi noted that an emphasis on counting sometimes drives inflated re- porting as programs compete for limited funds. Glenzer observed that the 

OCR for page 37
 EVALUATING THE IMPACT OF PEPFAR rush to produce quantitative outputs for goals like 2–7–101 can sometimes jeopardize sustainability. He called for a reduction in the number of indica- tors and a focus on a more strategic set of results. Such prioritization could be accomplished using a criteria-based approach, noted one participant. The generation of improved indicators, noted discussant Caroline Ryan of the Office of the U.S. Global AIDS Coordinator (OGAC), will ultimately provide better information for making programmatic course corrections. Many participants argued in favor of a broader interpretation of im- pact as opposed to a narrower, or more pure, notion of impact. Many participants also believed that priorities and opportunities for evaluation include not only those describing the results of program implementation, but also the process of implementation—for example, coordination and capacity building—because the means of implementation are inseparable from the ends. Speaker Julia Compton of the UK Department for Interna- tional Development (DFID) noted that defining the boundaries of what to measure—around what is AIDS and what is AIDS spending—is a challenge, and that defining the boundaries too widely or too narrowly involves risks. Defining boundaries includes answering the question of whether to measure direct AIDS impacts very narrowly or to measure broader impacts such as gender empowerment or land rights or health systems. Workshop participants identified questions for evaluating impact that can be clustered into the following nine broad categories: cost-effectiveness, conceptual approach, health impacts, impacts beyond health, capacity building and health systems strengthening, coordination and harmoniza- tion, sustainability, equity and fairness, and unintended impacts. COST-EFFECTIvENESS Workshop participants described the importance of developing indica- tors that track the cost-effectiveness of PEPFAR. “Are we getting the biggest bang for our buck?” asked workshop speaker Christos Tsentas from the of- fice of Representative Barbara Lee. Discussant Mead Over of the Center for Global Development emphasized that cost-effectiveness of interventions— and whether costs can be afforded in particular countries once donor fund- ing is discontinued—has important implications for program sustainability. Cost-effectiveness measures can be used to evaluate different approaches, strategies, and interventions. Speaker Stefano Bertozzi of the National Institute of Public Health, Mexico, urged that cost-effectiveness measures be designed to assess types of prevention services delivered. He described a 1 PEPFAR’s 5-year goals—known as 2–7–10—are to support treatment for 2 million people, prevention of 7 million new infections, and care for 10 million people, including orphans and vulnerable children.

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION study of prevention service delivery in five countries in which a difference of three orders of magnitude in cost per service delivered was found: there were voluntary counseling and testing (VCT) programs that cost $7, $70, and $700 per client. Cost-effectiveness measures can help to ensure that VCT clinics not be established in contexts where there are not going to be any clients for them. Discussant Over recommended that cost-effectiveness measures also be used to test AIDS treatment interventions. Such analysis can determine the overall balance of adverse biological effects of treatment (that is, the spread of resistance or the increased opportunity of infection of longer living HIV-positive individuals), adverse behavioral effects of treatment (that is, increased risk behaviors), treatment effects on nontarget populations, and treatment effects on orphans. He added that cost-effectiveness evaluation could be used to test the assumption that PEPFAR should focus support on government-financed, free, high-quality therapy as compared to high- quality private-sector therapy. He noted that the support of public health care may be a problem in countries where government is weak and where private health care is abundant, but of poor quality. Workshop participants debated whether cost-effectiveness should be expressed in terms of results per donor investment versus results per ap- proach investment (also see Chapter 1, “Meaning and Uses of Impact Evaluation,” and Chapter 4, “Methodological Challenges and Opportuni- ties in Evaluating Impact”). Even if attribution cannot be shown, observed workshop moderator Ruth Levine of the Center for Global Development, it is useful to show what programs or approaches have the greatest impact. CONCEPTuAL APPROACH A number of proposed impact evaluation questions centered on the conceptual approach of PEPFAR and the assumptions that had been made in the program’s design: the countries and populations targeted, the budget allocations for different types of interventions, and the management and financing of the program. Such impact evaluation questions can be help- ful, noted workshop speaker Compton, in thinking through the conceptual model of how PEPFAR inputs lead to outputs and outcomes. Countries Targeted Workshop speakers Allen Moore of the Center for Strategic and Inter- national Studies and Compton urged PEPFAR decision makers to evaluate the impact and usefulness of working with the 15 “focus countries.” Is PEPFAR targeting focus countries appropriately? Compton added that im- pacts of regional factors should be evaluated; for example, focus-country

OCR for page 37
0 EVALUATING THE IMPACT OF PEPFAR programs should track the impact of migration from a nonfocus country that is a major source of infection. Populations Targeted Several speakers urged the development of impact indicators that track whether the appropriate populations are being targeted. Speaker Compton cited work by David Wilson of the World Bank showing that although 76 percent of adult male infections in Ghana are linked to commercial sex workers, only 1 percent of World Bank prevention funding supports pre- vention programs for sex workers (MOH and ORC Macro, 2006). She and speaker Theresa Diaz of the U.S. Centers for Disease Control and Preven- tion emphasized the importance of evaluating the targeting of high-risk and marginalized groups—such as commercial sex workers and injection drug users—who are the main drivers of epidemics in many countries. Speaker Bertozzi further observed that because patterns of risk behavior may vary by country, PEPFAR may want to evaluate how effectively it targets not only infected individuals and orphans, but also populations where new infections are expected to occur. Budget Allocations for Types of Interventions Considerable discussion focused on the value of assessing the impact of the budget allocations for prevention, treatment, and care on PEPFAR’s programming. Many participants cited the Institute of Medicine (IOM) study recommendations urging the removal of constraining congressional budget allocations (IOM, 2007). Observing that the current allocations came about as a result of highly effective advocacy by HIV treatment in- terests, speaker Field-Nguer called for an evidence-based assessment of the effectiveness of these allocations. Speaker Moore added that the usefulness of budget allocations and the appropriateness of funding priorities should be evaluated. Discussant Ambassador Jimmy Kolker, OGAC, suggested that evaluative information would be useful in informing future targets for scal- ing up prevention, care, and treatment interventions. Citing the imbalance in enrolling 90,000 people in treatment programs while 135,000 people were newly infected each year over the first phase of PEPFAR, speaker Compton urged the development of impact indicators to assess the appro- priate balance among prevention, treatment, and care interventions. She also suggested an evaluation of the balance among interventions within prevention, noting that a significant proportion of the prevention funding is required to support abstinence and faithfulness (AB) programs. Speaker David Gootnick of the U.S. Government Accountability Of- fice suggested that the assessment of prevention interventions provides an

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION opportunity for a deeper level of analysis that could inform the dialogue about earmarks. Citing the Futures Group GOALS model (Stover and Bollinger, 2006), he suggested that specific interventions (that is, partner reduction, male circumcision, opt-out testing, abstinence programs, and sex worker peer education programs) can be evaluated for their effectiveness in averting infections. Another workshop participant suggested that impact evaluation be used to assess the degree to which earmarks and pipeline programming may have translated into a lack of integration of programs on the ground. Several workshop participants stressed the importance of distinguish- ing between the evaluation of implementation of a particular policy on the ground and the evaluation of a policy as it is written. In many countries, noted speaker Jessica Price of Family Health International, there is still ambiguity in interpreting PEPFAR policies and how money can be used. For example, those receiving AB money interpreted PEPFAR policies to mean they could not educate youth about sexually transmitted infections. Simi- larly, many countries interpret differently whether or not PEPFAR funds can be used for family planning, and this is reflected in differential implementa- tion on the ground. Speaker Tom Kenyon of OGAC added that often there has been misinterpretation or overinterpretation of what could not be done, particularly with regard to implementation of condom programming. He noted that PEPFAR has had to modify guidance because there was a retreat from programming using condoms and other prevention approaches. Workshop participants stated that because earmarks have a political basis, as opposed to an empirical or a scientific basis, there may be less value in evaluating earmarks or in combining their evaluation with a more rigorous scientific evaluation. Speaker Jim Sherry of George Washington University noted that room is needed for political decision making about what is most important in terms of the evaluation agenda. Workshop par- ticipant Naomi Seiler from the U.S. House of Representatives Oversight Committee warned that it may be risky to mix scientific and objective evaluation with evaluation of more difficult political questions, such as evaluating a policy that prevents a certain type of intervention from taking place. As speaker Compton noted, “An evaluation cannot answer what goals are most important; that is a political decision.” Management and Financing Workshop participants also encouraged the design of impact evaluation measures for the management and financing of the PEPFAR program across a variety of levels. One participant urged the evaluation of the impact of PEPFAR’s practice of awarding hundreds of small grants to community or- ganizations. Speaker Sara Pacqué-Margolis of the Elizabeth Glaser Pediatric

OCR for page 37
 EVALUATING THE IMPACT OF PEPFAR AIDS Foundation encouraged the development of impact indicators for performance-based financing systems. Speaker Agnes Binagwaho of the Rwanda National AIDS Control Commission called for a development of metrics to assess the effectiveness and results of program comanagement, such as that piloted between PEPFAR and the government of Rwanda. Speaker Bertozzi remarked that evaluation could be used to test the effec- tiveness of different technical assistance approaches in different contexts. He noted that PEPFAR, the World Bank, and The Global Fund have very different approaches in terms of levels of technical assistance, with PEPFAR providing intensive technical assistance and The Global Fund providing countries with funds to purchase their own technical assistance. Evaluation may reveal that some approaches may work more efficiently for certain contexts; for example, where there is high local capacity, The Global Fund approach might be more efficient, and in a country with lower capacity, the PEPFAR approach might be more efficient. HEALTH IMPACTS Workshop participants suggested the development of impact evalua- tion measures for tracking change in health, including indicators specific to HIV/AIDS as well as more general indicators. HIv/AIDS-Specific Health Impacts Prevalence, Incidence, Morbidity, Mortality, Longevity Many workshop participants suggested that future impact evaluation of HIV-specific factors move beyond measurements of those accessing care and treatment to measurement of longer term health status. Because antiretrovi- ral (ARV) treatment may achieve viral suppression in a small percentage of patients receiving drugs, observed speaker Bertozzi, measuring the number of people on antiretroviral therapy (ART) may not be enough to determine the lifesaving effectiveness of a program. Speaker Jaime Sepúlveda of the University of California–San Francisco urged the use of state-of-the-art methods to measure prevalence and incidence of HIV infection. Speaker Field-Nguer spoke of the importance of measuring HIV/AIDS mortality and morbidity, and speaker Savannah Lengsfelder from the U.S. Senate Committee on Foreign Relations called for the development of measures of PEPFAR impact on longevity.

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION Prevention of HIV Transmission The full range of impacts of treatment on HIV transmission—both biological and behavioral—should be evaluated, urged discussant Over (Table 2-1). In terms of biological effects, ARV treatment can reduce viral loads, but it can have a negative impact on prevention because of resistance that comes about through imperfect adherence to drug regimens and a greater period of infectivity of longer living, HIV-positive individuals. He and speaker Mwiindi urged the development of indicators to measure the completeness of viral suppression, the level of drug adherence, and the level of resistance created by PEPFAR-supported treatment programs. In terms of behavioral effects, treatment programs can on one hand encourage health- seeking behaviors, including prevention behaviors, but those receiving ART as well as HIV-positive and HIV-negative individuals in the community may on the other hand engage in more risky or other adverse behaviors TABLE 2-1 Possible Effects of ART on HIV Transmission Direction of Effect Beneficial Adverse Type of Effect (Slow transmission) (Speedy transmission) Biological Reduce infectiousness. ART may Select for resistance. lower viral loads and may therefore Imperfect adherence to ART lower the risk of transmission per selects for resistant strains of sexual contact. the virus, which can then be transmitted. Longer duration of infectivity. The greater longevity of HIV-infected people taking ART has the unintended negative consequence of increasing the period in which the patient can transmit the virus. Behavioral Encourage prevention, especially Offsetting behavior. People diagnostic testing. ART may increase receiving ART, and HIV- the uptake rates of prevention positive and -negative activities, particularly voluntary people in the surrounding counseling and testing. community, may engage in more risky behaviors than they would if ART were unavailable. SOURCE: Over et al., 2007.

OCR for page 37
 EVALUATING THE IMPACT OF PEPFAR to preserve eligibility for programs than they would have if ART were not available. The extent to which treatment has contributed, through such behavioral channels, to widening the gap between the number of people on treatment and the number of people newly infected can be measured through evaluation. Speakers Mwiindi and Moore, along with moderator Levine, emphasized the importance of evaluating PEPFAR’s success in pre- vention of mother-to-child transmission of HIV (PMTCT) and in preventing HIV infection in children, adolescent girls, and women. Quality of Life Many speakers identified an important need to broaden HIV-specific in- dicators to include those measuring quality of life, which speaker Sepúlveda described as an evaluation of overall physical, mental, and social well- being of people affected by HIV/AIDS. According to speakers Field-Nguer, Price, and William Holzemer of the University of California–San Francisco, other attributes of quality of life that could be measured through impact evaluation include restored productivity; effective management of chronic, secondary symptoms of ART patients; palliative care and use of analgesics to support dying with dignity; and family grieving. Behavioral Change Another major area of HIV/AIDS-specific health indicators that could be tracked is human behavioral change, which, according to workshop speaker Sepúlveda, includes the rates of both risky and protective behav- iors. Speaker Tsentas enumerated several measures that could be developed to track changes in human behavior, including retention of patients on treatment, protected sexual intercourse, increased circumcision of males, and participation in needle-exchange programs. Discussant Over empha- sized the importance of evaluating the impacts of treatment on prevention behaviors, including VCT. He argued that treatment may itself encourage or “disinhibit” risky behavior because it creates a perception that HIV/AIDS is now curable. He noted that such an effect has already been observed, for example, as decreases in condom use among sex workers in Kenya follow- ing announcements of “false cures” (Jha et al., 2001). This phenomenon is further discussed later in this chapter under “Unintended Impacts.” Speaker Martha Ainsworth of the World Bank suggested that impact evaluation could be used to assess the effect of knowledge on behavioral change. She cited an example from Thailand in which the mere dissemination of knowl- edge that 44 percent of sex workers in Chiang Mai were infected with HIV had a dramatic effect on sexual behavior. Speaker Compton suggested that evaluation could be used to assess the impacts of a lack of investment in

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION specific behavioral change interventions, such as clean-needle distribution programs for intravenous drug users, which currently cannot be supported using PEPFAR funds. Another workshop participant wondered if evaluation could assess the “dose response” of interventions required to maintain a behavioral change over time. Speaker Shannon Hader of OGAC asserted that measuring behavioral outcomes is much more difficult than measuring treatment out- comes. Tools and methods to measure and understand behavioral change outcomes are described further in Chapter 4, “Methodological Challenges and Opportunities in Evaluating Impact.” Stigma and Discrimination A final area of HIV-specific assessment, noted speakers Sepúlveda and Field-Nguer, is stigma and discrimination. To what degree have PEPFAR’s interventions contributed to promoting the acceptance of people living with HIV/AIDS in the community and to reducing stigma? Monitoring and evaluation (M&E) should consider the impact of stigma on a variety of factors among both people living with HIV/AIDS and health care workers, suggested speaker Holzemer and discussant Timothy Fowler of the U.S. Bureau of the Census. These include participation in HIV testing, use of services (that is, antenatal care), poor treatment by health care providers, adherence to medications, health status, and quality of life, such as loss of social support, isolation, violence, and limiting social interactions because of fear. Other Health Impacts Future impact evaluation of PEPFAR could also consider other health indicators in addition to those related to HIV/AIDS. Evidence from the Rwanda context, noted speaker Allen Moore, suggests that many health indicators beyond those specific to HIV/AIDS have improved as a direct outcome of PEPFAR investments. (This case study is described in greater de- tail in Chapter 4, “Measuring Impacts of Health Systems Strengthening.”) Speaker Savannah Lengsfelder also spoke of the importance of evaluating any negative impacts of PEPFAR investments on other health conditions. Speakers Sepúlveda, Binagwaho, and Field-Nguer emphasized the impor- tance of examining the effects of PEPFAR programs on mortality by all causes, overall survival, and lives saved from other diseases. Understand- ing the infant and child mortality contribution to overall mortality is an important piece of this. Speaker Pacqué-Margolis pointed out that attri- butes of reproductive health should also be monitored, including fertility, unintended pregnancy, and intended pregnancy. As for HIV/AIDS patients,

OCR for page 37
 EVALUATING THE IMPACT OF PEPFAR quality-of-life indicators should be tracked, noted speaker Tsentas, as well as the effectiveness of interventions in protecting, educating, and equipping people with tools they need to take care of themselves. Speaker Lengsfelder suggested that data on other health indicators be disaggregated to show if differential effects are occurring in specific popula- tions and in focus countries versus nonfocus countries. IMPACTS BEyOND HEALTH What have been the effects of PEPFAR interventions in the areas of gender equality, child welfare, security and development, and institutional change? Workshop participants discussed a number of measures of HIV/ AIDS interventions beyond health that could be tracked through impact evaluation. Gender-Focused Activities Numerous speakers called for an evaluation of gender-focused ac- tivities, particularly those aimed at empowering women and girls. Gender- related dynamics occur in both men and women and influence their risk of contracting HIV, noted speaker Julie Pulerwitz of the Population Council. For men, gender norms encourage multiple sexual partners and early sexual debut. For women and girls, power imbalances result in an increased vul- nerability to HIV/AIDS. As speakers Glenzer and Tsentas noted, power imbalances may be reflected in factors such as early and child marriage, weaker ability to negotiate sexual relations, susceptibility to pressure to engage in transactional and intergenerational sex, vulnerability to sexual violence, poor education, lack of economic opportunities and exclusion from control of strategic economic assets, exclusion from decision-making processes and from patronage networks, lack of property and inheritance rights, and lack of legal and enforceable rights. Indicators for tracking such factors could be developed, suggested Glenzer, as well as indicators to measure the effectiveness of PEPFAR in addressing the underlying causes of women’s and girls’ vulnerability to HIV/AIDS. Glenzer also noted that building the analytical skills and com- petencies of men and women themselves around what is happening in their societies is a way to accelerate change in power structures and accelerate improvements, and the extent to which interventions build such capacities is something that can be measured through evaluation. Speaker Rachel Glennerster of the Abdul Latif Jameel Poverty Action Laboratory suggested that impact evaluation be used to identify the most effective messages in persuading teenage girls to change their behavior, and workshop modera- tor Levine suggested that programs be evaluated to assess their effective-

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION ness in reducing the spread of HIV infection to women and girls. Speaker Mwiindi introduced a nuance that the targeting and packaging of gender interventions should also be assessed through impact evaluation. Interven- tions aimed at sensitizing men may be perceived more positively and may be more effective in some cultures, he noted, than interventions targeting women. Child Welfare Several speakers highlighted the importance of evaluating parameters related to child welfare. Evaluation could measure the effectiveness of PEPFAR in raising the ability of adults to parent, suggested speaker Paul De Lay of the Joint United Nations Programme on HIV/AIDS (UNAIDS). Another participant observed that evaluation measures could track whether PEPFAR interventions had resulted in healthier, better nourished, and bet- ter educated orphans and vulnerable children who end up leading more productive lives as adults. Security, Development, and Poverty Alleviation Parameters of a country’s security and development could be tracked using impact evaluation. Speakers Binagwaho and De Lay suggested that impact evaluation be designed to track the effects of PEPFAR interventions on peace and security in a nation, poverty alleviation and economic growth, and general national development. Institutional and Societal Changes Speakers De Lay and Field-Nguer suggested that impact evaluation of PEPFAR also focus on tracking how societies and institutions are changing and improving. Policy changes such as property rights, inheritance laws, and human rights could be monitored, as well as changes in the contextual environment more indicative of supportive systems, such as political will, community ownership, food and water security, engagement of vulnerable populations, destigmatization, and antidiscrimination measures. Speaker Compton raised the question of what technical assistance is doing to help change political views and priorities in a country. IMPACTS ON SuSTAINABILITy, CAPACITy BuILDING, AND HEALTH SySTEMS STRENGTHENING Workshop participants emphasized the importance of monitoring both the process and results of building local capacity, particularly in the area

OCR for page 37
 EVALUATING THE IMPACT OF PEPFAR half of the focus countries have taken advantage of abstinence waivers,2 a mechanism that provides some flexibility in the coordination process. The existence and effectiveness of such mechanisms might be included in future evaluations of the impact of PEPFAR’s harmonization efforts with partner countries. Coordination Among Program Implementers A third area of coordination identified for prospective evaluation is the coordination among program implementers. Workshop discussant Ryan suggested that the development and effectiveness of a variety of coordi- nation tools, or mechanisms for fostering the exchange of learning, be evaluated. Speaker Field-Nguer gave an example of such a coordination tool: an implementers’ group that has been formed, through John Snow International and World Learning with Global Health Council, to foster learning about what is happening and what could be improved. This group has been using the experience of implementers in the field to influence the reauthorization process. Speaker Marconi described the development of two coordination tools whose effectiveness could be evaluated: (1) the PEPFAR extranet, a mechanism for information sharing accessible to all U.S. government employees, and (2) systematic literature reviews on the latest intervention research. The systematic reviews are conducted by the Cochrane Group and circulated to PEPFAR partners all over the world. Speakers Mwiindi and Compton suggested that harmonization of the drug procurement system can provide a useful case study for assessing co- ordination among program implementers. Mwiindi noted that PEPFAR’s entry into the drug supply chain—and the program’s exclusive sourcing of U.S. Food and Drug Administration (FDA)–approved drugs or branded drugs—has resulted in parallel streams of drug qualification, delays in drug delivery, and inconsistency with national protocols. Compton called for an impact evaluation of the policy requiring FDA approval of drugs distrib- uted through PEPFAR. Mwiindi added that the effectiveness of PEPFAR’s harmonization with existing drug procurement systems also be evaluated, which might include tracking measures such as drug quality, drug cost, reliability of the drug supply, and level of engagement with organizations on the ground, such as religious groups, that function in the existing health care delivery and drug supply system. 2 PEPFAR partner countries can apply for a waiver that allows them to reapportion their prevention funds among abstinence–faithfulness–condom use (ABC) interventions to reflect the nature of the local epidemic.

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION SuSTAINABILITy IMPACTS Workshop participants appealed for the development of evaluation measures that would assess PEPFAR’s effectiveness in evolving into a long- term, sustained response. Moderator Levine, along with speakers Moore and Gootnick, emphasized that making the transition to sustainability will be a challenge for what Moore termed a “hurry-up effort” that was legis- lated with a sense of urgency and began as an emergency response. Other participants stressed the importance of program sustainability given the long-term nature of the HIV/AIDS disease. Discussant Phillip Nieburg at the Center for Strategic and International Studies noted the importance of acknowledging that HIV/AIDS is neither an epidemic nor a pandemic, but endemic, which implies a sustainable response. Speaker De Lay also under- scored the need for PEPFAR to ensure effective chronic care for infected persons into the future. Measuring the financial sustainability of PEPFAR was a major focus of discussions, and speaker Compton warned that the financial implications for PEPFAR not continuing into the future would be tremendous, given the substantial financial burden that would be passed on to countries. Speaker Mwiindi suggested that evaluation could help prompt the development of and measure the progress toward a clear exit strategy. Measures could be developed, for example, of the degree to which the PEPFAR program has leveraged additional resources by other donors and by national govern- ments, remarked workshop discussant Sherry. Workshop participants also emphasized the importance of evaluation for long-term learning about the program. Moderator Levine suggested that evaluation can help to generate a technical consensus and a stock of knowledge about what works. Speaker Ainsworth added that information on what combination of services is most effective—a potential product of evaluation—can help to improve the efficiency and sustainability of the program. Evaluation of PEPFAR’s impacts on sustainability needs to include mea- sures of capacity building and promotion of local independence, workshop participants said. Speaker Compton noted that evaluation should assess the degree to which PEPFAR is making a sustained contribution to institutions and systems that include research, M&E, policy, budgeting, planning, and programming systems, as well as the degree to which PEPFAR is sustaining public-sector and voluntary staffing. Speaker Pulerwitz advised that evalu- ation of sustainability also take into account the degree to which “least dependency” practices are in place that are conducive to local implementa- tion, ownership, and coordination with national systems and structures.

OCR for page 37
 EVALUATING THE IMPACT OF PEPFAR EQuITy AND FAIRNESS IMPACTS Speaker Norman Daniels of the Harvard School of Public Health fo- cused his remarks on the importance of evaluating the fairness of the PEPFAR program. He introduced the concept of fairness and defined its subcomponents—equity, accountability, and efficiency. He provided ex- amples showing the trade-offs and tensions between the competing goals of equity and efficiency in decision making. Finally, he offered guidance on indicators that can be developed to examine issues of equity as PEPFAR programs are scaled up. Defining Fairness Aspects of fairness are central in decisions to scale up ART treatment and prevention, noted Daniels. In the context of domestic and international health policy, the goal is to improve population health, as measured in the aggregate, but also to distribute the benefits of health policy in a fair way and reduce disparities. These dual goals may conflict with each other, how- ever, due to conflicting priorities. A framework integrating concerns about equity, accountability, and efficiency can be used for evaluating the fairness of health interventions in developing countries (Box 2-2). Benchmarks can be developed for each component and indicators generated to measure the improvement that a particular intervention produces for each of these components. Trade-offs Among Competing Goals in Evaluating Fairness The evaluation of fairness involves not simply the maximization of a specific outcome, but the way in which resources are being used to achieve several goals, which may at times conflict with each other. Trade-offs can exist, when attempting to achieve the goals of equity and efficiency, be- tween getting the best outcomes with scarce resources and giving people fair chances at some benefits. Several examples of trade-offs in evaluating fairness were shared. World Health Organization  by  Initiative Issues with equity implications have arisen in the context of scale-up decisions of the World Health Organization’s (WHO’s) 3 by 5 Initiative,3 such as cost recovery, eligibility criteria for patient selection, site selection, 3 The WHO 3 by 5 Initiative was a global target to provide ART to 3 million people living with HIV/AIDS in low- and middle-income countries by the end of 2005.

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION BOX 2-2 The Elements of Fairness Equity: A key aspect of equity is the absence of unjustifiable inequalities or • disparities across demographic groups. Such inequalities might include gen- der inequality, urban–rural inequality in access, child versus adult inequality, inequality of vulnerable and stigmatized groups, and inequality of immigrant and migrant populations, which are large in some high-prevalence countries. Benchmarks focusing on various aspects of equity include the equity for exposures to risk through intersectoral public health issues, financial and nonfinancial barriers to access to care, different levels of coverage in different parts of the health system, and equity in financing. • ccountability: The acceptance of responsibility for and readiness to justify A decisions, acts, or failures to act has both value as a means to achieving performance and intrinsic value, in that people want to know something about how decisions were made. Accountability works at three levels. “Upward” ac- countability to congressional leaders is important for justifying how taxpayer money was spent and whether the goals authorized by decision makers were achieved. “Horizontal” accountability among funders and donors providing technical assistance is important for cooperation. A “downward” accountability refers to the responsibility to populations affected by the decisions and imple- mentation of the program. Accountability to program beneficiaries reinforces ownership of goals within a population, transparency about how the goals were established, and commitment to sustaining achievement around those goals over time. Benchmarks for accountability include democratic accountability and empowerment, and patient and provider autonomy. • fficiency: Efficiency is the use of scarce resources in a way that gives value E for money. Benchmarks for efficiency include both clinical- and administrative- level efficiencies. SOURCE: Daniels et al.,1996. and selection of practitioner groups or health workers (Daniels, 2005). In the case of cost recovery, there are concerns about barriers to access created by charging for drugs, but user fees are arguably critical for sustainability of a program, a key element of efficiency. The issue of site selection was similarly controversial. Although mobilization of resources and trained personnel in tertiary care centers may be technically the most efficient solu- tion for rapid scale-up, a concentration of service delivery in areas where the largest numbers of people can be reached most rapidly leaves people in rural areas without a fair chance of any benefit because of poor access to services.

OCR for page 37
0 EVALUATING THE IMPACT OF PEPFAR Integration of Treatment with the Health System Another example of the tension that exists between health maximiza- tion and equity concerns the integration of ART treatment services with the rest of the health system. Despite direct investments in health system im- provement, there is concern about unintended effects of PEPFAR implemen- tation that may undermine other health programs in the form of negatively affecting personnel distribution in countries or draining parts of the health system toward politically driven programs. Daniels said that while AIDS in- terventions could be steering resources disproportionately away from other health problems, a focus on health system strengthening from the AIDS effort could also broaden the assistance with regard to other diseases. Equal Access to Care Through Randomized Trials Speaker Glennerster raised some of the ethical tensions that exist be- tween the goals of accessing care and generating knowledge about what interventions are effective through randomized trials. Because randomized trials traditionally require a control group, there is an ethical dilemma of excluding people from access to a program that might save their lives. Prevention and Treatment Another workshop participant raised the example of tension between the prevention and treatment approaches of PEPFAR, that is, between the people living with the disease who will die without treatment and those who will benefit from effective prevention programs—the uninfected population plus future generations. Daniels observed that while the ethical argument for prevention was initially based on “best outcomes” because it assumed that infected people would not have any chance of benefit and that not as many people would be helped overall if treatment were provided, that context has changed by the decreasing cost of treatment and by increasing political pressures favoring treatment. Indicators for Evaluating Fairness Daniels provided guidance on two types of measures that could be developed to assess the fairness of PEPFAR: (1) evaluation of the process of goal setting and (2) implementation and evaluation of the level of the program’s integration within the health system. To better understand and address basic aspects of equity, Daniels stressed, it is important in any eval- uation to disaggregate data and to collect it uniformly. Disaggregated data can tell us much about inequality related to gender, urban–rural access, age,

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION employment status, vulnerable and stigmatized groups, and immigrant and migrant populations. For example, noted speaker Binagwaho, evaluation of disaggregated data can tell us about the degree to which all geographic re- gions of a country and marginalized groups—prostitutes and prisoners—are benefiting from services. In Rwanda, she mentioned, prisoners now have access to the same prevention, care, and treatment services as communities, thanks to a successful national plan. The effectiveness of such programs can be assessed only if data on populations served are disaggregated. A Fair Process in Goal Setting and Implementation Processes for airing disagreements, surveying stakeholders for their in- put, and finding fair and legitimate solutions are constructive mechanisms for resolving tensions and for setting and implementing goals (Daniels and Sabin, 2002). Such processes have been used by the National AIDS Commission in Malawi and by planners of Mexico’s national insurance program. In Malawi, the National AIDS Commission set one of the best examples of a public ethics discussion by holding public hearings, involving stakeholders as members of the AIDS Commission, and publishing reports on decisions taken, including reasons why minority positions were not ad- opted (Daniels, 2005). Speaker Binagwaho contributed yet another example of a public ethics discussion from the Rwandan experience, in which people living with HIV/AIDS can themselves have a voice through their election to district-, provincial-, and national-level decision-making bodies. It is pos- sible to develop indicators to measure whether these types of processes are occurring as decisions are made about program implementation. Integration with the Health System Assessment of unintended fairness impacts requires monitoring of the extent to which a particular scale-up program is integrated with the health care system. A key monitoring indicator is population-disaggregated, geo- referenced information about the health system level of the site at which people receive treatment. Such data enables identification of where the ben- efits of scale-up are being delivered and to what parts of the population. Daniels noted that the issue of brain drain of trained health workers can be an element of evaluating PEPFAR integration with the broader health system. Useful information would include data on health personnel in parts of the health system that are adjacent to the ones where scale-up sites are being established. Such information would show whether health workers are being pulled out of places where services are needed. How developed countries meet their health care needs with the training of health workers and the impact of not meeting health care needs in devel-

OCR for page 37
 EVALUATING THE IMPACT OF PEPFAR oping countries can also be explored through evaluation. Daniels noted that the United Kingdom has now put in place a specific code to correct some of the drain on human resources from the developing countries, including making monetary contributions to Malawi to eliminate some factors that were driving health workers out of the country. The existence and effective- ness of such compensatory mechanisms could be assessed through impact evaluation. uNINTENDED IMPACTS Workshop participants discussed the need for evaluation to assess whether PEPFAR has had unintended consequences. Speaker Compton pointed out that both positive and negative synergies can occur and that methodologies are needed for capturing contextual data that will allow detection of these unintended impacts. Workshop participants discussed potential unintended effects of PEPFAR on program integration, diversion of resources from other health areas, corruption, access to services, ad- verse and high-risk behavior, nutrition, and reproductive health and family planning. Program Integration One workshop participant observed that evaluation could be used to assess the unintended impacts of earmarking of prevention funds on pro- gram integration. The participant hypothesized that earmarking of preven- tion funds may have resulted in more pipelining of funds, which in turn may have prevented integration of programs on the ground as separate groups of contractors arose to conduct AB programs independently of condom use programs, but few integrated abstinence–faithfulness–condom use (ABC) programs have arisen. Diversion of Resources Speakers Compton and Lengsfelder suggested that evaluation be used to consider unintended effects of PEPFAR on other diseases or health care areas. If shifts in emphasis have occurred, asked Lengsfelder, which areas are receiving the least attention? Compton suggested that the budgets and outcomes of malaria and less popular diseases should be tracked. Speaker Daulaire brought up the specific example of monitoring the change in al- location of resources to family planning, which he believed should be more actively sought as part of PEPFAR’s HIV/AIDS prevention and control strategy. While HIV programming from the United States has increased from $120 million a year in the mid-1990s to $5.3 billion, he noted, fam-

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION ily planning funding in the same period has been reduced 14 percent, even though the number of women of childbearing age has increased by 30 percent. Workshop participants also discussed the importance of monitoring un- intended impacts of PEPFAR on aspects of the broader health care system, such as the workforce and infrastructure. Compton suggested that evalua- tion assess the macroeconomic impacts of PEPFAR funding on brain drain from essential services resulting from hiring and per diem practices. Speaker Mwiindi proposed that unintended effects of PEPFAR’s large, multicoun- try supply chain program on existing supply chains be evaluated. Possible outcomes such as trading volume imbalance or brain drain from existing supply chains to the new system could be assessed. Discussant Ambassador Kolker suggested that evaluation be used to assess the extent to which PEPFAR’s investments in other diseases, in the health system, and in complementary development services (antipoverty programs, family-planning services, nutrition, women’s rights) have served as a counterweight to the diversion of attention from other areas. Corruption Speaker Compton suggested that evaluation be used to assess the effects of PEPFAR on corruption. She asked whether corrupt practices, such as double counting of infrastructure, are occurring in the context of PEPFAR implementation. Access to Services Speaker Pacqué-Margolis expressed interest in measuring possible posi- tive or negative effects of PEPFAR on the access and use of other services. For example, are wait times becoming longer and decreasing access, or are access and use of other services in the context of HIV/AIDS programs improving? Adverse and High-Risk Behavior Discussant Over noted that HIV/AIDS treatment programs may be in- advertent incentives for adverse behaviors, such as lack of drug adherence to maintain low CD4 (cluster of differentiation antigen 4) counts4 in order to qualify for disability, or expression of desire to become HIV positive to 4 CD4 is the receptor for HIV predominantly found on the surface of T-lymphocytes. The CD4 count is the number of helper CD4 T-lymphocytes in a cubic millimeter of blood. The absolute CD4 count declines as HIV infection progresses.

OCR for page 37
 EVALUATING THE IMPACT OF PEPFAR qualify for programs. Effects such as these should be monitored through impact evaluation. As Over described previously in this chapter in the “Behavioral Change” section of “HIV/AIDS-Specific Health Impacts,” the availability of treatment may also encourage high-risk behavior because of the perception that AIDS is now treatable. Speaker Compton reinforced this argument by drawing from the experience in Zambia, where there is anecdotal evidence that abstinence programs have been successful in in- creasing the age of first sexual relations but have simultaneously decreased the amount of condom use. Compton asked if the emphasis on abstinence and faithfulness approaches may be unintentionally increasing the stigma of condom use. Nutrition Several speakers raised the importance of evaluating unintentional im- pacts of complementary programs, such as nutrition and food aid. Speaker Binagwaho mentioned the example of a nutrition program that pushed some HIV-positive families to have a child in order to obtain food; evalu- ation may shed light on the relationship between nutrition programs and childbearing. Speaker Compton noted that the effects of food-aid distribu- tion by PEPFAR on local markets also could be evaluated. Reproductive Health and Family Planning Unintended impacts of PEPFAR on reproductive health and family planning was identified by workshop participants as an area of interest for evaluation. Several participants noted that PEPFAR’s reproductive health programs may have had some positive synergies with HIV prevention, treatment, and care programs. One participant noted that family planning was the most cost-effective way to prevent mother-to-child transmission and urged impact evaluation to look more closely at the links between reproductive health and changing sexual behavior for the prevention of in- fection. Speaker Kenyon observed that through HIV counseling and testing procedures, more women than ever are receiving needed attention during pregnancy. At the same time, Kenyon noted an example of the potential for negative synergies between treatment and reproductive health in that women on ART in one country have been getting pregnant at a high rate. He observed that family planning guidance had not been updated in that country since 1995 but needed to be revisited in light of HIV and avail- ability of ART. Discussant Over highlighted the phenomenon of continued or restored fertility during treatment, observing that treatment programs may thus create more orphans. Citing data from work in India and other countries, Over posited that while treatment programs can avert orphan-

OCR for page 37
 ENVISIONING A MEANINGFUL IMPACT EVALUATION hood years among a woman’s current children, new orphanhood years can also be created among children born during the extended life span of a woman on treatment. Speakers Daulaire and Kenyon suggested that evaluation could be used to assess positive or negative synergistic effects of PEPFAR’s policies not to procure family planning commodities and contraceptives. While Daulaire argued that nonprocurement of such commodities may undermine HIV/ AIDS prevention and control, Kenyon asserted that such investments may divert resources from, and thereby weaken, HIV-related activities.

OCR for page 37