National Academies Press: OpenBook

Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary (2008)

Chapter: 1 Introduction to Impact Evaluation for PEPFAR

« Previous: Overview
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 23
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 24
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 25
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 26
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 27
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 28
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 29
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 30
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 31
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 32
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 33
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 34
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 35
Suggested Citation:"1 Introduction to Impact Evaluation for PEPFAR." Institute of Medicine. 2008. Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12147.
×
Page 36

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction to Impact Evaluation for PEPFAR This chapter summarizes discussions at the workshop about the mean- ing and uses of impact evaluation. Uses of impact evaluation to judge performance—summative evaluation—and to inform decision making for program improvement—formative evaluation—are described. Next, the chapter reviews the approach for evaluating human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) interventions car- ried out through the President’s Emergency Plan for AIDS Relief (PEPFAR) and how that approach has evolved over time. Finally, the chapter consid- ers the major findings of the Institute of Medicine’s (IOM’s) preliminary evaluation of PEPFAR, PEPFAR Implementation: Progress and Promise (IOM, 2007), including recommendations to inform the design of future impact evaluations. Meaning and Uses of Impact Evaluation Defining Impact Evaluation Workshop moderator Ruth Levine of the Center for Global Develop- ment proposed a definition of impact evaluation as a measurement of net change in outcomes attributable to a specific program using a methodology that is robust, available, feasible, and appropriate, both to the question under investigation and to the specific context. In the context of PEPFAR, she noted, impact evaluation can provide insights about the outcomes from specific interventions, types of approaches, or different methodologies. 23

24 EVALUATING THE IMPACT OF PEPFAR Impact concerns not only outcomes, but also the change that leads to outcomes, noted speaker Julia Compton of the UK Department for Interna- tional Development. Impact evaluation, therefore, is a combination of both means—or processes—and ends, observed speaker Mary Lyn Field-Nguer of John Snow, Inc. Speakers Compton and Sara Pacqué-Margolis of the Elizabeth Glaser Pediatric AIDS Foundation emphasized the long-term nature of impact evaluation. Compton observed the importance of considering longer term results, such as unintended effects and sustainability, in impact evalua- tion. She noted that the Organisation for Economic Co-operation and Development’s Development Assistance Committee includes such concepts in its definition of impact. In the context of evaluating the impact of PEPFAR, several speakers called for a shift to a broader definition of impact evaluation and to a more nonlinear concept of causation. Speaker Paul De Lay of the Joint United Nations Programme on HIV/AIDS (UNAIDS) observed that although the traditional definition for infectious disease impact evaluation of prevalence, incidence, infections averted, morbidity, and mortality is important, an as- sessment about what PEPFAR has actually accomplished should include broader concepts such as the intensity, quality, targeting, and equity of services. Key aspects of economic development and social change should also be built into the definition, he noted. Pacqué-Margolis added that a broader definition of impact evaluation would include measuring changes in health status, systems capacity, and other social, economic, and political outcomes. All impact evaluations must be based on a conceptual model of causation and intervention, observed speaker Nils Daulaire of the Global Health Council, but cultural-, political-, and location-specific factors and shifting influences make causality in the world of HIV/AIDS highly non- linear. Innovative thinking may therefore be required when designing an impact evaluation for such a system. Uses of Impact Evaluation How is impact evaluation used? Workshop participants discussed uses of impact evaluation both to assess whether a project met its goals and to inform improvement in a project or indicate the need for midcourse corrections. Summative Evaluation for Accountability and Advocacy Impact evaluation has an important role in judging the performance of a program in order to account to specific constituents. In the case of PEPFAR, noted speakers De Lay and Agnes Binagwaho of the Rwanda Na-

INTRODUCTION TO IMPACT EVALUATION 25 tional AIDS Control Commission, impact evaluation is important to inform congressional decision makers, and the U.S. taxpayers they represent, about the success or failure of interventions. Evaluative information in turn has an important advocacy function, noted Pacqué-Margolis. To ensure sustained funding for a program, effort must be invested in interpreting and using research findings. Speaker Daulaire and moderator Levine described the tensions that exist between attributing impact to particular funders and building knowledge— regardless of attribution to a particular funder—about what programs or interventions work. Even among congressional staff at the workshop, there was a difference of opinion about what type of attribution—to funders or to programs—is most useful. While speaker Savannah Lengsfelder from the Senate Committee on Foreign Relations emphasized the importance of focusing less on money and the return on investment per U.S. taxpayer dollar than on the success or failure of programs, speaker Christos Tsentas from Representative Barbara Lee’s office acknowledged that data on what proportion of a program the United States supports are very helpful to promote a particular program. Attribution is further discussed in Chapter 4, in the section titled “Methodological Challenges and Opportunities in Evaluating Impact.” Formative Evaluation for Improved Decision Making Workshop participants agreed that a formative, or utilization-focused, emphasis on evaluation—to improve decision making and course correction related to a particular program—is important for better and longer lasting results. “Qualitative and operations research are a critical part of the learn- ing and doing process,” said speaker Daulaire. It is important to know who is taking the decisions informed by evalu- ation, observed speaker Compton: Who needs information at the top level, what information do they need, and when do they need it? What infor- mation is needed by the people who are taking decisions at the ground level? Some consideration for how the impact evaluation’s results will be packaged, disseminated, and used at these multiple levels is important in evaluation design, added Pacqué-Margolis. Speaker Jonathan Mwiindi of the Kijabe HIV/AIDS Relief Program in Kenya commented that decision makers at different levels may have different evaluation needs; sometimes the results and indicators of donors will benefit the donor but will be of little use to local operations. Speaker Binagwaho appealed to workshop participants to remember that local decision makers have decision-making needs that are just as important as those of top-level decision makers. “Impact evaluation should not just be for Congress,” she asserted.

26 EVALUATING THE IMPACT OF PEPFAR Decision making in Congress.  Workshop participants discussed what Con- gress wants and needs to learn from the impact evaluation. Speaker Allen Moore of the Center for Strategic and International Studies emphasized that a primary role for Congress now is to design the reauthorization of PEPFAR. He noted that data from impact evaluation will help speed the reauthorization process, which could prevent the undesirable extension of the program on a yearly basis, via the Foreign Assistance Act, through an appropriations bill. More rapid progress on PEPFAR reauthorization would have added benefits, Moore noted, in sending signals to recipient countries to increase internal investments in health, to donors to shoulder their con- tribution to HIV/AIDS, and to implementing partners who have geared up, built up, and hired staff and need assurance that the program will continue to support their efforts. Discussant Jim Sherry of George Washington University further rein- forced the value of progress on reauthorization in leveraging investments by the United States, international partners, and local partners. Although we are still early in the epidemic, Sherry noted, since phase 1 of PEPFAR, there has been a 30-fold increase in funding from other U.S. sources, fund- ing from other countries, and an increase in spending by national govern- ments to about half of the total resources. Sherry also remarked that a change in political context—a new president and a Democrat-dominated Congress—at the time of PEPFAR reauthorization may have implications for the level of support for investments in global health. Many workshop participants emphasized the importance of commu- nicating impacts of key provisions of the PEPFAR legislation—such as the focus country model and the earmarks for investments across preven- tion, treatment, and care interventions—to congressional decision makers. Speaker Daulaire stressed that implementers need to convey to both advo- cates and policy makers what works and what does not, so that what is being pushed has relevance on the ground. Programmatic decision making in partner countries.  Impact evaluation also has value for decision makers at the level of partner countries. Speaker Binagwaho outlined the following benefits of evaluation results at the country level: • To improve performance • To continue and expand good initiatives • To improve planning, monitoring, and evaluation • To provide examples of implementation practices where available opportunities and resources have been optimally used

INTRODUCTION TO IMPACT EVALUATION 27 Speaker Jody Kusek of the World Bank emphasized the importance of relevant, high-quality, and widely disseminated evaluation information for optimizing the usefulness of evaluation results in changing practices and policies on the ground. In Swaziland, she noted, wide dissemination of the results of trials in Kenya and Uganda showing that surgically appropriate male circumcision provides significant protection against HIV infection has stimulated the crafting of a new policy to create better access to the intervention. Speaker Field-Nguer stressed that the data collected by a program should be usable for program improvements and accessible and understand- able to providers, managers, and policy makers. A closed communication loop between program managers and service providers about service data collected and reported is critical to program improvements. PEPFAR’s Evaluative Approach PEPFAR’s Approach to Strategic Information Speaker Tom Kenyon, principal deputy U.S. Global AIDS Coordinator and chief medical officer of the Office of the U.S. Global AIDS Coordina- tor (OGAC), provided an overview of PEPFAR’s strategic information—or monitoring and evaluation (M&E)—approach. Kenyon noted that the ap- proach emphasizes the importance of sharing information not only verti- cally to the executive and legislative branches and to the taxpayer, but also horizontally to international partners and to country-level partners. PEPFAR’s monitoring approach is based on “breaking out of the donor– recipient paradigm” to a model of true partnership in which multiple U.S. agencies are coordinating with host-country agencies. One aim of PEPFAR’s monitoring approach is to stimulate a culture of accountability in partner countries through the establishment of monitoring systems, such as national health interview surveys. A key aspect of PEPFAR’s monitoring process and infrastructure is to track progress toward the program’s goals in prevention, treatment, and care (see Box 1-1 for an overview of PEPFAR). Other M&E priorities include enhanced surveillance of behavior incidence and prevalence, HIV incidence and prevalence, drug-resistant HIV, HIV within the tuberculosis (TB) population, and drug-resistant TB. Mortality, morbidity, orphans averted, and social and economic change are other PEPFAR impact mea- sures mentioned in a later presentation by Theresa Diaz of the U.S. Centers for Disease Control and Prevention. According to Kenyon, both the overall budget for M&E and the proportion of the budget devoted to field versus central operations have increased over the period 2004–2007 (Figure 1-1). Budgets for strategic

28 EVALUATING THE IMPACT OF PEPFAR BOX 1-1 Introduction to PEPFAR In May 2003, the U.S. Congress passed the United States Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 2003 (The Leadership Act) and es- tablished the U.S. Global AIDS Initiative. The legislation required the executive branch to establish a comprehensive 5-year strategy to combat HIV/AIDS, the President’s Emergency Plan for AIDS Relief (PEPFAR). The legislation also es- tablished the position of the Global AIDS Coordinator within the U.S. Department of State to oversee and coordinate all U.S. international activities conducted by numerous U.S. government agencies, including the U.S. Agency for International Development; the Centers for Disease Control and Prevention, the Food and Drug Administration, the Health Resources and Services Administration, and the Substance Abuse and Mental Health Services Administration of the Department of Health and Human Services; the Department of Defense; the U.S. Peace Corps; the U.S. Census Bureau; and the Department of Labor. The first 5-year phase of PEPFAR funding, from 2004 to 2008, is funded at $15 billion. The U.S. Global AIDS Initiative focuses on 15 partner countries selected on the basis of their ability to scale up prevention, treatment, and care response by 2009. PEPFAR’s 5-year performance targets for the 15 focus countries include • Prevention of 7 million new HIV infections • Treatment of 2 million HIV-infected people • Care for 10 million people infected with and affected by HIV/AIDS, including orphans and vulnerable children These targets—generated using limited HIV incidence and prevalence data available in 2003—represented at the time of PEPFAR authorization about half of those eligible for treatment, half of those in need of care, and half of the new infections, said speaker Tom Kenyon of the Office of the U.S. Global AIDS Coor- dinator. There were assumptions that some HIV-positive individuals would never seek services and others would obtain services from the private sector. SOURCE: Kenyon, 2007. information are about 5 percent to 7 percent of overall country budgets in fiscal years (FYs) 2004–2007. PEPFAR is expanding each country’s reporting infrastructure and in- creasing the number of personnel who are trained in the field of strategic information. In country, the PEPFAR approach is based on the following principles:

INTRODUCTION TO IMPACT EVALUATION 29 140 120 100 $ Millions 80 60 40 20 0 2004 2005 2006 2007 Field Central FIGURE 1-1 PEPFAR strategic information budget, 2004–2007. SOURCE: Kenyon, 2007. 1-1 • Supporting local leadership and ownership of HIV/AIDS response by encouraging countries to develop one national plan, one coordinating mechanism, and one monitoring and evaluation plan • Building capacity of indigenous partners, including infrastruc- ture and human capacity, through support of an annual implementers meetings • Using local health infrastructure and community structures • Implementing the program according to national guidelines • Monitoring using internationally agreed-upon indicators • Funding programs based on results Kenyon presented PEPFAR’s FY 2006 achievements, stating that 83 percent of partners were local organizations supporting 15,000 project sites: • Antiretroviral treatment for 822,000 people; treatment programs projected to save 3.4 million life years by 2009   One national plan, one coordinating mechanism, and one monitoring and evaluation plan are known as “the three ones.”

30 EVALUATING THE IMPACT OF PEPFAR • Prevention of mother-to-child transmission of HIV (PMTCT) ser- vices for women during more than 6 million pregnancies, averting an esti- mated 101,500 infant infections • 18.7 million counseling and testing sessions for men, women, and children • Care for nearly 4.5 million people, including more than 2 million orphans and vulnerable children • Aversion of approximately 229,000 orphans through 2006 us- ing treatment programs for parents; projected aversion of approximately 864,000 orphans through 2008 Evolution of Evaluation in PEPFAR Speaker Kathy Marconi, director of monitoring, evaluation, and stra- tegic information from OGAC, described how M&E have evolved over the life of the PEPFAR program and how they may change in the future. Evaluation at the Inception of PEPFAR PEPFAR’s initial definition of impact—infections averted and lives saved—was narrow, linear, and basic. Capacity for monitoring and avail- ability of data were similarly limited at that time. Within the U.S. govern- ment, each agency had its own reporting system, and there was a lack of consistent targets across countries. Program results were not yet based on robust data; for example, epidemic modeling was based primarily on urban antenatal clinic (ANC) sentinel sites. Although Marconi noted a general lack of country capacity to collect strategic information, some resources were available, such as in-country M&E leadership, a global commitment to M&E harmonization, international indicators of UNAIDS, reference groups working on modeling and M&E systems, and U.S. government tech- nical resources for surveillance, surveys, information, and communication. PEPFAR’s strategic information reporting was fed by an annual planning and reporting cycle for target setting, program implementation, funding tracking, and results. Development of an Evaluation Framework At the initiation of PEPFAR, evaluation planners established an “ideal” national strategic information system that would be informed by different surveillance approaches. Input on scale-up and coverage would be collected from facility surveys; estimates of behavioral change would be gathered from population-based surveys; and information on prevalence of HIV in- fection would be gathered from serum surveys at ANCs. Periodic targeted

INTRODUCTION TO IMPACT EVALUATION 31 evaluations would be conducted on selected topics. The evaluation frame- work developed using this simple logic model provided the results needed initially concerning, for example, the deployment and use of funds, the development and delivery of services, and the beneficiaries of program ser- vices. For example, in the context of treatment, the evaluation framework enabled PEPFAR to measure PMTCT, care, counseling, testing, scale-up over time, and people reached. Trends such as gender, children reached, infections averted, and years of life added through antiretroviral therapy can be measured using the framework. Future Design of PEPFAR Monitoring and Evaluation As the complexity of PEPFAR program strategies grows in the future, the definition of impact will need to be broadened to include factors such as long-term sustainability, health care workforce systems development, and other aspects of development such as nutrition, clean water, education, and gender equity. These factors will need to be reflected in the next evaluation framework. Future M&E frameworks for PEPFAR may be informed by the devel- opment and strengthening of a number of new tools, including a UNAIDS methodology for helping countries define their own evaluation targets, health provider reporting systems, surveillance studies, population surveys, HIV testing tools, and supply chain management tools. A Global Fund impact study also will be available soon to provide information on disease rates, mortality, morbidity, and health systems within countries. In the discussion following the presentation, speaker Marconi and workshop participants discussed some of the measurement challenges that will need to be addressed by planners of future PEPFAR evaluations. Several participants expressed concern that evaluation of progress on macro-level indicators needs to move beyond counting numbers of people touched by the program. For example, numbers of patients receiving drugs may not capture whether they are staying alive longer. Similarly, evaluation of orphan and vulnerable children programs may not capture impacts on improved nutrition, better education, and strengthened capability to be productive adults. Workshop participants also noted that predictive tools and evidence for assessing the effectiveness of prevention interventions at the country level are lacking. Marconi acknowledged that development of impact measures is still needed and in progress across areas such as qual- ity of care, successful service treatment, treatment within different care settings, and effectiveness of prevention interventions. She also noted that there is still debate and lack of consensus on what appropriate impact mea- sures should be in some areas. For example, she noted, years of life saved

32 EVALUATING THE IMPACT OF PEPFAR is accepted now as a treatment indicator, but no consensus yet exists on whether mortality should be used as a treatment indicator. Public Health Evaluation by the U.S. Global AIDS Coordinator Although evaluations have been conducted routinely over the life of PEPFAR, as the program undergoes a transition from an emergency re- sponse to a sustainable strategy—and a transition from a country-level program to a global program—an expanded and broadened evaluation ap- proach is now required. Shannon Hader, senior scientific advisor of OGAC, outlined PEPFAR efforts to develop public health evaluation (PHE), an approach that can be used to aggregate results across multiple countries, multiple time points, and multiple settings. In contrast to the targeted evaluations (TEs) used in the first phase of PEPFAR, which focused on im- mediate results for rapid project intervention and for individuals receiving services, PHE aims to improve services for communities and populations as a whole. The development of PHE will help to determine the effectiveness of interventions at the community and population levels, scaled-up services at the national level, and expansion of services and coverage to different types of populations, including difficult-to-reach populations. Ultimately, PHE is designed to support evaluation in order to strengthen scientifically sound and cost-effective methods of programming. Structures and Administration of Public Health Evaluation A new structure in OGAC has been established to support the quality, consistency, and coordination of PHEs and use of the results, method- ologies, and tools generated. This structure includes a formalized, annual PHE priority-setting process to identify the most important questions for advancing PEPFAR impact. With oversight by a PHE subcommittee, PHE teams—drawn from PEPFAR headquarters, field offices, and partners— provide technical assistance for developing projects, establish common protocols (that is, guidelines for studies involving human subjects), and coordinate projects across countries. Figure 1-2 shows the PHE organiza- tional structure. PHE identifies priority issues for study not addressed by current evalu- ation projects; for example, the limited number of studies on behavioral outcomes led PHE to prioritize sexual transmission as one of the first evaluation projects. Other pilot PHE teams have been established to inves- tigate care, treatment, and mother-to-child transmission of HIV (MTCT), and future teams are planned to focus on food and nutrition, orphans and vulnerable children, human capacity development, and counseling and testing. PHE teams emphasize an approach based on robust methodologies

INTRODUCTION TO IMPACT EVALUATION 33 Scientific Steering Committee Emergency Plan Community Ensures evaluations are timely, effective, and help to • Technical Working Groups shape policy and program decisions • USG Country Teams OGAC Evaluation • Implementing Partners Support • Local Investigators Public Health Evaluation Subcommittee Oversees the implementation of the evaluation agenda • USG Departments & Agency Headquarters • New Collaborators Evaluation Team (by topic) Evaluation Team (by topic) Evaluation Team (by topic) HQ HQ HQ Field Field Field Partner Partner Partner Investigator Investigator Investigator Evaluation Evaluation Evaluation Evaluation Evaluation sites sites sites sites sites FIGURE 1-2 Structure of PHE. SOURCE: Hader, 2007. FIGURE 1-2 reviewed by scientific experts, the participation and capacity building of local investigators at multiple levels, and a clear plan of analysis. Involving implementers in evaluation design, implementation, and sharing of results is intended to contribute to ensuring independence and transparency of the evaluation. Products of PHE Although PHE does not extend to basic or investigational clinical research activities, it will result in the production of studies of program activities, characteristics, outcomes, and impact, which can in turn be used to determine program effectiveness, compare program models, and answer operational questions for implementation. In FY 2007, PHE conducted a combination of country-funded studies (110) and centrally funded (18) studies. The studies were characterized by many common areas of evalua- tion, including treatment, TB/HIV co-implementation, and MTCT. A key strength of PHE is its ability to aggregate data and therefore maximize investments across countries, in contrast to TEs, many of which have been limited in terms of expert technical assistance; partner experience in methodology; comfort with sampling, statistical, and analytic plans; and access to tools. PHE can be useful in providing technical support and in connecting groups doing similar studies (that is, on drug adherence or infant-feeding interventions) so that outcomes can be measured in similar ways for purposes of cross-country comparison.

34 EVALUATING THE IMPACT OF PEPFAR PHE is also developing tools to ensure quality and consistency of data, enhance the capability to aggregate results across different countries and different settings, and set priorities using a more open and systematic process. Evaluative Approach and Major Findings of the IOM PEPFAR Evaluation Committee Speaker Jaime Sepúlveda of the University of California–San Francisco provided background on the work of the IOM expert committee appointed by Congress to conduct an evaluation of PEPFAR implementation. The committee, which Sepúlveda chaired, began work on the project early in its implementation because the evaluation of PEPFAR was mandated to be delivered 3 years after the legislation was passed. Because of the time frame, it was only possible to evaluate the first phase of the implementation. Sepúlveda summarized the major conclusions and recommendations from the committee’s report, PEPFAR Implementation: Progress and Promise (IOM, 2007) (see Box 1-2). Sepúlveda also offered the committee’s perspec- tive on the design of impact measures for future evaluation of PEPFAR. BOX 1-2 Main Recommendations from IOM Evaluation of PEPFAR Address long-term factors • Emphasize prevention • Empower women • Build workforce capacity • Expand knowledge base Improve harmonization • Improve coordination • Support the World Health Organization prequalification process • Remove budget allocations Expand, improve, integrate services • Data-driven prevention • Adequate medications for treatment • Community-based, family-centered care • Target for orphans and vulnerable children • Attention to marginalized populations

INTRODUCTION TO IMPACT EVALUATION 35 Transition to Sustainability An overarching recommendation of the IOM committee was a needed shift in PEPFAR from an emergency relief mode to a greater emphasis on capacity building for sustainability. The need for continuity, improvement, and flexibility of programming are common themes of the IOM report. The committee underscored the importance of long-term factors, such as expanding and improving prevention interventions, empowering women and girls, strengthening the capacity of the workforce, and expanding the knowledge base through conduct and sharing of research. The committee also made specific recommendations to increase the program’s flexibility and harmonization with other actors by supporting the World Health Organization (WHO) drug prequalification process and removing specific budget allocations for prevention, treatment, and care. Expansion, Improvement, and Integration of Services The committee recommended the expansion, improvement, and integra- tion of prevention, treatment, and care services, with emphasis on evidence- driven prevention interventions, an adequate supply of medications for treatment, and care based on a family-centered, community-based model. The committee emphasized the importance of evidence-based programming and robust and ongoing program evaluation. The committee highlighted the importance of addressing the needs of marginalized populations. Design Considerations for Measuring Impact Sepúlveda next described the committee’s perspective on how PEPFAR’s impact should be measured in the future. The committee urged PEPFAR to participate in joint attribution of outputs, outcomes, and impact with other actors in HIV/AIDS. The committee also recommended the development of both AIDS-specific and more general indicators.   The Prequalification Project, set up in 2001, is a service provided by WHO to facilitate access to medicines that meet unified standards of quality, safety, and efficacy for HIV/AIDS, malaria, and tuberculosis. Any manufacturer wishing its medicines to be included in the prequalified products list is invited to apply. Each manufacturer must present extensive information on the product (or products) submitted to allow qualified assessment teams to evaluate the product’s quality, safety, and efficacy. (See http://www.who.int/mediacentre/ factsheets/fs278/en/index.html.) PEPFAR funding can be used only to purchase Food and Drug Administration–approved drugs.

36 EVALUATING THE IMPACT OF PEPFAR AIDS-Specific Impact Indicators AIDS-specific indicators should be developed that track change of the following “three generations” of HIV/AIDS surveillance: measurement of prevalence and incidence of HIV infection using specific state-of-the-art methods; measurement of behavioral change, including risky behaviors and risk-reducing behaviors; and assessment of stigma and discrimination. Other indicators to be developed include measures of survival, quality of life, development of drug resistance, and the overall physical, mental, and social well-being of people affected by HIV/AIDS. General Impact Indicators PEPFAR evaluation should also develop more general indicators, such as the empowerment of women and girls, general health (that is, infant mortality and overall mortality), capacity of community-based organiza- tions to respond, and public health infrastructure and capacity (that is, sup- ply chain and health care workforce). Among those indicators developed to track change in public health capacity, measures for monitoring the health care workforce are particularly important in light of the depletion of that workforce by the disease itself, the flux of health care workers to developed countries, and the sequestration of the health care workforce to vertical health programs. In addition, PEPFAR should also develop measures of the degree to which HIV/AIDS interventions are used to drive both desired improvements in the health system and incremental incorporation of other health priorities into national agendas. PEPFAR should contribute to the development of the knowledge base for how best to implement, scale up, and sustain prevention, care, and treatment services.

Next: 2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting »
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary Get This Book
×
Buy Paperback | $50.00 Buy Ebook | $39.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Design Considerations for Evaluating the Impact of PEPFAR is the summary of a 2-day workshop on methodological, policy, and practical design considerations for a future evaluation of human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) interventions carried out under the President's Emergency Plan for AIDS Relief (PEPFAR), which was convened by the Institute of Medicine (IOM) on April 30 and May 1, 2007. Participants at the workshop included staff of the U.S. Congress; PEPFAR officials and implementers; major multilateral organizations such as The Global Fund to Fight AIDS, Malaria, and Tuberculosis (The Global Fund), the Joint United Nations Programme on HIV/AIDS (UNAIDS), and the World Bank; representatives from international nongovernmental organizations; experienced evaluation experts; and representatives of partner countries, particularly the PEPFAR focus countries. The workshop represented a final element of the work of the congressionally mandated IOM Committee for the Evaluation of PEPFAR Implementation, which published a report of its findings in 2007 evaluating the first 2 years of implementation, but could not address longer term impact evaluation questions.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!