The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Overview
A 2-day workshop on methodological, policy, and practical design considerations for a future evaluation of human immunodeficiency virus/ acquired immunodeficiency syndrome (HIV/AIDS) interventions carried out under the President’s Emergency Plan for AIDS Relief (PEPFAR) was convened by the Institute of Medicine (IOM) on April 30 and May 1, 2007. Participants at the workshop included staff of the U.S. Congress; PEPFAR officials and implementers; major multilateral organizations such as The Global Fund to Fight AIDS, Malaria, and Tuberculosis (The Global Fund), the Joint United Nations Programme on HIV/AIDS (UNAIDS), and the World Bank; evaluation experts experienced with similar kinds of evaluations; and representatives of partner countries, particularly the PEPFAR focus countries. The workshop represented a final element of the work of the congressionally mandated IOM Committee for the Evaluation of PEPFAR Implementation, which published a report of its findings in 2007 (IOM, 2007) evaluating the first 2 years of implementation, but could not address longer term impact evaluation questions.
This overview describes core messages from the workshop’s presentations and discussions. First, background is provided on the definition and uses of impact evaluation, the process of internal evaluation within the PEPFAR program, and the recommendations concerning the design of
The workshop summary has been prepared by the workshop rapporteurs as a factual summary of what occurred at the workshop.
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole. Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Overview
A 2-day workshop on methodological, policy, and practical design considerations for a future evaluation of human immunodeficiency virus/ acquired immunodeficiency syndrome (HIV/AIDS) interventions carried out under the President’s Emergency Plan for AIDS Relief (PEPFAR) was convened by the Institute of Medicine (IOM) on April 30 and May 1, 2007. Participants at the workshop included staff of the U.S. Congress; PEPFAR officials and implementers; major multilateral organizations such as The Global Fund to Fight AIDS, Malaria, and Tuberculosis (The Global Fund), the Joint United Nations Programme on HIV/AIDS (UNAIDS), and the World Bank; evaluation experts experienced with similar kinds of evaluations; and representatives of partner countries, particularly the PEPFAR focus countries. The workshop represented a final element of the work of the congressionally mandated IOM Committee for the Evaluation of PEPFAR Implementation, which published a report of its findings in 2007 (IOM, 2007) evaluating the first 2 years of implementation, but could not address longer term impact evaluation questions.
This overview describes core messages from the workshop’s presentations and discussions. First, background is provided on the definition and uses of impact evaluation, the process of internal evaluation within the PEPFAR program, and the recommendations concerning the design of
The workshop summary has been prepared by the workshop rapporteurs as a factual summary of what occurred at the workshop.
OCR for page 2
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
future impact evaluations by the IOM committee that evaluated PEPFAR implementation. Next, a vision is described for the types of questions workshop participants would like to see addressed in future impact evaluations, along with suggestions for how the process of impact evaluation would ideally be carried out. The final section addresses methodological issues that were raised by participants as being important to consider in the design of future impact evaluations.
DEFINING AND USING IMPACT EVALUATION
Meeting participants proposed a working definition of impact evaluation as a measurement of net change in outcomes attributable to a specific program using a methodology that is robust, available, feasible, and appropriate both to the question under investigation and to the specific context. Workshop participants noted that impact evaluation is not only about outcomes, but also the process that leads to outcomes; that is, it includes both means and ends. Participants argued for a definition of impact evaluation that is longer term, more broadly defined, and less linear. Although a more traditional definition of infectious disease impact evaluation (that is, one that is limited to metrics such as prevalence, incidence, infections averted, morbidity, and mortality) is important, the broader and deeper impact evaluation envisioned would also include measurement of changes in health status, systems capacity, quality of services, economic development, and social, economic, and political outcomes.
Two major uses of evaluation were described: (1) use of evaluation for judging the performance of the program for purposes of accountability (summative evaluation) and (2) use of evaluation for informing the improved decision making within a program (formative, or utilization-focused, evaluation). Formative evaluation of PEPFAR is important to inform both congressional decision making and programmatic decision making in partner countries, although decision makers at different levels may have different evaluation needs, participants said. For each evaluation question, participants noted, it is important to clarify who needs the information, what information is needed, and when.
INTERNAL EVALUATION OF PEPFAR
According to PEPFAR officials, a key aspect of PEPFAR’s monitoring process and infrastructure is to track progress toward the program’s goals in prevention, treatment, and care. PEPFAR internal monitoring supports the principles of local leadership and ownership of the HIV/AIDS response by building local capacity, using local infrastructure, implementing the program according to national guidelines, monitoring using locally developed
OCR for page 3
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
indicators, and funding programs based on results, PEPFAR officials noted. Over time, PEPFAR’s evaluation activities have expanded field operations relative to central operations and have expanded reporting infrastructure and trained personnel in-country.
Initially, reported PEPFAR officials, the conceptual framework for PEPFAR impact evaluation was much more narrowly defined, and country capacity for monitoring was limited, with a lack of consistent targets across countries and a lack of hard data. Evaluation efforts and reporting requirements were loosely coordinated among U.S. implementing agencies. Over the first 5 years of PEPFAR, evaluation planners developed a monitoring and evaluation system that relied on survey data and periodic targeted evaluations on selected topics. Although this approach provided results on the deployment and use of funds, the development and delivery of services, and the beneficiaries of program services, as the complexity of PEPFAR program strategies grows and as the program undergoes a transition from an emergency response to a sustained response, the definition of impact and approaches used to measure impact will need to be broadened. PEPFAR has recently developed a new, centrally managed public health evaluation (PHE) structure that can be used to aggregate results across multiple countries, multiple time points, and multiple settings. The PHE approach is designed to support evaluation by helping to set priorities, provide technical assistance, establish common protocols, and coordinate projects across countries.
FUTURE EVALUATION DESIGN: PERSPECTIVE FROM THE IOM COMMITTEE
Speaker Jaime Sepúlveda, chair of the IOM committee that authored the report, PEPFAR Implementation: Progress and Promise (IOM, 2007), reviewed the recommendations of the committee on the design of impact measures—both AIDS-specific and more general indicators—for future evaluation of PEPFAR. Future evaluation should include measurement of what Sepúlveda referred to as the “three generations” of HIV/AIDS surveillance: prevalence and incidence of HIV infection, measurement of behavioral change, and measurement of stigma and discrimination. Other important indicators include measures of survival, quality of life, development of drug resistance, and the overall physical, mental, and social well-being of those people affected by HIV/AIDS. PEPFAR evaluation should also develop more general indicators, such as the empowerment of women and girls, overall health status, capacity of community-based organizations to respond, and public health infrastructure and capacity.
OCR for page 4
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
USE OF EVALUATION TO FOCUS ON MEANINGFUL ENDPOINTS
There was substantive focus during the workshop on what participants wanted to accomplish through PEPFAR. Although the workshop drew participants from wide-ranging and diverse perspectives—representatives of the U.S. government, evaluation methodological experts, representatives of partner countries, and global partners—most participants converged on the same kinds of endpoints. Participants emphasized a need to move beyond counting numbers of people who are “touched” by the program and to instead focus on a set of more meaningful and more strategic questions about what interventions succeed. Participants cautioned that an emphasis on counting can be misleading, can drive inflated reporting, and can jeopardize sustainability of a program.
Most participants called for a broader interpretation of impact that includes measurement of not only AIDS-specific impacts, but also more general impacts; measurement of not only results of implementation, but also the process of implementation; and measurement not only of the overall benefits, but also the distribution of benefits. Workshop participants identified questions for evaluating impact that can be clustered into the following nine broad categories: cost-effectiveness, logic of conceptual approach, health impacts, impacts beyond health, capacity building and health systems strengthening, coordination and harmonization, sustainability, equity and fairness, and unintended impacts. These questions are summarized in Box O-1.
USE OF EVALUATION TO FOCUS ON COLLECTIVE OUTCOMES
The meeting set a tone for the ideal conduct of evaluation—emphasizing collaboration, consultation, harmonization with the host countries, and coordination among global partners. Several workshop participants observed that the principles of coordination and harmonization need to be reflected in both the evaluation effort and in the overall implementation of PEPFAR. Workshop participants noted that many different actors—PEPFAR, The Global Fund, UNAIDS, the World Bank and others—are all working in HIV/AIDS response in many of the same countries and can learn from one another. Given the costs of evaluation, setting up evaluation in a collaborative way can take advantage of this synergy and save resources. Many participants observed that although exclusive attribution of program successes to specific funders may not be realistic or constructive, coordinated evaluation may be able to better illuminate what types of interventions are the most effective. Workshop participants acknowledged that there have been earnest efforts to improve coordination and harmonization over the life of PEPFAR, and there is increased agreement to focus evaluation on collective outcomes.
OCR for page 5
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
BOX O-1
Summary of Impact Evaluation Questions as Identified by Workshop Participants
Cost-effectiveness
Approaches, strategies, and interventions, such as prevention services and treatment options
Conceptual Approach
Countries targeted
Populations targeted
Budget allocations for types of interventions
Management and financing
Health Impacts
HIV/AIDS-specific health impacts
Prevalence, incidence, morbidity, mortality, longevity
Prevention of HIV transmission
Quality of life
Behavioral change
Stigma and discrimination
Other health impacts, disaggregated by population
Overall mortality, lives saved, survival
Child mortality
Fertility, unintended and intended pregnancy
Impacts Beyond Health
Gender equality
Effectiveness of PEPFAR in addressing underlying causes of women’s vulnerability
Effectiveness in building men’s and women’s analytical skills and competencies
Effectiveness of messages for behavioral change
Effectiveness of interventions to reduce the spread of HIV infection to women and girls
Effectiveness of the “packaging” of gender interventions
Child welfare
Effectiveness in improving parenting skills
Health, nutritional, and educational status of orphans and vulnerable children
OCR for page 6
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Security, development, productivity, and poverty alleviation
National peace and security in a nation
Poverty alleviation and economic growth
National development
Institutional and societal changes
Policy changes such as property rights, inheritance laws, and human rights; political will; community ownership; food and water security; engagement of vulnerable populations; destigmatization; and discrimination measures
National priorities and political views
Impacts on Capacity Building and Health Systems Strengthening
Health care workforce
Effects of PEPFAR on workforce shifts
Impact of PEPFAR’s training approaches
Impact of PEPFAR’s programs to support health care workers
Effectiveness in sustaining local workforce development systems
Effectiveness of institution-building efforts
Institution building of community-based organizations
Capacity building of nontraditional institutions
Infrastructure
Effectiveness of strengthening supply chain management and drug delivery systems
Quality of care and service delivery
Prevention, care, treatment, support, and mitigation; antiretroviral (ARV) drug retention rates; levels of client satisfaction; appropriateness of referrals; community attitudes toward people living with HIV/AIDS; and rational prescription behavior
Development of a knowledge base of what interventions work
National-level health agenda
Integration with other health issues; change in national-level health agendas
Coordination and Harmonization
Coordination among U.S. government implementing agencies
Consistency of targets among implementing agencies
Positive and negative impacts of complementary interventions, or “wraparound” programs
OCR for page 7
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Harmonization and alignment with partner countries
Openness and accessibility of plans and projects at the community level
Presence of instruments and structures for joint decision making
Degree of information sharing and joint implementation among partners
Existence of mechanisms to make coordination more flexible
Coordination among program implementers
Development and effectiveness of a variety of coordination tools or mechanisms for fostering the exchange of learning
Level of harmonization of drug procurement systems
Sustainability Impacts
Degree to which additional resources have been leveraged from other donors
Extent to which long-term learning and research have been promoted
Measures of capacity building and sustained contributions to institutions and systems
Degree to which local implementation, ownership, and coordination have been promoted
Equity and Fairness Impacts
Existence and effectiveness of processes for goal setting and implementation
Fairness impacts, disaggregated by group, of program integration within the health system
Existence and effectiveness of compensatory mechanisms to improve fairness
Positive and Negative Unintended Impacts
Impacts of earmarking on program integration
Diversion of resources from neglected health care areas and the broader health care system
Impact of PEPFAR on corruption
Impact of PEPFAR on access to services
Impact of treatment on adverse and high-risk behavior
Impact of nutritional programs
Impact of PEPFAR programs on reproductive health and family planning
Impact of counseling and testing on pregnancy care
Impact of ARV treatment on fertility and orphanhood
OCR for page 8
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Coordination and harmonization of the evaluation process can provide the benefits of mutually influencing others’ work, minimizing transaction costs, ensuring more efficient use of funds, and bringing to bear the strengths and perspectives of key stakeholders. Partner countries, implementing partners, and beneficiaries are among the perspectives that are critical to evaluation design, workshop participants said. Partner countries add value because they are accountable to their citizens and have experience in dealing with the challenges of service delivery. Implementing partners offer a familiarity with program data and lessons, along with a keen understanding of the challenges of delivering services. Local people add value to the evaluation process because of their deep contextual knowledge and expertise about program impact at the community level.
Although the process of coordination can be very time-consuming, there is great potential for coordination and harmonization to improve the sharing of data, approaches, and evaluation research among partners. Workshop participants highlighted key opportunities to share the outcomes of PEPFAR’s evaluation efforts with the 5-year evaluation of The Global Fund, a planned evaluation by UNAIDS, and evaluative efforts by the World Bank, the Organisation for Economic Co-operation and Development, and the World Health Organization. Participants also noted that opportunities to harmonize the design, conduct, and interpretation of evaluation results with country-level partners include the conduct of joint field evaluations by collaborating partners; the development of centralized funding, knowledge management, and data aggregation systems; and discussions leading to consensus among partners on monitoring approaches and overall program objectives.
USE OF EVALUATION TO BUILD LOCAL CAPACITY
The evaluation effort is an element of the larger effort to build capacity in partner countries so that they have, going forward, a lasting capacity to respond to their own epidemics, workshop participants said. Many participants reflected on the fact that evaluators often rely on their own data collection and evaluative capacities instead of helping countries to develop their own. Impact evaluation is most constructive when it is done in a way that builds countries’ capacities to collect, analyze, and use program information, thereby strengthening the sustainability of programs.
Participants noted, however, that multiple constraints stand in the way of building local evaluative capacity, including weak or absent country-level systems for gathering data, funding mechanisms that limit the prioritization of monitoring and evaluation, competition for resources between evaluation and implementation, poor engagement of stakeholders in defining the
OCR for page 9
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
monitoring agenda, and a tendency of researchers to drive a more narrow and less useful evaluation agenda.
Opportunities for strengthening local capacity to conduct impact evaluation include the dissemination of methodologies, provision of technical assistance, recruitment and training of personnel in evaluation methods, and promotion of country-driven priority-setting processes for evaluation.
DESIGNING AN EVALUATION BASED ON ROBUST METHODOLOGIES
Workshop participants stressed the importance of designing impact evaluation on the basis of a logical conceptual approach and robust methodologies. Case studies of evaluations of HIV/AIDS interventions were presented, and challenges and opportunities in evaluating impact were discussed.
A number of general principles or observations by participants emerged from the discussions:
Prioritization is needed to narrow down what needs to be measured. For long-term evaluations, for example, only those issues common to all projects might be selected. For a large portfolio of activities, a more narrowly defined set of indicators might be selected. Information about who needs the evaluative information and when can inform the prioritization process.
Formative or “learning” evaluation is a common component of many of the evaluations discussed, that is, ongoing evaluation to improve programming and to inform decision making, as opposed to evaluation at the end of a program to judge the success or failure. Negative evaluation results also offer value for learning and thus should be as widely disseminated as positive results.
Multiple methodologies were used in many of the evaluations discussed and can provide richer results than just one or two methods. Many evaluations used both qualitative and quantitative methodologies. These included case studies, working papers, interviews, models, literature reviews, surveys, field work, participatory approaches, and theory. Multiple methodologies can be more valuable if selected strategically to complement each other.
Randomization is a powerful technique that can add value and credibility to evaluation. Although perceptions persist that randomization is difficult to implement and impractical at the country level, new methods are available to more easily incorporate randomization into a study.
Consultation and communication are an important part of the evaluative process. Changes resulting from the evaluation may have more
OCR for page 10
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
to do with the communications and consultations used during the process than with the actual results of the evaluation. Consultation is an important part of understanding what decision makers want from evaluation.
Limitations of data and models need to be well understood. The data collection systems in many partner countries are weak, and empirical data are often inadequate or inaccurate. Similarly, models have limitations that need to be understood. Models need to be validated with empirical data and made more accurate through the addition of variables.
Early design of the evaluation is critical to ensuring that the design is appropriate and that impacts can be detected early. Early design is especially needed to facilitate the use of randomized approaches.
Comparison across contexts is an important attribute of evaluations, but change may be highly contextual. Workshop participants noted that success in one context may not necessarily be transferable to another. Factors independent of program interventions may have a significant influence on change.
Methodological Challenges and Opportunities in Evaluation
Challenges and Opportunities in Measuring HIV/AIDS-Specific and General Impacts
Workshop participants explored the limitations of commonly used methods in evaluation, as well as new prospects, in measuring both HIV/AIDS-specific and more general impacts. These are summarized in Table O-1.
Challenges and Opportunities in Attributing Impact
Attribution—relating the impact of a particular investment to a particular donor—is one of the greatest methodological challenges in impact evaluation, workshop participants said. It is extremely difficult to tease out the exclusive impacts of efforts of any one donor from those of others given the number and diversity of programs and funders working in the area of HIV/ AIDS. Evaluating donor-specific attribution, however, may not be constructive. Several participants said that it is perhaps more useful to determine, using evidence-based approaches, what interventions are most effective and then to judge donors by whether they invest in those approaches.
Challenges and Opportunities in Aggregating Evaluation Results
The statistical synthesis or aggregation of the results of multiple studies is a methodological frontier, workshop participants said. Meta-analysis,
OCR for page 11
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
which combines results from multiple studies as if they were a single large study, is a tool that is currently underdeveloped for application to impact evaluation. Multiple analyses also have value in the independent validation of results.
OCR for page 12
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
TABLE O-1 Challenges and Opportunities in Measuring HIV/AIDS-Specific and General Impacts
HIV/AIDS-Specific Impacts
Metric
Definition
HIV prevalence
The proportion of individuals within a population infected by HIV. Prevalence is a function of both the death rate of those infected and the rate at which new infections occur.
HIV incidence
The number of new cases of HIV within a population at risk over a given period of time.
Infections averted
The difference between expected and actual annual incidence.
OCR for page 13
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Challenges
Opportunities
Measurement through testing of pregnant women at antenatal clinics (ANCs) tends to overestimate prevalence because ANCs are urban.
New tools that overcome some of the limitations of HIV prevalence measurement through ANC surveillance have been developed.
A second population-based survey of HIV testing will soon become available, allowing further analysis of prevalence.
Respondent-driven sampling methods are being developed for examining prevalence in high-risk groups.
Longitudinal cohort studies may not reflect the true incidence in the population, and many participants in cohort studies may be lost to follow-up.
Laboratory assays used to distinguish recent infections from long-term infections tend to overestimate the proportion of most recent infections.
Modeling tools are limited in their ability to accurately measure population-level risk.
Empirical data on incidence by age and sex are lacking.
An adjustment formula for the HIV incidence laboratory assay has been developed.
A more specific laboratory assay will improve the ability to distinguish long-term and recent infections.
A new population-based survey will provide important age-, sex-, and geography-specific incidence information.
Modeling from prevalence data and accounting for survival of infected individuals can be used to calculate incidence.
Infections averted are a “nonevent,” and their measurement requires multiple assumptions.
Population projection modeling has limitations because of the gaps in data available in developing countries.
Models may not account for epidemiological contextual factors.
A new model called Spectrum takes epidemiological contextual factors into account.
A Futures Group model can be used to attribute infections averted to specific interventions.
Serial HIV population surveys can help to deduce changing incidence and infections over time.
Cross-country comparative analyses of HIV dynamics and intervention uptake can be used to measure the relative effectiveness of interventions in averting infections.
OCR for page 14
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Metric
Definition
Survival and mortality rates
Mortality rate: the ratio of deaths in an area compared to the population of that area per unit of time.
Survival rate: the percentage of people in a study or treatment group who are alive for a given period of time after diagnosis or treatment.
Behavioral change
Modification of sexual, injection, and drug-adherence practices.
OCR for page 15
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Challenges
Opportunities
Measuring overall mortality change in response to treatment is challenging because increased survival of HIV-infected individuals and increased opportunities for viral transmission to others decrease and increase mortality, respectively.
Mortality data are poor because of weak and inaccurate mortality surveillance systems and loss of patients to follow-up. Cause-specific and cohort-specific survival data are lacking.
Models frequently use only vital registration, population-level data.
Mortality rate data quality can be improved by aggressively pursuing information on those patients lost to follow-up and by standardizing methods for collecting information on deaths from hospital records.
Use of “verbal autopsies” can be used to follow up on deaths in households and determine the cause of death.
Corporate-sector surveillance systems can provide early indicators of the impact of treatment programs on mortality.
Age-specific and population-based mortality data can be gathered for improved measurement of mortality impact.
Data and surveillance gaps exist, particularly in behavioral surveillance with biomarkers.
Surveys and simulation models do not illuminate why specific populations are affected differently.
Current methods might not be able to determine the extent and coverage of behavioral change. Incomplete behavioral change can have worse consequences than no behavioral change.
Factors independent of the program intervention may influence behavioral change.
Change at the social and institutional levels to build and sustain infrastructure for risk reduction can be tracked.
Opinions of leaders, impediments to behavioral change, and unintended negative consequences of behavioral change can be measured.
Behavior surveys, particularly those targeting younger people, who are an early indicator of prevalence changes, can be useful for attributing changes in HIV incidence to specific changes in risk.
Models combining trends in prevalence and incidence with studies of risk behavior can be a useful tool for retrospectively understanding how interventions might have worked to maximize declines in HIV prevalence.
OCR for page 16
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Metric
Definition
Stigma and discrimination
Negative attitudes, beliefs, and actions toward people who are perceived to have HIV/AIDS and those associated with them.
Orphanhood prevention
Prevention of the death of one or usually both parents of a child.
OCR for page 17
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Challenges
Opportunities
Rigorous research and data collection approaches are absent. Most of the literature is based on anecdotal evidence, testimonials, and a few qualitative studies.
Absence of scales to measure stigma and its effects and tools to measure the effectiveness of strategies for mitigating stigma.
An International Planned Parenthood Federation stigma index is now available. Another instrument, reflecting 33 factors measuring people’s perceptions, has also proved reliable for measuring stigma.
New sources of data from focus groups have been useful in assessing stigma.
Measurements do not distinguish between children who have lost one parent (single orphans) and children who have lost both parents (double orphans) to HIV.
Treatment has an unclear impact on orphanhood. Treatment of HIV-positive orphans extends years of orphanhood, and while treating HIV-positive parents can reduce orphanhood years of existing children by prolonging parents’ lives, it can also generate years of orphanhood among children who are born to HIV-positive parents during treatment.
Methods do not exist for conducting cost-effectiveness analysis on interventions to prevent orphanhood.
New models are under development to better quantify the impact of treatment and prevention in preventing the orphaning of children.
Useful indicators, such as “years of orphanhood averted” and “number of children who reach age 18 before the death of a parent whose life is extended by antiretroviral therapy (ART),” have been developed.
OCR for page 18
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Metric
Definition
Development of drug resistance
The evolved capability of HIV to withstand a drug to which it was previously sensitive.
General Impacts
Metric
Definition
Health systems strengthening
Improvement of a broad range of factors related to health care service delivery, including accessibility, quality, efficiency, and equity of services; management; procurement and distribution systems; human resource use; policy environment; and infrastructure.
Health care workforce strengthening
Improvement of a range of capacities related to health care personnel, including training, supervision, and job satisfaction.
OCR for page 19
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Challenges
Opportunities
The threshold survey can be used to assess transmitted HIV infection using blood tested at ANC sentinel surveillance sites. Blood sampled from young women (age 25 or younger) in their first pregnancies who are likely not to be in ARV treatment can be used to track the transmission of drug-resistant HIV strains.
Therapy monitoring can be used to measure drug resistance by sampling and monitoring patients in ARV treatment from the initiation of therapy over a 1-year time period. Indicators of drug resistance such as outcome, viral load, and drug adherence can be monitored.
Challenges
Opportunities
Effects on the health system can be positive or negative, intended or unintended.
Health systems represent a diverse set of institutions that may or may not be easily compared.
Health systems include a diverse range of elements.
Empirical estimates of impacts are lacking.
The small sample size and short time interval over which change is often evaluated limit many studies.
Impact attribution is difficult in this type of analysis.
Facility surveys, provider surveys, and qualitative interviews can be used to measure a variety of attributes of the health system.
The quantity of non-HIV health services delivered before and after the introduction of basic HIV care can be compared, using regression analysis to control for independent effects.
A range of new indicators could be developed to measure impacts of interventions on capacity development, training and supervising effectiveness, gender equality, competencies, etc.
A system could be created to track health care workers over time, from registration to retirement.
Methods could be developed to evaluate the degree to which interventions strengthen institutions that regulate the workforce (that is, accrediting associations).
OCR for page 20
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Metric
Definition
Effectiveness of complementary interventions
Effectiveness of programs complementary to more narrowly focused HIV services, including interventions in areas such as malaria, tuberculosis, nutrition education, food security, social security, education, child survival, family planning, reproductive health, medical training, health systems, and potable water.
Effectiveness of gender-focused activities
Improvements in gender equality and women’s empowerment.
Effectiveness of coordination and harmonization
Increased alignment of HIV/AIDS interventions with country-level plans and coordination of efforts among other implementing partners.
Effectiveness of community- or population-level service delivery
Improved service delivery for specific populations, that is, children, families, communities, HIV-infected groups, high-risk groups, etc.
OCR for page 21
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
Challenges
Opportunities
Prospective randomized evaluation can be used to compare later program enrollees to earlier program enrollees by monitoring a range of indicators (that is, education and health indicators).
Given the multidimensional, open, complex, nonlinear, and adaptive nature of gender, it is difficult to define what constitutes success.
Few outcome evaluations and few tools have been developed on how gender-focused activities affect HIV risk, and few good indicators exist that are useful in understanding social dynamics.
Evaluations of gender activities tend to underrepresent the perspectives of local people.
The Gender Equitable Men’s (GEM) scale can be used to look at gender norm attitudes and how they change over time. The scale is an index of 24 items, including home and child care, sexual relationships, health and disease prevention, violence, homophobia, and relations with other men.
The Country Harmonization and Alignment Tool (CHAT) can be applied to the standardization of approaches for alignment of interventions with country-level plans and coordination of efforts among partners.
Community-level program information reporting systems (CLPIR) have been developed to examine community-level service delivery and help answer questions such as when, how, and where people want testing and treatment.
OCR for page 22
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary
This page intentionally left blank.
