Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 17
PART II
Proposed Evaluation Approach
OCR for page 18
OCR for page 19
CONGRESSIONAL CHARGE AND PLANNING PHASE APPROACH
Under the Lantos–Hyde Act of 2008, 9 Congress mandated that the IOM conduct a study
that includes an assessment of the performance of U.S.-assisted global HIV/AIDS programs and
an evaluation of the impact on health of prevention, treatment, and care efforts that are supported
by U.S. funding, including multilateral and bilateral programs involving joint operations (see
Appendix A). Based on clarifications with congressional staff and OGAC, 10 the charge is
intended to focus on the performance and impact of bilaterally funded PEPFAR programs in the
current partner countries (see Table 1 in Part III, Section 2 for a list of countries). This will
include programs and activities that are operated jointly with both bilateral funding through
PEPFAR and funding through the Global Fund. Consistent with the clarified congressional
intent, U.S. contributions to the Global Fund that are not a part of activities jointly funded or
implemented by PEPFAR will not be the focus of the evaluation, and the evaluation will not
compare the performance of bilateral PEPFAR programs to that of Global Fund programs
(Bressler, 2009; Marsh, 2009). The study will consider PEPFAR’s performance and impact since
funding first became available in 2004. The timing of the study, with a final report to be
delivered in 2012, dictates that the evaluation will consider data that are now, or will become,
available through 2011.
As the first phase of this study, the IOM was charged to form an ad hoc committee to
develop a strategic plan for the assessment and evaluation of HIV/AIDS programs implemented
under the Lantos–Hyde Act of 2008 and to issue a short report to the U.S. Congress describing
the plan’s proposed design, taking into consideration the requirements for the congressionally
mandated study. These requirements and the charge for developing the evaluation plan can be
found in full in the Statement of Task in Appendix A. More information about the members of
the planning committee can be found in Appendix B. This report documenting the proposed
evaluation approach is the product for the first phase of the study.
To produce this report, the planning committee met three times to deliberate in person,
conducted two teleconferences, and engaged in additional deliberations in smaller working
groups by telephonic and electronic communications as needed. In the development of its plan,
the committee consulted widely, and remains open to receiving input from, the broad range of
parties interested in and affected by PEPFAR. To solicit input and gather information from a
wide range of stakeholders, public sessions were held in conjunction with the first and second
committee meetings, and delegations from the committee and IOM project staff also held
information-gathering meetings with a range of global stakeholders, including UNAIDS, WHO,
the United Nations Children’s Fund (UNICEF), and the Global Fund. The primary purpose of
these meetings was to establish working relationships with these stakeholders and to discuss
potential data sources and methodologies, as well as strategies and lessons learned from large-
scale programmatic or organizational evaluations. The agendas for these activities can be found
in Appendix C. In addition, one staff member attended the PEPFAR Annual Implementers’
Meeting in 2009. The committee also consulted the available literature on PEPFAR, global
HIV/AIDS, and the state of the art in large-scale program evaluation, including the summary of a
9
Supra., note 6 at §101(c), 22 U.S.C. 7611(c).
10
Personal communications from Congressional Staff at the U.S. House Committee on Foreign Affairs and U.S.
Senate Committee on Foreign Relations and OGAC, 2009.
19
OCR for page 20
20 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
workshop convened by the IOM, “Design Considerations for Evaluating the Impact of
PEPFAR,” which focused on methodological, policy, and practical considerations (IOM, 2008).
The committee and staff also conducted an initial scan of potential data sources for the
forthcoming evaluation using a range of sources, including a preliminary review of documents
from OGAC and other bilateral and multilateral agencies and of relevant published literature, as
well as communications with a wide range of staff from OGAC, implementing partners, and
multilateral stakeholders. The committee used this information to assess the methods that could
be employed to answer evaluation questions based on the charge in the statement of task,
focusing on data and methodology that will be robust, available, feasible, and appropriate to the
questions. Through this information gathering and deliberation, the planning committee
developed a conceptual framework for the evaluation that is based on both the committee’s
expertise and current standards in evaluation methodologies for large-scale programs.
This report is intended to provide Congress and OGAC with an overview of the strategic
plan for the forthcoming evaluation. As agreed upon contractually with the study sponsor, the
planning process for the evaluation will culminate with a transitional period for operational
planning that will take place between the delivery of this report and the implementation of the
evaluation itself in the fall of 2010. During this operational planning phase, IOM staff, planning
committee members, and consultants will carry out activities to further develop and refine the
plan described here. These activities, which will inform the implementation of the evaluation, are
described as part of the work plan later in this section. This structure for the study, with a report
describing strategic elements of the plan delivered to Congress before detailed operational
planning is complete, was intentionally designed to allow uninterrupted progress in preparation
for the evaluation during the time necessary for review of the report and budget planning by the
sponsor and for subsequent preparation of the contract for the evaluation.
After congressional review of the plan’s proposed design and budget, the final phase of
the project will be to carry out the assessment/evaluation of the program. The IOM will convene
a new ad-hoc committee to conduct the evaluation as a consensus study. The intent is for the
evaluation committee and staff to have considerable overlap from the planning committee.
Standard IOM procedures will be followed to ensure that the evaluation committee and project
staff have the appropriate expertise to conduct the evaluation activities described in this report.
EVALUATION GOALS AND CONCEPTUAL FRAMEWORK FOR
EVALUATION DESIGN
The legislative mandate to evaluate PEPFAR is a complex challenge. As described
above, PEPFAR is a large, multifaceted program with many activities carried out by many
different partners in a diverse group of countries. In addition, PEPFAR activities are being
implemented in the context of programs supported by other funders that have the same ultimate
aim. PEPFAR is also by necessity a dynamic program; the ability to change the program over
time can be beneficial, but makes evaluation difficult as it presents a “moving target.” Therefore,
this report not only outlines an approach for evaluating the performance of PEPFAR, but also
delineates the challenges in evaluating the impact of such a complex, large-scale foreign
assistance program and provides information about reasonable and appropriate expectations for
an evaluation of this kind. The committee has endeavored to present a plan that is thorough and
well-defined in its approach yet maintains ample flexibility. This will allow the evaluation to be
adapted in response both to the evolving goals of the program and to the additional information
OCR for page 21
21
PROPOSED EVALUATION PLAN
the evaluation committee will gather during operational planning and as the evaluation itself
proceeds.
The proposed conceptual framework for the evaluation and its limitations will be
described briefly here, followed by a more thorough discussion of the methods and data sources
that will be used. The subsequent sections in Part III of this report address specific components
of the evaluation in greater detail.
Evaluation Goals and Assumptions
The planning committee understood the mandate from Congress as a charge to develop a
plan to assess the program with two primary goals. The first of these is an assessment of the
success of the program in meeting the performance goals and targets laid out in two sources: the
reauthorization legislation and the new PEPFAR Five-Year Strategy. Although the statement of
task was written before the new strategic plan was available, the committee interpreted the
charge to take this document into account because it articulates the current guiding principles and
the future direction for the program. Therefore, the evaluation will include a careful review and
comparison of these guiding documents in order to more clearly define the targets and goals of
the program. The second goal of the committee’s charge is to evaluate the health impact of
PEPFAR, including impact of treatment, care, and prevention programs; effects on health
systems; efforts to address gender-specific aspects of HIV/AIDS; impact of programs on child
mortality; and impact of interventions on behalf of orphans and vulnerable children. The findings
and conclusions of the evaluation of PEPFAR’s progress toward its stated goals and the impact
of the program will then be used to make recommendations for improving the USG response to
global HIV/AIDS, in particular through PEPFAR programs.
It is important to note that the IOM is being charged to conduct an evaluation early in the
implementation of changes to the program in response to the reauthorization legislation and the
new PEPFAR Five-Year Strategy. These changes reflect a progressive transition to a new era of
challenges and goals for the program, which include efforts to improve sustainability of the
response over time, to enhance coordination with partner governments and other global funding
partners, and to support accountable ownership of HIV program delivery by countries
themselves. They also reflect efforts to give greater consideration to the relationship of PEPFAR
to broader health and development needs in partner countries. The timing of this evaluation, with
data collection extending through 2011 for a final report due in 2012, will make it difficult to
evaluate the outcomes or impact of these most recent changes so soon after implementation. For
example, it could take several years or even decades for a full effect to be realized from some
efforts to strengthen health systems, such as the training and retention of new health care workers
or the strengthening of health information systems to support M&E efforts. However, the
evaluation will assess efforts, process, and initial results in these areas to provide insight into
whether PEPFAR is making reasonable progress toward these new goals and to lead to
recommendations for how the program can be improved to ensure that these evolving goals for
the program can be met. As part of this, the evaluation will assess whether there is sufficient
M&E capacity in place to eventually evaluate whether the program has met these goals as well as
the resulting outcomes and impact.
The legislative mandate calls for the assessment of PEPFAR to be delivered in 2012; this
would coincide with reauthorization discussions for the program, which the current legislation
extends through 2013. It is of course not possible to predict the future needs and priorities of
OCR for page 22
22 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
Congress and OGAC with complete accuracy, but the planning committee’s goal was to design
an evaluation approach that, to the extent possible, looks ahead to anticipate the evolution of the
program and therefore produce findings that address key issues under consideration at the time of
the report release, including discussions about possible future legislative reauthorization.
Conceptual Framework for Evaluation Design
The planning committee developed an overall conceptual framework for the evaluation,
which calls for the use of a program impact pathway to guide an assessment of the contribution
of PEPFAR to changes in health impact within the context of multiple international and national
funding streams. This program impact pathway, described in more detail below, illustrates the
committee’s understanding of how PEPFAR programs are currently structured and intended to
ultimately translate into health impacts, laying out a plausible pathway for causal effects. It
represents the theory of change that underlies the program—in other words, the rationale for how
the combination of activities supported by PEPFAR are logically expected to produce
intermediate outcomes, which are then expected to collectively contribute, along with programs
funded by other sources, to the desired population health impact. The use of a program impact
pathway, which is also referred to as a logic model or results chain, has become well established
as a method for evaluating complex, large-scale development assistance programs and is
becoming widely accepted as a standard in the global HIV/AIDS community (Leeuw and
Vaessen, 2009; UNAIDS MERG, 2010). 11
Guided by the program impact pathway, the evaluation committee will use a mixed
methods approach that will draw on a combination of analytical techniques and on a range of
both quantitative and qualitative data sources. By assessing whether there is convergence and
consistency among different data sources and methods, the evaluation committee will seek to
triangulate findings that support reasonable, plausible linkages to outcomes and impact (Greene
et al., 1989; Leeuw and Vaessen, 2009). The methods and data in the mix will complement each
other, and will each have different strengths and limitations. This approach helps to account for
the reality that, even given access to all potential data sources and extensive evaluation
resources, there still would not be direct measures to answer many of the evaluation questions
posed in the charge to the IOM. However, when taken together, the totality of evidence will
allow the evaluation committee to draw conclusions and make recommendations for the program
as a whole.
Program Impact Pathway
Figure 3 shows the program impact pathway that the planning committee developed to
represent a plausible causal chain of results for PEPFAR. The pathway begins with a series of
investments or inputs to the program. For PEPFAR, these inputs include not only funding and
other resources but also strategic planning, programmatic and policy guidance, technical
assistance, and knowledge transfer and research that represent the evolving evidence base. These
inputs support activities that provide services and support to children, adolescents, and adults in
need. Although these services are described by PEPFAR in categories like prevention, treatment,
11
Many of the terms used in the program impact pathway have different meanings in different fields of research. In
this report, the terms correspond to definitions that reflect the current consensus in program evaluation. Definitions
can be found in the glossary (Appendix D).
OCR for page 23
23
PROPOSED EVALUATION PLAN
and care and support, the conceptual framework acknowledges that they are all part of an
interrelated and overlapping approach, which also includes activities around gender issues and
capacity building. These activities result in outputs that are measureable proximal effects. When
the program is implemented well, these outputs are expected to produce outcomes as
intermediate effects on the pathway to the ultimate goal of health impact. These intermediate
outcomes include, for example, the delivery of high-quality, efficient services that are available
and accessible to the targeted populations and that are achieving the intended and appropriate
coverage. Other target outcomes include, for example, health systems strengthening; changes in
individual risk behavior; and changes in knowledge, norms, and attitudes that affect sexual
behavior, stigma, and gender issues. Ultimately, the program is intended to operate through this
pathway to contribute to an impact on individual and population health and well-being, including
HIV incidence, HIV prevalence, morbidity, and mortality.
Data will not be available to directly measure all of the outcomes and impacts illustrated
in the impact pathway, and when available may need to come from sources other than PEPFAR.
In some cases, such as assessing effects on incidence, proxy measures or modeling data will have
to be used. A critical advantage of the program impact pathway approach is that it identifies the
intermediate steps between the inputs invested in the program and the ultimate impact on health.
This allows the evaluation to consider not just the beginning and endpoints, but also to assess
whether the program is performing in the way it is intended along the full range of its
implementation. Thus, even when it is not possible to assess impact directly, the evaluation
committee will be able to state plausible findings about the effects of the program and draw
conclusions that provide more refined and useful information about elements of the program that
are functioning well or that could be improved in order to result in a greater impact on health.
For each of the programmatic areas that will be assessed in the evaluation, the committee
will work from more specific program impact pathways. These are described in Part III of this
report, along with illustrative evaluation questions based on the committee’s interpretation of its
charge to assess PEPFAR’s performance and impact. All of the specific program impact
pathways are oriented to describe outcomes that contribute to the HIV-related health impacts
shown in Figure 3, which represent the stated overall goals of PEPFAR.
OCR for page 24
24
FIGURE 3 Program impact pathway for evaluation of PEPFAR’s effects on HIV-related health impact for children and adults. In the case of joint
PEPFAR and Global Fund programs, some inputs may be provided by the Global Fund.
OCR for page 25
25
PROPOSED EVALUATION PLAN
Although it provides a critical guide for developing evaluation questions, assessing data
sources, and selecting methodologies, the program impact pathway is of course a simplified view
of PEPFAR programs and their impact. Of particular importance for this evaluation’s conceptual
framework is the reality that in any country that receives PEPFAR support, the program operates
within the context of a wide range of other factors that affect the implementation of the program
as well as health outcomes (see Figure 4). Investments from a range of other sources support
programs that are aimed at the same desired outcomes, and the proportion of total HIV/AIDS
support that is provided by PEPFAR varies from country to country. In some cases, multiple
funding sources may be co-mingled to support the same programs. Therefore, changes in
population health that can be used to reflect program impact cannot be separated by specific
programs or investments. Even individual measures can be difficult to attribute directly, as an
individual or household may be receiving different services from different programs funded
through different sources, all of which have an impact on the health outcomes of the beneficiary.
Health outcomes are also influenced by a wide range of cultural, societal, geographical, and
political factors and influences that the program cannot control. In addition, as PEPFAR
programs increasingly operate with an emphasis on country ownership and harmonization with
national plans, the extent to which central USG guidance and authority can influence all levels of
priority-setting, decision making, and implementation can be quite limited. Finally, with a
foreign assistance program that is implemented as broadly and on the scale of PEPFAR, there is
rarely an appropriate comparison available in order to attribute outcomes to the program based
on what would have happened in the absence of the investment.
Therefore, although the ideal goal in a program impact assessment may be to determine
to what extent a desired outcome can be attributed directly to the program or policy investment,
the realities of a large-scale program such as PEPFAR can make it difficult to determine the
extent to which successes or failures in achieving the intended effect can be attributed directly to
the program. Thus, the aim of this proposed evaluation approach is not to attempt to determine
the direct attribution of PEPFAR funds to health outcomes. Rather, the aim is to assess the
plausible contribution of PEPFAR to changes in health impact, both globally and by country,
within the landscape of broader funding, programs, and other factors that influence health. This
contribution analysis approach is consistent with the guidance given to the committee by
congressional staff about expectations for the evaluation (Bressler, 2009; Marsh, 2009). It is also
accepted as an appropriate standard for large-scale development assistance programs (Leeuw and
Vaessen, 2009).
There may be some areas in which attribution can be more readily determined or
approximated, as in the direct relationship at the first step of the impact pathway between inputs
and the activities they support, or in the case of controlled experimental studies that assess the
effects of intervention components that are distinct to PEPFAR, or in countries where PEPFAR
is or has been the nearly exclusive funder of all national HIV/AIDS activities. If feasible, when
these opportunities arise, the evaluation committee will consider whether a finding of attribution
may be plausible.
OCR for page 26
26 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
FIGURE 4 Context for PEPFAR program implementation.
NOTES: M&E = monitoring and evaluation; USG = U.S. government; NGOs = non-governmental
organizations.
OCR for page 27
27
PROPOSED EVALUATION PLAN
OVERARCHING EVALUATION CHALLENGES AND LIMITATIONS
There are a number of overarching challenges to carrying out this evaluation. These are
described here, while more specific challenges and limitations are described in more detail in the
following sections on evaluation methodologies and data sources as well as in the subsequent
sections in Part III of this report that address specific components of the evaluation.
One of the primary challenges to the evaluation is that there are limited data to address
health impact and other evaluation questions about the whole of the PEPFAR program.
Therefore, many of the evaluation questions will require additional sources of data, analytical
approaches, and methodologies. In the mixed methods approach described in this evaluation plan
some limitations with readily obtained impact data may be overcome by using other proposed
comparison methodologies, ancillary studies, key informant interviews, and site visits. These
methods are described in more detail in the subsequent sections on evaluation methodologies and
data sources. The type of methodology and analytical approach used to answer specific questions
requested by Congress will differ depending on the rigor and feasibility of collecting existing
data or the feasibility of gathering new information—as the committee intends to do during
country visits and during interviews with OGAC staff, implementers, and other key stakeholders
that will occur outside of country visits. This poses limitations on the evaluation approach and on
the interpretation of the findings.
There are indicators that are reported centrally to OGAC across the entire PEPFAR
program; however, these provide only limited answers to the evaluation charge. Although data
from within PEPFAR that go beyond the centrally-reported indicators may be available, a
preliminary scan of sources revealed that these data will have to come from disparate sources
that are not currently catalogued or coordinated. These data sources, such as recommended
indicators not reported to OGAC and data collected by the major USG implementing agencies
and other implementation partners, are not managed through a process that allows for easy
cataloguing of what is available from whom. Therefore, accessing the data to answer many of the
evaluation questions will require a significant data-mapping and data-gathering effort that adds
to the resource requirements of the evaluation. Requests from the IOM will also likely impose a
burden of time and resources on staff at OGAC and other implementing agencies as well as on
country teams and implementing partners; this introduces a dependence on the timely efforts of
many different actors who, in many cases, are already overburdened.
Another possible source is data that are collected through other multilateral organizations
and can be made available to the IOM. However, some of these data, as well as data from OGAC
and implementing partners, are not already analyzed in a way that answers the questions posed
by this evaluation in response to the congressional mandate. Therefore, to make full use of these
data would require new analyses. In addition to questions requiring new analyses, some
questions would require developing and using new data collection tools, which may or may not
be feasible during the time period of the evaluation. Each of these approaches could enhance the
quality of the evaluation, but each will also require a greater investment of resources than
restricting the evaluation to existing data and analyses.
Another important limitation to note is that most of the additional data sources will not be
PEPFAR-wide data or population data but will instead be country-specific, program-specific, or
component-specific. When data are not collected systematically across all PEPFAR countries,
this will limit the ability to generalize findings to the whole of the program. There is also
considerable heterogeneity in the implementation of PEPFAR across different countries and
OCR for page 38
38 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
available infrastructure and capacity prior to the introduction of PEPFAR. In addition, many
countries where PEPFAR has not been implemented may have implemented similar
interventions to achieve the same objective through programs with support from other external or
national funding sources. When this is the case, comparisons cannot evaluate the presence or
absence of the intervention activities supported by PEPFAR per se, but rather the implementation
and delivery strategy used by PEPFAR. As a result, the approach of comparing PEPFAR to non-
PEPFAR countries would require special care in implementation, analysis, and interpretation.
For this kind of comparison approach to be useful for this evaluation, it would be critical
to identify control countries that can be suitably compared to PEPFAR countries. The control
counties would be selected from the same geographic regions as the PEPFAR countries (e.g.,
sub-Saharan Africa, Asia, the Russian Federation or Eurasia, and the Caribbean). The evaluation
committee’s work would begin by assembling a database of baseline country-level variables in
both PEPFAR and non-PEPFAR countries that might relate to the course of the epidemic. The
committee will also document investments in HIV activities from country governments and
external donors. The validity of using this comparison approach to draw reliable inferences about
the effects of PEPFAR will depend on whether the analyses can be adequately adjusted to make
fair comparisons between PEPFAR and the candidate non-PEPFAR control countries. Adjusted
analyses that statistically control for differences will be considered. The dependent variables in
the adjusted analyses will be the relative or percentage changes in key indicators before and after
introduction of PEPFAR. The before-after percentage changes in PEPFAR countries will be
compared to non-PEPFAR countries, adjusting for differences in baseline variables and taking
into account HIV activities supported from other sources. As with the comparisons among
countries within PEPFAR, a major limitation is that there are important uncertainties in some of
the key benchmark indicators used as the dependent variables, such as HIV prevalence and
numbers of HIV-related deaths. In addition, there are a number of measures of interest for this
evaluation for which data are not collected across PEPFAR and non-PEPFAR countries, which
would limit the scope of this approach in addressing many of the evaluation questions drawn
from the statement of task.
DATA SOURCES AND ANALYSIS
The extent to which the goals of this evaluation can be met depends on the availability of
relevant and timely data. As described previously, most evaluation questions will require the
evaluation committee to draw on data that go beyond the indicators that are reported centrally to
OGAC. These data will have to come from a range of disparate sources. The availability of this
data will partly depend on the feasibility of access within the timeframe of the evaluation. There
will also be challenges of sampling and interpretation due to heterogeneous data sources with
different data collection systems and criteria, as well as the potential for reporting bias in the
responses to data requests from the committee. The approach for collecting and assessing data
that could be used for the evaluation is described here and in the subsequent section on the
workplan for the evaluation.
Mapping of Data Sources
The time and resources available for the planning phase did not allow for a complete
mapping of all currently available and anticipated data sources in time for this report. In the
OCR for page 39
39
PROPOSED EVALUATION PLAN
operational planning period the IOM staff, under the guidance of the planning committee, will
continue an extensive data-mapping effort, expanding on the preliminary scan of data sources
conducted during this strategic planning phase. The mapping will occur through document
review, informant interviews, information obtained from domestic and international data
requests, and qualitative methods used during three pilot country visits. The timing of this
evaluation, with a final report to be delivered in 2012, dictates that the committee will be
considering only data that are or will become available to the committee through 2011.
This mapping will determine what data are available for each of the PEPFAR countries,
providing the evaluation committee with a data matrix similar to the template that can be found
in Appendix G. Some of these data sources will also be mapped for non-PEPFAR countries to
inform the feasibility of the comparison approaches described earlier—these approaches would
rely on the availability of data from these countries, and on the willingness and capacity of
stakeholders in non-PEPFAR countries to participate in country visits and other data-gathering
requests. This mapping of available data will also include an assessment of the feasibility of
collecting data from each source, taking into consideration the burden that additional data
requests would place on each source’s resources and staff time. In addition, this data mapping
will assess whether data from each source would require new data analysis in order to answer the
evaluation questions posed by the committee.
The categories and some examples of available data sources that will be mapped and, if
available, used for the evaluation are listed in Boxes 2a–d. These include central USG data
sources, data from multilateral organizations, country-level data from both PEPFAR and other
sources, and data from additional sources, which may be from single countries or multiple
countries. The applicability of specific data sources to address illustrative evaluation questions in
some specific programmatic areas will be discussed in the subsequent sections of this report.
OCR for page 40
40 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
BOX 2a
Central U.S. Government Data Sources
Office of the U.S. Global AIDS Coordinator (OGAC) Reports and Planning/Guidance
Documents: OGAC periodically releases reports of its activities as well as programmatic,
policy, and reporting guidance for field programs. Most of the reports are requested by
Congress or required under federal regulations. Guidance for field programs includes both
formal guidance documents and other communications from headquarters to implementing
partners and country teams.
5-year strategic plans
Country Operational Plan guidance
PEPFAR indicators reference guide (including the Next Generation Indicators Reference
Guide)
Programmatic guidance
Partnership Frameworks and Partnership Framework Implementation Plans Guidance
Public health evaluation guidance
Reporting guidance for the annual program results (APRs)/semi-annual progress results
(SPRs)
PEPFAR annual reports and other reports to Congress
PEPFAR operational plans
Obligation and outlay reports
PEPFAR State of the Program Area
News to the Field
Data Reported to OGAC through the Country Operational Plan Reporting System (COPRS
II): As part of PEPFAR’s monitoring and evaluation (M&E) of activities, countries are required to
report program data through COPRS II. Countries submit two reports to OGAC annually (APRs
and SAPRs), which include data from the essential and reported PEPFAR indicators* collected
from implementing partners on all technical areas.
Congressional Appropriations Bills and Conference Reports: The U.S. House of
Representatives and U.S. Senate Committees on Appropriations and their appropriate
subcommittees have the broad responsibility over the discretionary budget for global HIV/AIDS
bilateral funding and the U.S. government funding for multilateral organizations such as the
Global Fund to Fight AIDS, Tuberculosis and Malaria.
Office of Management and Budget (OMB): OMB Circulars are instructions or information
issued by OMB to federal agencies. These are expected to have a continuing effect of 2 years
or more. PEPFAR funding for HIV/AIDS is subjected to OMB Circulars.
PEPFAR Implementing Agencies Data: Program monitoring, evaluation, and research data as
well as other relevant information over which these agency have oversight (e.g., principally the
Office of HIV/AIDS within the Global Health Bureau at the United States Agency for International
Development [USAID] and the Global AIDS Program at the U.S. Centers for Disease Control
and Prevention).
OCR for page 41
41
PROPOSED EVALUATION PLAN
PEPFAR External Evaluations: Reports of evaluations of PEPFAR conducted by other U.S.
government agencies, including the Government Accountability Office. Congressional Research
Service, OMB, and the Offices of the Inspector General for the Department of State, USAID,
and the Department of Health and Human Services. These include reports on topics such as
program management and implementation, coordination, funding allocations and oversight,
technical assistance, harmonization, and program efficiency and effectiveness.a
* PEPFAR guidance classifies indicators in three ways: by degree of importance and
aggregation level (i.e., essential and reported to headquarters, essential and not reported to
headquarter, or recommended indicators), by reporting level (i.e., direct program or national
indicators), and by standard M&E classification (i.e., output, outcome, or impact indicators).
SOURCE: Compiled from U.S. Government publicly available information and PEPFAR’s website
(www.pepfar.gov).
a
For example: CRS (2005,2007, 2008a, 2008b, 2009); DoS OIG (2008, 2009); GAO (2004, 2005, 2006,
2008, 2009).
OCR for page 42
42 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
BOX 2b
Multilateral Donor and Other International Data Sources
Multilateral donors and international organizations play an active role in implementing
global commitments on HIV/AIDS and supporting these through funding and technical
assistance. Data available from multilateral donors and international organizations are reported
by national governments, which are generally required to report on the progress of externally
supported HIV/AIDS programs. The following are examples of these types of sources of data:
Joint United Nations Programme on HIV/AIDS (UNAIDS)
Global HIV statistics and other estimates (e.g., “Report on the Global AIDS Epidemic” and
“AIDS Epidemic Update” reports)
Frameworks and Indexes (e.g., National Composite Policy Index and Stigma Index)
National AIDS spending assessments
United Nations General Assembly Special Session on HIV and AIDS country progress
reports
Project and indicator data collected, analyzed, and reported through the Country Response
Information System or the Indicator Registry
United Nations Children’s Fund (UNICEF)
HIV Statistics and other socio-economic statistics affecting child well-being (i.e., “State of the
World’s Children” annual reports)
Frameworks (e.g., Five-year global campaign on children and AIDS)
Publications (e.g., the “Children and AIDS Stocktaking Reports”)
Technical and policy documents
Review of status of programs (addressing focus areas: preventing mother-to-child
transmission of HIV; providing pediatric treatment; preventing infection among adolescents
and young people; protecting and supporting children affected by HIV and AIDS)
World Health Organization (WHO)
Data and statistics (e.g., data on testing and counselling, mother-to-child transmission of
HIV, antiretroviral therapy, and pediatric HIV)
National Health Accounts
Country-specific antiretroviral drug costs
HIV drug resistance monitoring reports and literature
WHO normative guidance and publications
The International Health Regulations 2005
The Global Fund to Fight AIDS, Tuberculosis and Malaria (Global Fund)
National Health Accounts
Progress reports including technical support grants from PEPFAR
Key performance indicators
Global Fund five-year evaluation: Study area 3 reporta
Global Fund evaluation country case studies
OCR for page 43
43
PROPOSED EVALUATION PLAN
The World Bank
Public expenditure reviews
Project documents
Analytic work/research
Health and HIV/AIDS project evaluations
Evaluation of HIV/AIDS support Bank-wideb
Country assistance evaluations
Organisation for Economic Co-operation and Development (OECD)
HIV-related funding data
Country surveys
Evaluation studies
Other Multilateral or International Data Sources:
aids2031 reports and working papers
Committee on the Rights of the Child, States reports
Millennium Development Goals reports
UNITAID reports
European HIV/AIDS Funders Group
Interagency Group for Mortality Estimation
Funders Concerned about AIDS
SOURCE: Compiled from the Global Fund, OECD, UNAIDS, UNICEF, WHO, and World Bank publicly
available information and personal communications with individuals at these organizations.
a
TERG (2009).
b
For example: IEG World Bank (2009) and World Bank (2007).
OCR for page 44
44 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
BOX 2c
Country Data Sources
PEPFAR Country Sources: Program data and other information generated at the country level.
Country operational plan (fiscal years 2004–2011)
Partnership framework and implementation plan
Prime and sub-prime partner reports
OGAC indicators not centrally reported*
HIV Programs costing data
Other communications among country teams and implementing partners
National Policy Documents and other National AIDS Response Information: Relevant
national policy documents, strategies, and plans of action supporting PEPFAR activities and/or
beneficiaries of PEPFAR-funded activities.
National AIDS Coordinating Authority’s strategy and framework
Agencies or departments policy documents and plans (e.g., Ministries of Health, Finance)
Country harmonization and alignment tool surveys
National Health Information Systems: National health information systems play an important
role in ensuring that reliable and timely health information is available for operational and
strategic decision making about HIV/AIDS country programs.
Census data
Civil registration and vital statistics
Ministries of Health and Finance data
Health services records
Population surveys (e.g., Multiple Indicator Cluster Survey, Demographic and Health
Survey, AIDS Indicator Survey, Behavioral Surveillance Survey, and Biologic and Behavioral
Surveillance Survey)
Antenatal care surveillance data
Facility surveys (e.g., Service Provision Assessment and Service Availability Mapping)
*Additional essential/not reported to headquarters and recommended indicators beyond the 25
essential indicators reported to headquarters collected at the country and partner level.
SOURCE: Compiled from publicly available information and personal communications.
OCR for page 45
45
PROPOSED EVALUATION PLAN
BOX 2d
Other Data Sources (Single or Multi-Country)
Public Health Evaluations: Concept papers, protocols, and/or progress reports for each
approved PEPFAR public health evaluation (PHE). PHEs are investigator-initiated studies
intended to guide the PEPFAR program and future policy development, to provide evidence to
the HIV/AIDS community on programs that work, and to identify gaps in knowledge that can be
filled with timely program evaluation and research.
Published Literature: Peer-reviewed journal articles, grey literature, and other reports relevant
to PEPFAR’s activities. These will address country-specific or program-specific studies as well
as technical areas such as operations research of HIV programs, prevention of mother-to-child
transmission, sexual prevention, blood safety, injection safety, intravenous and non-intravenous
drug use, male circumcision, adult and pediatric care and treatment, tuberculosis and HIV co-
infection, counseling and testing, health systems strengthening, gender-related HIV issues.
Existing Modeling Data Sources for Costing: PEPFAR works with countries in estimating
resources needs for interventions. At the country-level, PEPFAR uses several models including
the ART Cost Project, HIV/AIDS Program Sustainability Analysis Tool, and Spectrum Policy
Modeling System.
Existing Modeling Data Sources for HIV Infections Averted: Since the numbers of HIV
infections averted due to the implementation of a specific intervention(s) cannot be measured
directly, modeling approaches provide a proxy to measure impact (e.g., models that estimate
the efficacy of the intervention on incident HIV infection, models that describe how HIV
infections are mediated by behavior, and models that compare incidence trends with the
expected or baseline HIV incidence trends).
SOURCE: Compiled from publicly available information and personal communications.
Analysis and Interpretation of Data
The evaluation committee will guide the implementation of the evaluation and data
analysis, interpret the data, and deliberate to come to consensus on the findings, conclusions, and
recommendations. Primary data and secondary data that require additional analysis will be
analyzed, using appropriate statistical methodologies, by the members of the evaluation
committee and, with the committee’s guidance, by the study staff team, which will be augmented
for the implementation of the evaluation with additional staff trained in statistical analysis and
data management. In addition, the committee will use specific subcontractor services for some
areas where there is specialized knowledge needed with a substantial time commitment above
what the volunteer committee members can provide. For example, expert consultation will
contribute to the design of the tools and methods for qualitative data collection and to oversight
of the analysis of primary qualitative data collected during country visits, other structured
interviews, and other qualitative methods. Expert consultation will also be used to advise and
assist in designing and supervising appropriate data requests and quantitative/qualitative analysis
OCR for page 46
46 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
of secondary data. The committee will oversee all analyses performed by subcontractors to
ensure validity and rigor as well as integration with the overall evaluation methodology.
The committee, staff, and consultants will take steps to quantify the quality and
completeness of the data used for the evaluation. For the primary data collected by the
committee, the methods used to assure the quality of the data will be described in full in the final
report. When existing data analyses are used, the committee will review and assess the
methodology and quality of the data in the original analyses. When secondary data are requested
and used for new analyses conducted by the committee, a request will also be made for a
description of the data management plan, and the committee will assess the procedures in place
to assure the quality of the data, including, whenever possible, parameters such as reporting
rates, sampling frame, and data completeness. This information will allow the evaluation
committee to assess possible reporting bias and data quality and take these factors into account to
inform the evaluation committee’s interpretation of the data based on the likely reliability and
quality. In its final report, the committee will include an accounting of data requests as well as a
summary and analysis of the data quality and completeness. This will include the number of data
requests made and the extent to which these requests were completed as requested. For data
requests within PEPFAR, this will also afford the committee an opportunity to assess the
completeness and validity of data as a metric of program progress toward sufficient data
collection capacity for M&E, a critical component of sustainability. Although the assessment of
data requests and data quality will be reported in the aggregate, data request outcomes will not be
linked in the report to the specific organizations that receive data requests to avoid inhibiting the
reporting of data.
WORKPLAN
Operational Planning Phase
As the culmination of the planning phase for this study, a transitional operational
planning period will take place between the delivery of this report and the implementation of the
evaluation itself. As described previously, the operational planning activities were intentionally
structured and approved by Congress and OGAC as part of an ongoing planning phase, after
delivery of the report, so that work on the evaluation could continue uninterrupted and so that the
evaluation committee would not be starting de novo with respect to data availability and
cataloguing, pilot-testing of instruments and methods, and development of relationships with
relevant stakeholders. The results of these operational planning activities will be detailed in staff-
authored planning documents for the evaluation committee as part of their background
information and preparation to implement the evaluation.
Activities in this period will be carried out by IOM staff and planning committee
members and will be designed to further develop and refine the plan described here and to
inform the implementation of the evaluation. The operational planning will focus on data
mapping (sources and availability of relevant data); mapping of methods and data sources,
including key indicators, to the mandated tasks and illustrative questions in order to refine and
prioritize key evaluation questions and identify key indicators; developing procedures for data
requests; initiating data requests; designing and initiating data quality review methods for data
received; refining and testing country visit selection criteria; preparing country timelines and
other background materials for PEPFAR countries, and developing country study frameworks
OCR for page 47
47
PROPOSED EVALUATION PLAN
and methods for country visits. Some initial structured interviews with key informants will also
take place during the operational planning period, and a final operational planning task will be
continued relationship-building with relevant stakeholders such as contacts in PEPFAR countries
and at implementing partner organizations.
Initial discussions with OGAC staff about data availability and the preliminary data scan
conducted during this planning phase revealed that much of the data that the committee will need
are not available through the headquarters level. In addition, the committee learned that
implementing partners, agencies, and countries do not necessarily have to share a lot of their
available data with OGAC. In light of this, OGAC agreed to partner with the IOM to help
facilitate, to the extent they are able, access to these data by making introductions to field,
headquarters, and agency staff and by disseminating information about the purpose of the
evaluation. An initial introduction was sent in a News-to-the-Field posting from OGAC on June
4, 2010, which explained the mandate for the study, the progress of the planning committee as of
the posting date, the proposed data-mapping activities and pilot country visits during the
operational planning phase, and that the IOM data requests, country visits, site visits, and
interview would be entirely independent of the relationship of implementing partners and other
country-level stakeholders with OGAC and other USG implementing agencies. It also assured
that the evaluation is not a financial audit or evaluation of specific programs; that findings,
examples, and comments would not be attributed without expressed permission; and that
participation in the evaluation is voluntary. 14
Operational planning activities will also include the use of a qualitative research and
evaluation consultant. This consultant will help to develop and refine data collection instruments
and processes for country visits, in-depth studies, and other qualitative data collection; to
determine what design issues, options, and qualitative methods beyond interviewing might be
appropriate and feasible (e.g., content/thematic, statistical, or combination analyses; systematic
triangulation; focus groups; direct observations for contextual information; town halls;
photovoice); to plan logistics for field work; to test illustrative questions for refinement; to make
determinations about the balance of breadth versus depth within the design options and data
collection instruments; to develop audit trails to assure rigor of the fieldwork; and to train IOM
staff in qualitative methods and the use of qualitative analytical software. In addition, pilot
testing and refinement of field research methods and data collection instruments, with the
qualitative consultant, will occur during visits to three PEPFAR countries, one bilaterally- and
two multilaterally-funded.
Implementation Phase
The evaluation committee will produce one consensus report with its findings and
recommendations. This report is targeted for delivery to Congress by Fall of 2012. The overall
time line for the evaluation will be approximately 24 months. The first 18 months will be data
collection and analysis, building on the activities of the operational planning phase. This will
also include consultation with relevant domestic and international stakeholders, implementing
partners, and others with relevant expertise. The remaining six months will include final data
analysis and interpretation of findings, determination of conclusions and recommendations by
consensus among the committee members, finalization of the committee’s report, an
14
Personal communication from OGAC, June 4, 2010.
OCR for page 48
48 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
institutionally-overseen peer-report review, report production, and briefings for the sponsor as
requested.
Over the course of the evaluation, the full committee will meet at least four times in
person, with participation of the subcontractors and consultants. Additional virtual meetings will
be conducted as needed using videoconferencing, teleconferencing, and web-based conferencing
tools. In addition, working groups within the committee focused on specific content areas will
hold in-person and virtual meetings, as needed, for ongoing deliberations as well as data analysis
and interpretation. These committee activities will be augmented by ongoing communications,
by telephone and electronic mail among the committee members, staff, and subcontractors and
consultants.
A summary schematic of the proposed work plan and timeline for the evaluation can be
found in Appendix F. Adjustments may be needed to the timeline and work plan due to any delay
in the start time of the evaluation phase or to uncontrollable external shocks such as man-made
or natural disasters (e.g., Haitian earthquake), political instability that could jeopardize the safety
of members in countries that are identified for committee visits, or unforeseen scheduling
problems for traveling (e.g., the Icelandic volcano eruption in 2010).
