Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 67
3 Designing an Evaluation That Incorporates the Guiding Principles of Coordination, Harmonization, and Capacity Building Workshop participants focused their discussions not only on what should be evaluated, but on the ideal process of designing and conducting the evaluation itself. Participants stressed the importance of an earnest ef- fort to design and conduct the evaluation in a way that truly incorporates the principles of coordinating evaluation efforts among global partners, harmonizing with evaluation needs of country partners, and contributing to strengthened local evaluative capacity. Many lessons for designing the process of evaluation can be learned, participants noted, from previous experience with harmonization, coordination, and capacity building in the context of implementation of human immunodeficiency virus/acquired im- munodeficiency syndrome (HIV/AIDS) interventions through the President’s Emergency Plan for AIDS Relief (PEPFAR). This chapter summarizes the benefits, costs, and opportunities of coordination and harmonization, as well as for capacity building in evaluation. BENEFITS, COSTS, AND OPPORTuNITIES OF COORDINATION AND HARMONIzATION IN EvALuATION Benefits of Coordination and Harmonization in Evaluation Workshop participants outlined the value and benefit of coordina- tion and harmonization in impact evaluation. Discussant Jim Sherry of George Washington University noted that coordination and harmoniza- tion in evaluation design and implementation are important in influencing 

OCR for page 67
 EVALUATING THE IMPACT OF PEPFAR others’ work, which is in turn critical for addressing strategic questions at a broader policy- or program-practice level. It is useful to distinguish, he noted, between an internal, institution-specific evaluation that relates to tracking needs for program survival and direction change and a shared international evaluation that relates to influencing how other actors use resources and to strengthening overall capacity. Given the high program transaction costs of evaluation for countries and partners, collaborating as much as possible also minimizes work and ensures more efficient use of funds, observed speakers Julia Compton of the UK Department for Interna- tional Development (DFID) and Agnes Binagwaho of the Rwanda National AIDS Control Commission. Speaker Sara Pacqué-Margolis of the Elizabeth Glaser Pediatric AIDS Foundation further underscored the importance of coordination to ensure maximum use of evaluation dollars, noting that suc- cessful completion of impact evaluations involves high human and financial resource costs, an extensive time frame, and serious commitment. Speaker Mary Lyn Field-Nguer of John Snow, Inc., pointed out that the strengths and perspectives brought to the table by focus-country government part- ners, implementing partners, and other stakeholders are a further benefit. She and other workshop participants outlined the value each stakeholder brings to the evaluation process. Partner Countries The value of engaging partner countries is their accountability to their citizens, observed Field-Nguer. Government partners were dealing with the challenges of service delivery to their populations long before PEPFAR and can provide a critical perspective on health system issues, such as health care workforce and supply chain issues, and how PEPFAR is addressing these, she added. Even in the context of an emergency situation, involve- ment of partner countries in the design of evaluation questions and meth- odology can improve the quality of the overall evaluation design and the interpretation of evaluation findings, noted speaker Binagwaho. Speaker Kathy Marconi of the Office of the U.S. Global AIDS Coordinator (OGAC) stressed that involvement of partner countries can support the process of developing evaluation priorities that are actually meaningful to the coun- tries, and this has implications for sustainability. Implementing Partners The value of engaging implementing partners is their familiarity with program data and lessons and their understanding of the challenges of delivering services across a continuum of care, stated Field-Nguer. Coordi- nation with implementing partners, with their knowledge of the lessons of

OCR for page 67
 AN EVALUATION THAT INCORPORATES THE GUIDING PRINCIPLES decades of AIDS work, for example, can contribute to avoiding mistakes, such as repetition of research to answer questions already answered and implementation of program strategies that do not work. Implementers add value not just as reporters of data and information, but also as users of data and information, and hence need to be involved in the evaluation design process, observed speaker Nils Daulaire of the Global Health Council. Beneficiaries and Other Stakeholders Local people add value to the evaluation process because they can place change in context, said workshop participant Joanne Capper of the U.S. Peace Corps. Speaker Field-Nguer added that service beneficiaries are the experts on program impact and are an audience that can define the preven- tion, care, and treatment service parameters of acceptability, accessibility, and affordability. In the end, patients, clients, and community members define the effectiveness of care services. However, she noted, few studies are available about the community level of knowledge. Field-Nguer identified specific groups of stakeholders who can add value to the design phase of impact evaluation through choosing the ques- tions and the methodologies. These groups include the following: • People living with HIV/AIDS • Women • Youth, including those in and out of school • Other populations at risk • Government ministries—beyond ministries of health—and local management units • Health care workers at all levels in urban and rural settings • Health facilities • Nongovernmental and community organizations working inside and outside facilities • Community leaders • Religious leaders Constraints of Coordination and Harmonization in Evaluation High transaction costs are among the greatest constraints to coordina- tion and harmonization in evaluation, workshop participants said. There is tension between the benefits of coordinating and taking advantage of exist- ing synergies by linking with others’ evaluation work and the costs of that coordination, remarked workshop moderator Ruth Levine of the Center for Global Development. “The mere sharing of information is a huge task that can be all consuming,” observed Ambassador Jimmy Kolker, OGAC.

OCR for page 67
0 EVALUATING THE IMPACT OF PEPFAR In addition to being costly, the processes of design, consensus building, commitment, planning, and working with host countries are very complex, asserted speaker Pacqué-Margolis. Speaker Paul De Lay of the Joint United Nations Programme on HIV/ AIDS (UNAIDS) pointed out that the transaction costs of inclusiveness and consensus building are particularly pronounced when immediate results are desired, as was the case for PEPFAR. However, he noted, in order to sustain desired results, harmonization, integration, and sharing across partners will be necessary. Overtaxing or overextending evaluators with efficiency-level questions was a further constraint to coordination identified by discussant Sherry. Questions at this level prevent partners from influencing others’ work and being influenced. He emphasized that the focus of coordina- tion and harmonization should be on strategic, broad, or program-level questions. Institutionalizing accountability to program beneficiaries through co- ordination may be constrained when there is a large power imbalance such as that existing between donors and partner countries, observed another workshop participant. When countries depend on donor resources, which could potentially be removed, there is concern that countries may not stand up to donors and speak up for what they want, particularly if a strong coordinating plan and leadership are not in place. Institutionalization of such accountability, noted speaker Norm Daniels of the Harvard School of Public Health, depends heavily on the effectiveness of the national coor- dinator and the national plan. Donors who view country ownership as a desirable objective also must be open to acknowledging what counts as fair and reasonable at the national level. Opportunities for Coordination and Harmonization in Evaluation Efforts Workshop participants described a number of opportunities for coor- dination and harmonization with global partners and country partners in evaluation design and implementation. Coordinating Evaluation Design, Conduct, and Results with Global Partners Workshop participants strongly articulated the need for PEPFAR to coordinate its evaluation efforts with other global partners. PEPFAR should not just be learning from its own evaluations, said speaker Rachel Glennerster of the Abdul Latif Jameel Poverty Action Labora- tory, it should be looking at work being done elsewhere. Another par- ticipant stressed that such coordination should include divulging and

OCR for page 67
 AN EVALUATION THAT INCORPORATES THE GUIDING PRINCIPLES sharing negative results with global partners. Speaker De Lay noted that in parallel to PEPFAR’s evaluation efforts, The Global Fund to Fight AIDS, Malaria, and Tuberculosis (The Global Fund); the World Bank; and UNAIDS are conducting evaluative efforts, and bilateral efforts, such as DFID’s AIDS program, are also undergoing evalua- tion. Although these different evaluations address unique issues, con- stituencies, and time frames, there is strong potential for sharing data, approaches, and evaluation research. De Lay suggested that outcomes of PEPFAR evaluative efforts be tied to and shared with other global and bilateral evaluations of AIDS institutions and initiatives. Several of these efforts are described in greater detail below. The Global Fund. Speakers De Lay, Compton, Marconi, and John Novak of the U.S. Agency for International Development (USAID) supported the suggestion that PEPFAR collaborate with The Global Fund evaluation, particularly with regard to broader questions such as systemwide effects. As De Lay noted, The Global Fund is just starting a 5-year evaluation with a specific set of questions about the effectiveness of the funding models, level of partner support, and technical assistance. The Global Fund evaluation will devote nearly $15 million of the $17 million total evaluation budget to research, using prospective study survey evaluation research methods. Discussant Kolker noted that OGAC is currently in close contact with The Global Fund impact study organizers. Speaker Theresa Diaz of the U.S. Centers for Disease Control and Prevention (CDC) added that an OGAC technical group will work with The Global Fund impact evaluation techni- cal group to review all methodologies and analyses used. In every PEPFAR country, a group of government contacts has been designated as part of the task force working with The Global Fund on the impact evaluation, she said. uNAIDS. Speakers De Lay, Marconi, and Stefano Bertozzi of the Na- tional Institute of Public Health, Mexico, along with discussant Kolker, suggested that PEPFAR engage collaboratively with UNAIDS. According to De Lay, UNAIDS had a major evaluation in 2001–2002 and will now start a second evaluation. These evaluations have focused on the role and impact of UNAIDS cosponsors in a changing environment. Bertozzi noted the particular value of engaging with the economics reference group at UNAIDS; Marconi mentioned the strengths of the UNAIDS modeling reference group in developing global-level impact measures, such as those for stigma and gender discrimination. Kolker noted that the United States is the main supporter of the UNAIDS monitoring and evaluation (M&E) reference group.

OCR for page 67
 EVALUATING THE IMPACT OF PEPFAR World Bank. PEPFAR collaboration with the World Bank on evaluation may be beneficial in developing an integrated HIV/AIDS research agenda, noted speaker Jody Kusek of the World Bank. She said the World Bank evaluation will focus on the impact of technical assistance such as treat- ment, scale-up of treatment facilities, prevention, programs that affect policy instruments (such as cash transfer policy instruments as incentives for behavioral change), and socioeconomic impacts of HIV/AIDS; the ef- fectiveness of HIV/AIDS programs in achieving goals; and the design of new investments to ensure that impact can be assessed. OECD. Collaboration with the Organization for Economic Cooperation and Development (OECD) on evaluation may be helpful in the area of HIV/ AIDS that deals with gap analysis and meta-evaluations, speaker Compton said. WHO. Discussant Kolker reported that OGAC has been involved in the annual meeting on the World Health Organization’s (WHO’s) HIV/AIDS impact evaluation. Coordinating with partners beyond the AIDS community. Discussant Sherry suggested that drawing on evaluation expertise beyond the AIDS community, such as in the areas of sustainability and broader development issues, may be beneficial to the evaluation of PEPFAR. Coordinating Evaluation Design, Conduct, and Results with Country-Level Partners Workshop participants described several opportunities for harmoniza- tion of evaluation with country partners. Speaker Pacqué-Margolis asserted that the evaluation function should be prioritized in country operation plans; partners should be included in evaluation planning; and funding mechanisms should promote harmonization. Workshop discussant Phillip Nieburg of the Center for Strategic and International Studies envisioned that country partner organizations would be given an opportunity to re- view drafts and submit comments and concerns in the evaluation process. Speaker Binagwaho and workshop participant David Stanton of USAID highlighted the importance of community-level interpretation of evaluation results. Several speakers pointed out that impact evaluation should mirror, and perhaps draw lessons from, the experience in coordinating and harmonizing with partner countries in program implementation. Field-Nguer reminded participants that the guiding philosophy of PEPFAR, the “three ones,” includes the concept of a single monitoring and evaluation plan shared with partner countries. Speaker Daulaire commented that lessons about

OCR for page 67
 AN EVALUATION THAT INCORPORATES THE GUIDING PRINCIPLES coordination and harmonization also can be drawn from the experience in universal childhood immunization and smallpox eradication, in which stakeholder engagement was done effectively and with a view toward sustainability. Specific opportunities for country-partner harmonization on evaluation are described below. joint field evaluations. Discussant Kolker remarked that aid effectiveness could be improved if common program and sectoral approaches used joint evaluation visits based on a single national plan and a single moni- toring and evaluation system. Speaker Binagwaho offered the Rwandan experience—in which partners from PEPFAR, The Global Fund, and the government of Rwanda routinely conduct joint field visits for evaluation purposes—as a potential model. Centralized funding and data aggregation. Echoing points made in Shan- non Hader’s presentation about the value of a central coordinating hub for impact evaluation, speaker Pacqué-Margolis argued that a dedicated, central funding source is operationally better suited to coordinate the evalu- ation effort across countries. She noted that funding mechanisms that send money to the country level do not promote harmonization on evaluation because countries often have little money left over for M&E. Pacqué- Margolis further stated that evaluation findings need to be aggregated across countries—also by a central coordinating unit—to be meaningful and to see patterns. knowledge management. Speaker Field-Nguer spoke of the opportunity to use evaluation coordination to develop a mechanism for knowledge management, both for existing data and for methodologies to gather new data. Harmonization of fairness-monitoring approaches. Speaker Daniels sug- gested that harmonization be used to work toward agreement on which aspects of equity, accountability, and efficiency will be part of the M&E program. He noted that such discussions provide an opportunity to clarify the overall program objectives. BENEFITS, CONSTRAINTS, AND OPPORTuNITIES OF BuILDING CAPACITy IN EvALuATION Benefits of Building Capacity in Evaluation Workshop participants emphasized the importance of designing an evaluation that itself strengthens local capacity. In-country capacity is

OCR for page 67
 EVALUATING THE IMPACT OF PEPFAR needed not just for service delivery, noted speaker Pacqué-Margolis, but for M&E and for continuous quality improvement, advocacy, national planning, and budgeting. Building local evaluative capacity has particular benefit as PEPFAR undergoes a transition from an emergency program to a long-term, sustained program, speakers noted. In an emergency response, observed speaker Bertozzi, the time to build local capacity on evaluation is- sues is limited. In contrast, a long-term response involves, for example, not just training existing health practitioners but also educating new ones, and not just involving local researchers in a project, but also building the capac- ity of local health researchers to do prospective evaluations. Field-Nguer added that local capacity and systems to collect, analyze, and use program information are of critical importance to program success and sustainability and should be built into the process of PEPFAR program implementation and impact evaluation. Constraints to Building Local Capacity in Evaluation Lack of systems for gathering data, inadequate funding mechanisms, and poor stakeholder engagement are among the constraints to building local capacity for evaluation, workshop participants said. Speaker Kusek observed that many countries are unable to take advantage of the 5 percent to 10 percent of total project budget available for evaluation under World Bank loans because they often don’t have systems in place for gathering data. There is powerful competition for resources between the develop- ment of monitoring systems and the implementation of the program, she said. Pacqué-Margolis observed that although funding mechanisms should develop and promote capacity for evaluation, information dissemination, and advocacy, they often fall short. Funding earmarks and directives limit the prioritization of M&E, operations research, clinical research, and advo- cacy in Country Operation Plan planning. Poor or distorted engagement of stakeholders is another constraint to building local capacity for evaluation. Many partners, including country-level stakeholders, do not have a place at the table in defining the research agenda and conceptual frameworks, noted Pacqué-Margolis. In contrast, researchers tend to drive the evaluation agenda according to their own interests, noted Kusek, using resources to conduct more narrowly focused impact evaluations that are less helpful. Opportunities for Strengthening Evaluative Capacity Workshop participants suggested a number of opportunities for strengthening local capacity to conduct impact evaluation. Diaz suggested that donors could disseminate suggested methodologies, offer technical assistance, conduct training workshops, and engage in one-on-one mentor-

OCR for page 67
 AN EVALUATION THAT INCORPORATES THE GUIDING PRINCIPLES ing. Binagwaho added that the recruitment and training of independent consultants in partner countries is another way to develop competency in evaluation methods that will remain in the country. Country-driven deci- sion making and priority-setting processes were also suggested as potential models or tools for strengthening local evaluative capacity. Binagwaho suggested that the experience of comanaged decision making in Rwanda potentially could be applied to evaluation design. Workshop participant Laura Porter from CDC suggested that a country-driven evaluation priority- setting process—drawn up by a multidisciplinary team—could be another mechanism for building local capacity. She noted that a need for such a process was articulated at a recent meeting of the M&E Reference Group Evaluation Subgroup.

OCR for page 67