Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
8 Recommendations This report spans from the Human Resources for Health (HRH) Pro- gramâs vision of increased institutional capacity to produce highly qualified health workers, to this evaluationâs specific charge to assess the Programâs effects on outcomes for people living with HIV (PLHIV). It thus reflects a balancing act that is widespread in the global health landscape. Most countries have both broad health system needs, including health workforce needs, and deep unmet needs with respect to specific disease burdens. In the context of Presidentâs Emergency Plan for AIDS Relief (PEPFAR) funding, this balancing act takes the form of decision making about how to meet HIV-specific needs, which is the core of PEPFARâs mission, in the context of health systems that lack sufficient capacity to meet either HIV- specific or broader population health needs. Over time and across different investments, PEPFAR has exhibited both relatively siloed (or âverticalâ) ap- proaches, focused on HIV-specific efforts, which might have spillover effects on the health system as a whole, and broader (or âhorizontalâ) approaches, focused on systems-based efforts that also meet the needs of PLHIV who are served within that system (PEPFAR, 2011a,b, 2014, 2019; Samb et al., 2009). The vision for the HRH Program framed its impact through the latter horizontal perspective. The charge of this evaluation framed impact from a more vertical perspective. OVERARCHING EVALUATION CONCLUSIONS The HRH Program had many successes with respect to its goal to expand the quantity and quality of the health workforce in Rwanda, with 227
228 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA particular examples in the value it added to the quality of health profes- sional education and training for different cadres of health professionals, especially in nursing, and improvements in the overall preparation and mo- tivation of new professionals entering the workforce. The Program was seen by those in both health professional education and health service delivery roles as contributing to improving the quality of care for all Rwandans, including PLHIV, through direct and indirect pathways. Some of these suc- cesses resulted from the original design, as intended, whereas others were more unexpected, resulting from adaptations that were made in response to operational realities or challenges encountered. Given the complexity of the health system and HRH within that system, these successes were ac- companied by challenges that offer opportunities for learning. There were mixed results with respect to the ambitious goals of the HRH Program to increase institutional capacity for health professional edu- cation, resulting from a truncated time frame, operational challenges in its implementation, and insufficient design and planning around the intended mechanisms of change and the complexity of structural changes needed to achieve improvements in health professional education. With respect to outcomes for PLHIV, it was much more difficult to assess the Programâs effects. There are indications that it contributed to improved quality of care, and the evaluation found no indications to sug- gest that the allocation of funds to this Program undermined Rwandaâs continuing progress in the HIV response. However, the evaluation was con- strained in being able to fully assess the Programâs contribution to impact on PLHIV-specific health outcomes. The HRH Program represented what was, at the time, a relatively un- common (although not unique) donor-funded approach to strengthening HRH capacity in low- and middle-income countries by focusing on a large investment, at a foundational level, for capacity building in institutions for health professional education. For PEPFAR, it also represented a departure from the usual operational model between funder and government. When seen in light of this committeeâs charge, the exceptional nature of the Pro- gram ended up being a missed opportunity to learn from what could have been a more intentionally designed approach that could have added new insights to the knowledge base not only for how to strengthen HRH capac- ity, but also, more broadly, for how to navigate the balancing act between disease-specific priorities and broader health system needs. IMPLICATIONS FOR HIV AND HUMAN RESOURCES FOR HEALTH PROGRAMMING Lessons learned from the experience of this Program point to broader implications for how to undertake this balancing act in future HRH pro-
RECOMMENDATIONS 229 gramming. For HIV in particular, as Rwanda and other countries make laudable progress toward targets related to epidemic control and im- proved coverage of antiretroviral therapies, more PLHIV are living longer, with health needs that increasingly lie at the intersections of managing HIV and its complications over time, managing comorbid conditions, and attending to quality of life. Increasingly, supporting the needs of PLHIV and sustaining comprehensive control of the HIV epidemic rely on the foundation of a strong and comprehensive health system. It is therefore in the interests of PEPFAR and other disease-specific funders to contribute to HRH and other health systems efforts that, to be most effective, are not designed around a specific disease, but meanwhile need not interfere with progress related to that disease. Yet, it is also reasonable for funders to expect that those investments in broader efforts will have effects that contribute, albeit not exclusively, to disease-focused outcome goals. Pro- grams or initiatives can be designed in ways that optimize and monitor that disease-specific effect, without interfering with the broader systems effects. Such investments have the greatest potential to yield sustainable results when aligned with long-term systems strengthening strategies and coordinated with concurrent investments from governments and other donors. RECOMMENDATIONS The committee was tasked with making recommendations âto inform future HRH investments that support PLHIV and to advance PEPFARâs mission.â The committeeâs recommendations are therefore framed with the intention of helping to make the balancing act between disease speci- ficity and systems strengthening more achievable, and more measurable, for future investments in HRH. The recommendations presented here posit that a future role for PEPFAR, or any other funder with a disease- specific mandate, would be to make investments in HRH that take a more explicit, intentional, and longer-term âdiagonalâ approach that focuses on finding the intersections between the vertical and the horizontal. These recommendations provide a framework for how future efforts could build on the lessons learned from this Program, both reinforcing its successes and making accommodations to address its challenges, with a design that more fully accounts for needs and feasibility at baseline and ensures more of the needed information will be available to learn about the effects on the system and the effects on the response to HIV. Although the primary audiences for this evaluationâs findings and conclusions are PEPFAR and the government of Rwanda, the committee hopes that the con- clusions and recommendations will inform other funders and other institu- tions contributing to strengthening the health workforce, such as medical
230 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA and health professional training institutions, professional societies, patient advocacy groups, and other civil society organizations. Furthermore, the hope is that the lessons learned from this Program and the committeeâs recommendations might inform not only future efforts in Rwanda but also elsewhere in the region. Recognizing the inherently complex interactions among many factors, stakeholders, institutions, and sectors when it comes to HRH, these recom- mendations emphasize the following aspects of an integrated approach to improving HRH through health professional education: â¢ Codesign among relevant stakeholders; â¢ A systems approach with adequate needs assessment and planning time; â¢ Operational planning that emphasizes adaptive management; â¢ Selection of a tailored set of components from models for improv- ing health professional education; â¢ A prospective and multifaceted approach to monitoring, evalua- tion, and learning; and â¢ In all of the above, an explicit connection from disease-specific ele- ments to interrelated broader systems elements. Program Codesign To ensure a robust and feasible programmatic design, an effective approach is to employ a collaborative design process at the level of key decision makers representing funders and government leadership across relevant sectors, while including implementers and beneficiaries (in the case of health professional education, faculty, trainees, the public and private health systems that will employ program graduates, and ulti- mately, patients). When embarking on a health systems strengthening program, it is important to engage all relevant government entities be- yond the Ministry of Health, including the Ministries of Finance, Labor, and Civil Service, and other government bodies, to ensure the national budget and policies support the programmatic objectives. This inclusive, multilayered design process can ensure that the effort responds to the need, reflects contextual realities, and has the potential to be executed effectively. Recommendation 1: Funders investing in strengthening human re- sources for health should support a codesign model through a pro- cess that engages representatives from diverse stakeholders as the
RECOMMENDATIONS 231 designers,1 including funders, program administrators, implementers, regulatory bodies, and those who will use or benefit from the funded activities. The 2008 Accra Agenda for Action and the 2011 Busan Partnership for Effective Development and Co-operation both endorse collaborative ap- proaches throughout the program cycle. The Accra Agenda noted that part- nership efforts for development should âfully harness the energy, skills, and experience of all development actorsâbilateral and multilateral donors, global funds, CSOs [civil society organizations], and the private sectorâ as only then are development efforts most effective (OECD, 2008). The Busan Partnership elaborated, with development partners pledging to encourage parliaments and local governments to strengthen their role âin the oversight of development processesâ and to â[f]urther support local governments to enable them to assume more fully their roles above and beyond service delivery, enhancing participation and accountability at the subnational levelsâ (Fourth High Level Forum on Aid Effectiveness, 2011). Similarly, Busan acknowledged civil society organizationsâ key roles in âshaping de- velopment policies and partnerships, and in overseeing their implementa- tionâ and urged development partners to create an enabling environment that allows these organizations to maximize their contributions (Fourth High Level Forum on Aid Effectiveness, 2011). In addition, Busan strongly encouraged the expansion of South-to-South partnerships and âtriangular cooperation,â noting that these âhave the potential to transform developing countriesâ policies and approaches to service delivery by bringing effective, locally owned solutions that are appropriate to country contextsâ (Fourth High Level Forum on Aid Effectiveness, 2011). Busan also supported âparticipation of the private sector in the design and implementation of development policies and strategiesâ and encour- aged ârepresentatives of the public and private sectors and related organiza- tions to play an active role in exploring how to advance both development and business outcomes so that they are mutually reinforcingâ (Fourth High Level Forum on Aid Effectiveness, 2011). When it comes to strengthening 1â Later recommendations that actions be taken by âdesigners of programs to strengthen human resources for healthâ refer to this group of diverse stakeholders: funders (e.g., multi- lateral donors, bilateral external government donors, philanthropic donors, national govern- ments, and private payors), program administrators (e.g., government leadership in relevant sectors and nongovernmental leadership), implementers (e.g., program managers and health professional education program leaders), regulatory bodies (e.g., accreditors and professional councils), and those who will use or benefit from the funded activities (e.g., faculty, trainees, public or private health systems, and patients, including people living with HIV).
232 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA HRH, the role of the private sector is particularly salient, given the increas- ing presence in low- and middle-income countries of both private-sector health care delivery and private medical, nursing, and other health profes- sional schools. As one example of a version of codesign, the United States Agency for International Development (USAID) has begun applying cocreation methods to the design of its funded activities (USAID, 2017). During this process, diverse stakeholders (such as implementing partners, host-country governments, private-sector representatives, and local organizations and experts) are invited to a participatory workshop, in which they contribute to the design and structure of an upcoming activity. The aims of this ap- proach are to enhance local ownership of the activity and to increase the probability that the activity will achieve its intended results, because its design is informed by diverse stakeholders. For complex and long-term initiatives such as the HRH Program, which sought to strengthen HRH through building health professional education, including a process whereby different stakeholders formally articulate their commitments over different time spans can reduce longer- term risk. These risks include the donorâs ceasing funding of the program, as occurred with the HRH Program, or the host governmentâs not follow- ing through on commitments to recruit and retain the health care workers who have graduated. These eventualities can always occur, but any steps to minimize this risk should be considered. Design with a Complex Systems Thinking Lens The World Health Organization (WHO) has described the application of systems thinking to health systems strengthening, recognizing that âevery intervention, from the simplest to the most complex, has an effect on the overall system, and the overall system has an effect on every interventionâ (WHO, 2009). The interactive, nonlinear, and often unpredictable relation- ship among parts of a health system warrants approaches to strengthening HRH that are designed around its inherent complexity. Recommendation 2: Designers of programs to strengthen human re- sources for health should employ a complex systems thinking lens, including multisectoral approaches that mix top-down and bottom-up models with long-term flexible funding that can support both the im- mediate needs of a health system and longer-term issues, such as the retention of health workers. Applying complex systems thinking can change how program design- ers conceive of the challenges in the health system, the questions they ask
RECOMMENDATIONS 233 about how to improve the system, and their understanding of the environ- ment that either supports or hinders improvement (Swanson and Widmer, 2018). A systems approach to strengthening the health system should also recognize that the health system is nested within a larger government, and the health workforce is nested within regional health labor markets. This necessitates multisectoral collaboration and coordination across the health, education, labor, and finance sectors and among governmental and nongov- ernmental institutions. Other features of the system also need to be recognized as additional layers of complexity, such as cultural considerations and concurrent shifts in standards and norms. The HRH Program, for example, accompanied a transition in Rwanda from French- to English-language health professional education and a transition in the region away from the Francophone model of medical education. Global health movements should take such local transitions into account in the design and implementation of programs that aim to affect systemwide changes. Similarly, the quality of primary and secondary education need to be considered to ensure that students progressing to health professional education programs have adequate foun- dational knowledge to be successful. Therefore, complex systems thinking needs to be an integral part of the codesign process described under the first recommendation. Time Frame Considerations for Strengthening HRH Effective systems strengthening for HRH, especially in relation to fac- ulty production and building university infrastructure, requires decades. In- vestments in health professional education and training should be expected to take many years to yield effects, given the time required for students to complete training programs, for trainees to make their way as fully qualified health professionals into the service delivery system, and for new or newly upgraded faculty to produce ongoing cohorts of providers. There are examples of longer-term projects that have yielded systems changes, such as a nearly 20-year collaboration between Uganda and the Swedish International Development Cooperation Agency, executed via a partnership between Makerere University and the Karolinska Institute of Sweden, that has produced a joint Ph.D. program and a wealth of research products contributing to policy and program design (Sewankambo et al., 2015). This example indicates that the duration required to build institu- tional capacity is on the order of decades, whereas the HRH Program, as laid out in the 2011 proposal, was planned for 9 years and faced a signifi- cant drop in funding following the cessation of PEPFAR investment after 5 years, which had implications for addressing sustainability, institutionaliza- tion, and other issues that surfaced in the midterm review.
234 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA Although it may be more feasible to invest in shorter, concrete projects with observable benefits, such as infrastructure and commodities distri- bution, doing so may overlook investments that require more time but have more widespread effects. There is value in being strategic about the investments, âwith an eye toward making long-term investments in global health instead of focusing on short-term expendituresâ (NASEM, 2017). With respect to HIV/AIDS, these investments need to be made in response to the anticipated future of the epidemic, strengthening a health system to be able to care for an aging PLHIV community with an increasing burden of comorbid conditions. Building systems thinking approaches into health professional education is one mechanism for moving toward a âculture of health,â positioning clinical staff as leaders (Phillips and Stalter, 2018). Funding Considerations for Strengthening HRH Because systems strengthening takes decades and the HRH pipeline spans multiple stagesâfrom recruitment of students to preservice training through specialization and continuing professional development of the workforce, to longer-term issues such as health worker motivation and retentionâdesigners of HRH programs should articulate and work to- ward comprehensive long-term goals and outcomes. This will require local governments and funders to collaboratively develop funding strategies that can outlast political terms and agendas, as well as typical donor funding cycles, and enable a built-in transition to sustained country-led ownership and financing. Governments should focus on assembling diversified funding sources and partners for HRH programming, recognizing that some donors have adopted a more broad-based approach to development assistance, while others take an approach focused on specific outcomes or interventions. Re- liance on a single donor can jeopardize broader goals for sustained systems change if donor priorities shift over time. Governments are also positioned to identify and bring together public- and private-sector actors with vested interests in national HRH goals in order to coordinate financing initiatives. When aligned with a comprehensive HRH strategy based on a decades-long time frame, a diversified approach would enable governments to assemble a portfolio of shorter-term investments and programs with disease-specific, activity-specific, or time-bound parameters that are all coordinated toward achieving the defined overall strategy. Donors should accommodate this longer-term, coordinated funding approach by being open to the explicit integration of their investments into a broader strategy that catalyzes sustainable change. This can be achieved through participation in a codesign process that encompasses not only a specific donor-funded program, but also how it ties into the broader national
RECOMMENDATIONS 235 HRH strategy and building sustainable local capacity. This coordinated funding and design approach would also provide an opportunity to attend to the extent to which donor funding adheres to the Addis Ababa Action Agenda. This agenda calls for shifting the balance toward domestic funding and away from donor funds that ultimately go to external parties, as was the case in the design of the HRH Program, where the majority of the budget was dedicated to U.S. institutions for faculty contracts (UN, 2015). Once committed to contributing to a longer-term approach, it is im- portant for donors to recognize that subsequent shifts in their priorities that affect funding midcourse, as occurred with PEPFARâs funding for the HRH Program, will have broader consequences. Mutual expectations for a transparent process around potential revisions in funding and program- ming should be clearly outlined from the outset, and if anticipated shifts in funding arise, they should be considered and planned collaboratively and in relation to the broader coordinated strategy. Donors should also, to the extent feasible, offer greater flexibility in shaping and adapting program budgets and processes to more readily ac- commodate a role in a broader funding strategy. U.S. government funders should build on USAID practices that have begun to address program funding constraints. Some of that agencyâs procurement processes have in- corporated an âinception periodâ to flexibly yet systematically revisit initial objectives, targets, and outputs, instead of requiring program proposals to commit rigidly to achievements and outputs within an established time frame. Additionally, USAID has allowed âwindows of opportunityâ to ded- icate a portion of project budgets for adapting strategies and development programming based on changing conditions and enhanced understanding of needs (Brinkerhoff et al., 2018). Systematic Approach in the Context of the Labor Market Although programs and policies that focus on boosting health pro- fessional education are key to addressing the health workforce shortage, adopting a labor market lens can both leverage health professional educa- tion investments and redress factors that undermine the capacity of the health workforce (Evans et al., 2016; Sousa et al., 2013). Health service delivery is highly labor intensive and requires the appropriate number and mix of trained and motivated health professionals to provide high-quality health services. A health labor market is a dynamic relationship between the supply of health workers and the demand for health workers (McPake et al., 2013; Scheffler et al., 2016). When supply and demand are placed in the context of national goals for access and coverage, the need for health services and the health needs of the whole population come into play as a third important and dynamic factor. It is important to identify the optimal
236 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA mix of cadres and their geographic distribution in response to different needs and goals, such as addressing the HIV epidemic, having a broader impact on health, and teaching and workforce professionalization. This mix will need to be adjusted as goals and needs change over time. In a health system, prevailing demand-side forces, especially in the absence of universal health coverage, can skew the supply of health work- ers away from population health needs; the private sector can fuel such imbalances with competing wages, working conditions, and regulations for health professionals (Evans et al., 2016; Sousa et al., 2013). Governments should develop production policies in the education sector in tandem with policies that address how the new supply of health workers can be absorbed into the labor market (Sousa et al., 2013). Furthermore, career progression, which differs according to cadres of health workers, needs awareness and consideration to ensure that existing health workers have opportunities for growth that can be absorbed by the health system and that sufficient num- bers of health workers are being produced to take on the duties of those who have advanced. Governments also have a role in regulating the private sector to ensure the quality of training, given the rise of private health training institutions, and the equitable distribution of health care workers (Evans et al., 2016; Sousa et al., 2013). In turn, the private sector can drive innovative publicâ private models for financing growth in health worker education in response to market opportunities and other areas where governments are unable to respond (Evans et al., 2016). Planning and Adaptive Management Because it relies on complex systems change, strengthening HRH re- quires not only visionary leadership and effective program activities, but also an appropriate and sufficient management structure to shepherd a pro- gram through an inevitably multifaceted and complicated implementation process. The experience of the HRH Program points to the need for strong management structures and processes that allow for continuous learning and improvement as a means of moving toward the defined programmatic goals, even in the face of policy pivots such as PEPFARâs shift from its 2.0 to 3.0 strategy, which resulted in a determination to cease funding. Recommendation 3: To maximize the effectiveness of investments in human resources for health, which inherently require change within a complex system, designers of programs to strengthen human resources for health should spend time before implementation to establish a shared vision, proposed mechanisms to achieve that vision, and an operational plan that takes an adaptive management approach.
RECOMMENDATIONS 237 Often, the reality of implementation after program design necessitates changes to the proposed mechanisms to ensure progress toward the origi- nal vision and goals. This is particularly true in projects aimed at making changes in complex systems, such as human resources and the health sys- tem. Adaptive management, or problem-driven iterative adaptation, enables an intentional approach to making decisions and adjustments to program- matic activities in response to emerging information, unintended conse- quences, unexpected challenges, or changes that take place in the context in which the program is being implemented (USAID, 2018; Woolcock, 2018). An adaptive approach can begin as early as the design stageâwith a focus on adapting best practices and external solutions to create a âbest fitâ version for the context (Woolcock, 2018)âand continue throughout the life cycle of the program. The key principles of adaptive management include â¢ Reframing project design and implementation from a linear project trajectory to a more flexible sequencing; â¢ Building in management structures that are capable of being flexible; â¢ Creating explicit, periodic windows for assessing and reconsidering implementation decisions; and â¢ Linking adaptation to learning by creating a feedback loop between decision making and real-time information on the programâs prog- ress and its struggles (Brinkerhoff et al., 2018). The intent is not to find the best action, but rather to identify which of the available and feasible options move closer to realizing the systems-level changes the program envisioned (Ripley and Jaccard, 2016). Adaptive management is also underpinned by robust and continuous data collection for real-time information and decision making (Brinkerhoff et al., 2018). Processes for reviewing the data generated and documenting the rationale behind decisions taken should also be institutionalized. As discussed under Recommendation 6, there is a need for HRH programs to include a comprehensive approach to monitoring, evaluation, and learning as an integrated responsibility not only for designated staff but also for other technical and operational staff. Adaptive management is only possible if the people implementing the program are open to critically engaging with and learning from rapidly collected and analyzed data to make programmatic adjustments. This re- quires a culture of improvement at the programmatic and organizational levels. Debating the decisions being made, including consideration of effects on other aspects of the system, both within and beyond the program, can strengthen the quality of decisions while fostering a culture of learning and improvement (Allana, 2016). These conversations need to happen not
238 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA just within the program, but in consultation with a department within a Ministry of Health that holds the broader HRH strategy, comprising not only administrators and managers, but also economists and finance experts, political scientists, and other scientific and technical personnel. Although there are various potential modalities to achieve this, it is the coordinated consultation across functions that is essential for effective adaptive man- agement. Simultaneously, defined roles and responsibilities around decision making are necessary to move beyond debate and data review and into action. One lesson from the HRH Program is that it could have benefited from a more clearly defined and more robustly supported decision-making structure to facilitate timely and data-driven adaptive management to en- able improved implementation. For adaptive management to succeed in the context of reliance on multiple funding sources, external donors and governments that fund HRH programs need to embrace the approach of expecting well-executed imple- mentation to include clarity of rationale and specificity of design at the out- set, and learning-based adjustments as implementation proceeds. Program assessment and accountability mechanisms should be based partly on the process of implementation and on achieving reasonable progress toward goals, in the context of the realities encountered during implementation, rather than basing accountability narrowly on adherence to the origi- nal design. Where incremental changes are made as part of an adaptive management plan, sufficient time needs to be given to allow the changes to yield improvements. Donors are increasingly recognizing this need for adaptability to make effective investments, as evidenced by initiatives such as the World Bankâs Global Delivery Initiative and the Doing Development Differently manifesto (ODI, 2016; USAID, 2018). Models for Improving Health Professional Education Varied models and approaches have been developed for improving health professional education (including twinning, interprofessional educa- tion, and technology-enabled education) and have been established as op- tions that could be considered in the design phase of a program, depending on the programâs needs and goals and the country context. Recommendation 4: Designers of programs to strengthen human re- sources for health should, on the basis of the vision and goals of the program, evaluate different models for improving health professional education that best fit the workforce needs to be met and the local structural and contextual considerations for human resource capacity building.
RECOMMENDATIONS 239 Crosscutting Considerations Across Models Selection of a health professional education model should be based on the goals and vision of the program and the needs of the health workforce. There should also be a focus on future institutionalization through: â¢ Ensuring structures are in place to support faculty in the longer term (career progression, time to commit to the program); â¢ Ensuring faculty have time built into their schedules to commit to additional health professional education development; â¢ Ensuring time and funding are provided to focus on accreditation, research skills, and other elements that are not a direct transfer of teaching skills; â¢ Emphasizing longer-term institutional partnerships; and â¢ Teaching models that require less time from faculty (blended learning). Technology is used in a wide variety of ways for health professional education throughout low- and middle-income countries. E-learning, dis- tance learning, web-based training, and m-learning (using mobile de- vices) enable remote education and training interactions and can be key strategies for reaching the workforce in resource-limited environments (Ballew et al., 2013; Buabeng-Andoh, 2018; Murebwayire et al., 2015). Simulation-based tools are another use of technology to provide training and assessment to improve quality of care and reduce medical errors (Puri et al., 2017). Blended learningâcombining technology tools with traditional face- to-face teaching approaches (Pavalam et al., 2010)âhas been put for- ward as a way to create a more learner-centered environment, build student engagement, and relieve overcrowding (Frantz et al., 2011). It is a growing model owing to its effective pedagogy, cost-effectiveness, and increased faculty time for student mentorship (Geoffrey, 2014). In Rwanda, students and staff have reported positive attitudes toward the integration of blending learning (Pavalam et al., 2010), and the imple- mentation of a blended e-learning approach has been used for nursing education to make access to training and classes more feasible for those living in rural areas (Murebwayire et al., 2015). When blended learning is adopted, however, it is important to consider the potential implications and additional needs it can create, such as requiring additional training to understand how to implement blended learning, designing appropri- ate curricula, and the need for adequate access to reliable technology (Geoffrey, 2014).
240 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA Optimizing the Use of Twinning When twinning models are used as part of efforts to improve health professional education, this evaluation offers several lessons for potential improvements to the process and its effectiveness, depending on the time frame, the goals, and the desired type of skills transfer. Under the HRH Program, the objective was to transfer teaching and clinical skills to Uni- versity of Rwanda faculty. Recommendation 5: Designers of programs to strengthen human re- sources for health who want to employ paired partnerships, or âtwin- ning,â should identify clear objectives to drive design decisions and consider an integrated design, with twinning partnerships at both the institutional and individual levels that are based, to the extent avail- able, on best practice guidelines. There is wide variation in the use of terminology to describe the types of activities that could fall under the concept of twinning. There are a few categorical distinctions that can be mapped to different strengths and to different considerations for implementation. Institutional twinning Institutional twinning comprises partnerships based on the relationships between institutions, through which individual faculty members or practitioners may participate. WHO, the European Unionâs ESTHER2 Alliance for Global Health Partnerships, and the United King- domâs Tropical Health Education Trust have been employing institutional twinning partnerships with African hospitals and other institutions since at least 2009 (European ESTHER Alliance Secretariat/GIP ESTHER, 2018; THET, 2019; WHO, 2019b). These organizations have well-developed definitions, practices, processes, and tools for designing, implementing, and assessing the effectiveness of institutional twinning models as well as an evidence base to support their use. Effective institutional twinning needs clear objectives that can be op- erationalized in the context of available funding and country environment. Broadly speaking, however, programs should focus on longer-term institu- tional partnerships and should consider different models of twinning and adapt to what best fits the context and need, including forming South- to-South partnerships, when possible. Institutional twinning partnerships require clearly communicating objectives, aligning interests across institu- 2â The organizationâs original name was Ensemble pour une solidaritÃ© thÃ©rapeutique hospi- taliÃ¨re en rÃ©seau (ESTHER), or Network for Therapeutic Solidarity in Hospitals against AIDS, as it was known in English.
RECOMMENDATIONS 241 tional partners, and taking into account the inherent cultural and historical dynamics of involved institutions. Individual twinning Individual twinning comprises partnerships based on the pairing of individuals in peer-to-peer, mentoring, or trainerâtrainee re- lationships. These can occur with or without the context of an institutional twinning partnership. Individual twinning may be mandated, typically by an institutional partnership wherein no prior relationship exists among twins, or emergent, developing out of an established personal relationship or other interactions. All types of peer-to-peer support should be considered as partnerships. However, there is a need to clearly define roles and relationships within these partnerships to enable effective and efficient outcomes. For example, roles could include exchanging knowledge while dividing or sharing respon- sibilities (clinical or teaching duties or shared curriculum development), mentorship (coaching and sharing of experience), training (teaching of new clinical, teaching, or research skills), or mixed roles. These roles should be predefined, shared transparently, and formally agreed upon prior to initi- ating a twinning relationship. Although emergent twinning may have an advantage in terms of the initial quality of the relationship between twins, mandated twinning, when done well under the umbrella of an institutional partnership, may have the advantage of better delineating expectations about roles and responsibilities before the relationship begins. It is imperative to consider the inherent cultural and linguistic dynamics involved in any type of individual twinning relationship. In mandated twin- ning, a matching process is needed to align skills, language, and interests of both twins as closely as possible. This matching can be enhanced by using regional experts or incorporating opportunities for emergent twinning. Ad- ditionally, offering similar conditions to the twins with respect to compen- sation and incentives, where possible, could facilitate greater engagement in twinning relationships. Preparing and coaching twins through their relationships can be a mechanism for building skills such as cross-cultural understanding, communication, and conflict management and resolution. Operationalizing peer-to-peer twinning support should also consider methods such as mixed distance learning or bidirectional international place- ments of shorter durations. This could be particularly effective for deans of institutions, or for medical or surgical subspecialists for whom longer-term placements are impractical. Using ratios greater than one to one for the part- nering between external and local twins could also be an effective approach to optimize capacity building for health professional education. Contributing to the evidence base on twinning There is a limited evidence base on twinning methodologies and their effectiveness. Programs that
242 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA develop robust plans for learning, as discussed further in Recommendation 6, will have important and much needed opportunities to contribute to the knowledge base. Monitoring, Evaluation, and Learning Although the importance of monitoring, evaluation, and learning in health systems strengthening programming is recognized, operationalizing an effective and adaptive monitoring, evaluation, and learning system is often undervalued and underresourced. Recommendation 6: Designers of programs to strengthen human re- sources for health should craft and resource a robust and rigorous framework for monitoring, evaluation, and learning that fits the com- plex, interconnected, and often changing nature of health systems, and that balances costs and feasibility with transparency, accountability, and learning. For future investments in HRH, a low-cost but rigorous monitor- ing, evaluation, and learning plan and system will be most effective if it is included in the design phase and incorporates ongoing mixed-methods monitoring, with pause points for learning throughout the program, and the resources and staff to achieve realistic and actionable learning. Moni- toring and evaluation capacity among in-country program managers and implementers should be strengthened to support ongoing monitoring and data use for decision making. The advantages and disadvantages of also using an external third party for evaluation should be weighed and con- sidered as part of the design. In addition to planning for learning about implementation processes and program outcomes, it would be valuable for future efforts to prospectively plan for analysis that would allow program designers, implementers, and others looking to learn from such programs to understand the costs of program implementation and select, plan for, and carry out assessments of return on investment. Below is a breakdown of monitoring, evaluation, and learning options at different points in a programâs time line. Design and Start-Up It is important for the program to draw on a wide base of evidence to increase relevance and effectiveness in the countryâs current context. This could be done in a multitude of ways. One is to conduct background re- search on other approaches, especially models in the region, and how they would need to be adapted for the Rwandan (or other countryâs) context,
RECOMMENDATIONS 243 as well as what the trade-offs and opportunity costs would be in choosing from among different approaches to achieve the desired objectives. In the HRH Program, although the designers referenced other regional twinning programs, there was not a clear process for how to better understand the operational infrastructure and overhead needed to operationalize such programs in the Rwandan context. Another option is to conduct a land- scape mapping of existing actors (nongovernmental organizations, govern- ment, private sector) working to improve HRH. To deepen this mapping, a network analysis to investigate the social and organizational structures, relationships, and interactions can help designers understand the roles of different actors in an existing system and how they influence each other and work together toward a common goal, such as strengthening HRH (Ramanadhan et al., 2010). The design phase should include a priori development of indicators to evaluate the programâs effectiveness, efficiency, and outcomes within the overarching structure of the partnership, and a plan, including funding, for analysis and dissemination of findings to improve the knowledge base for similar endeavors in the future. Designers and program administrators should also conduct a baseline assessment that maps the countryâs HRH needs to the design of the program, helping to assess how to balance spe- cialized care and primary care, the cadres required to address HIV and to have a broader health impact, and the teaching cadres who can continue building the needed workforce. A baseline assessment also gives the pro- gram a starting point for midpoint or endpoint learning. As described below, global tools and local evidence can inform a baseline assessment of this kind. WHO guidance and tools WHO provides guidance on HRH, such as the recommended density of health workers in relation to the general popula- tion at a minimum of 4.45 per 1,000 (WHO, 2016), as well as tools and guidelines to assist countries with HRH planning (WHO, 2019a).3 These serve a number of purposes, such as identifying gaps and estimating specific needs within the health workforce (WHO, 2010),4 estimating or evaluating the size and mix of nursing teams (Hurst, 2002),5 and supporting planning, policy, and leadership for HRH (MSH and WHO, 2006). Several tools are available specifically to assess HIV-specific workforce needs and the intersection between HRH and the response to HIV (MSH 3â WHOâs collection of HRH tools and guidelines can be accessed at https://www.who.int/ hrh/tools/planning/en (accessed December 19, 2019). 4â See https://www.who.int/hrh/resources/WISN_Eng_UsersManual.pdf (accessed December 19, 2019). 5ââââSee https://www.who.int/hrh/documents/hurst_summary.pdf (accessed December 19, 2019).
244 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA and WHO, 2006).6 These tools can be used according to their relevance to the specific context, and in combination, as needed, with locally tailored needs assessments. Assessments of evolving workforce needs in Rwanda As life expectancy continues to grow in Africa (Allen Ingabire et al., 2013), the population will face an increased volume of diseases and medical issues, potentially impair- ing quality of life and survivability. Many of these conditions fall outside the realm of HIV/AIDS, and indicate a shifting burden of disease over time. In the past decade, several studies in Rwanda have spoken to evolving or emerging clinical needsâand, in some cases, the associated health work- force needs. Many of these studies have highlighted the needs of children, such as their orthopedic needs (Allen Ingabire et al., 2013) and help with management of type 1 diabetes (Marshall et al., 2013). Surgical needs as- sessments have included examinations of capacity for emergency surgery (Petroze et al., 2012), nonobstetric surgical care (Muhirwa et al., 2016), and the epidemiology, management, and outcomes of surgically treated malignancies (Atijosan et al., 2009). Other assessments have touched on specialty needs such as trauma care (Ntakiyiruta et al., 2016), oncology capacity building (Stulac et al., 2015), and cancer control (Martin et al., 2019). One study concludes that weak infrastructure for health care, insuf- ficient clinicians and training programs, and a lack of supplies pose risks to the treatment of and capacity building for cancer (Stulac et al., 2015). An- other suggests that to improve accurate and timely diagnosis, national can- cer control plans should include capacity building for general practitioners, and continuing professional development should address âcontext-specific educational gaps, resource availability, and referral practice guidelinesâ (Martin et al., 2019). There are also more generalized and unifying workforce and capacity- building needs. Rwanda faces barriers of unmet need and educational and resource gaps that affect the quality of care its health workforce can pro- vide. For example, significant variability and fundamental gaps have been described in adult and adolescent primary care delivery (Vasan et al., 2013). An additional, widespread capacity-building and workforce develop- ment challenge lies in the availability of information. For example, Nahi- mana and colleagues (2015) describe limited information on technology scale-up in rural health facilities, and Egziabher and colleagues (2015) describe a lack of data on quality of care for obstetric fistula management. A similar issue exists in emergency medicine, where there are no data on 6â Questionnaires to measure the effect of HIV/AIDS on HRH and tools for planning and developing human resources for HIV/AIDS and other health services can be accessed at https:// www.who.int/hrh/tools/tools_planning_hr_hiv-aids.pdf (accessed December 19, 2019).
RECOMMENDATIONS 245 the effects of training programs on patient-centered outcomes in resource- limited settings (Mbanjumucyo et al., 2017). These studies provide assessments of need that span a variety of spe- cialties and diseases. While they indicate varied and evolving treatment needs throughout Rwanda as the burden of disease shifts over time, they also reveal overarching commonalities regarding workforce and capacity needs. Future HRH investments would benefit from a more comprehensive and coordinated assessment of these evolving needs, in light of both the common barriers and opportunities and those that are specific to diseases and specialties. Program Implementation Throughout the life of a program, it is imperative for an adaptive moni- toring system to build on existing government data systems and represent a low burden to those implementing and benefiting from the project. Ensur- ing monitoring plans move beyond collecting outputs and conceptualize implementation outcomes and mechanisms of change, including creating prospective measurement architecture, is necessary for systematically under- standing the potential impact of a program and for implementation to be responsive and reactive to the real-world context. Similarly, it is important to include early process indicators to determine whether the initiative is on the right track and make quick course corrections. To ensure data for deci- sion making are accurate, it is helpful to include data quality assessments to examine the quality of implementersâ monitoring data at various points throughout the program. Building in time for capacity-building opportunities for implementers can improve on findings from data quality assessments. This could include existing courses on standard health and health systems data (such as the District Health Information System, national health accounts, national AIDS accounts), as well as project- or program-specific capacity building. For an HRH program that uses a twinning model, monitoring the twinning process and interactions and adapting recruitment and onboarding based on this evidence could improve implementation and achievement of results. Finally, including a mapping mechanism that tracks where trainees are placed and what their roles are following the program could facilitate com- parison analysis to determine the programâs impact on patient outcomes. Pause Points for Accountability, Learning, and Adaptation Adaptive management and implementation require pause points for evaluating or assessing the program to identify opportunities for improve- ment. A mix of national and international evaluation mechanisms in HRH
246 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA and academic programs can provide rich data for informed decision mak- ing. Options include hiring an external evaluation team or funding an internal monitoring, evaluation, and learning team to conduct learning at pause points throughout the implementation life cycle. This could include more traditional baseline, midline, and end line assessments that use mixed methods and assess not only what progress has occurred, but also why it has or has not occurred. Designers could also build in special research studies between baseline and midline, midline and end line, or after a pro- gramâs completion. Special studies enable designers and implementers to learn in real time and improve implementation. This could include looking at a particular aspect of the program, such as conducting an assessment of curricula development in year 2 or an ex-post evaluation 5 to 7 years after a program has ended. If designers are interested in learning about what it takes to establish and maintain an effective HRH system, they could also build in a devel- opmental and/or process evaluation. Process evaluations allow funders to learn in real time, examining outputs and processes of why interventions are or are not working. Process evaluations provide a good platform to teach other designers what they should consider during the design of HRH programs. A developmental evaluation is particularly well suited to stimu- lating innovative program design in response to dynamic realities, possible desire for program replication, and complex issues and environments (Pat- ton, 2010). Developmental evaluations ask program administrators to work closely with evaluators as partners, sharing data and lessons throughout the program cycle. Another evaluation approach is social network analysis; in addition to being useful for the design phase, as described above, social network analysis has been applied as a tool for evaluating partnerships (Kamya et al., 2016; Kenis, 2017). Whatever approach is taken, all future HRH programs should avoid repeating this HRH Programâs missed opportunity for greater learning. This can be achieved by ensuring programs are funded with a sufficient amount of design, planning, management, and staffing with the capabilities to work with implementers to monitor, evaluate, and learn what it takes to success- fully build, implement, and sustain an effective HRH program. REFERENCES Allana, A. 2016. Navigating complexity: Adaptive management in the northern Karamoja growth, health, and governance program. Portland, OR: Mercy Corps. Allen Ingabire, J. C., R. Ssebufu, P. Kyamanya, J. Bayisenga, J. Bikoroti, A. Kiswezi, and D. Mazimpaka. 2013. Epidemiology, management and outcome of malignancies surgically treated at a rural referral hospital in Butare, Rwanda. Rwanda Medical Journal 70(4):29.
RECOMMENDATIONS 247 Atijosan, O., V. Simms, H. Kuper, D. Rischewski, and C. Lavy. 2009. The orthopaedic needs of children in Rwanda: Results from a national survey and orthopaedic service implica- tions. Journal of Pediatric Orthopaedics 29(8):948â951. Ballew, P., S. Castro, J. Claus, N. Kittur, L. Brennan, and R. C. Brownson. 2013. Developing web-based training for public health practitioners: What can we learn from a review of five disciplines? Health Education Research 28(2):276â287. Brinkerhoff, D. W., S. Frazer, and L. McGregor-Mirghani. 2018. Adapting to learn and learning to adapt: Practical insights from international development projects. Research Triangle Park, NC: RTI International. Buabeng-Andoh, C. 2018. New technology in health education: Nursing studentsâ applica- tion of mobile technology in the classroom in Ghana. Interactive Technology and Smart Education 15(1):46â58. Egziabher, T. G., N. Eugene, K. Ben, and K. Fredrick. 2015. Obstetric fistula management and pre- dictors of successful closure among women attending a public tertiary hospital in Rwanda: A retrospective review of records. BMC Research Notes 8(1):774. European ESTHER Alliance Secretariat/GIP ESTHER. 2018. Bringing together our expertise for capacity building and quality health care, edited by E. France. Paris, France: ESTHER. Evans, T., E. Araujo, C. Herbst, and O. Pannenborg. 2016. Addressing the challenges of health professional education: Opportunities to accelerate progress towards universal health coverage: A special report of the Health Professional Education Advisory Board. Doha, Qatar: World Innovation Summit for Health 2016. Fourth High Level Forum on Aid Effectiveness. 2011. Busan Partnership for Effective De- velopment and Co-operation: Fourth High Level Forum on Aid Effectiveness, Busan, Republic of Korea, 29 November-1 December 2011. Paris, France: Organisation for Economic Co-operation and Development. Frantz, J. M., S. Himalowa, W. Karuguti, A. Kumurenzi, D. Mulenga, and M. Sakala. 2011. Challenges and opportunities related to postgraduate evidence-based practice module using blended learning. African Journal of Health Professions Education 3(1):15â18. Geoffrey, N. M. 2014. Challenges of implementing quality assurance systems in blended learn- ing in Uganda: The need for an assessment framework. Huria Journal 18. Hurst, K. 2002. Selecting and applying methods for estimating the size and mix of nursing teams. Leeds, UK: Nuffield Institute for Health. Kamya, C., J. Shearer, G. Asiimwe, E. Carnahan, N. Salisbury, P. Waiswa, J. Brinkerhoff, and D. Hozumi. 2016. Evaluating global health partnerships: A case study of a Gavi HPV- vaccine application process in Uganda. International Journal of Health Policy and Management 6(6):327â338. Kenis, P. I. 2017. Passed the age of puberty: Organizational networks as a way to get things done in the health field comment on âEvaluating global health partnerships: A case study of a Gavi HPV vaccine application process in Uganda.â International Journal of Health Policy and Management 6(11):677â679. Marshall, S., D. Edidin, V. Sharma, G. Ogle, V. C. Arena, and T. Orchard. 2013. Current clinical status, glucose control, and complication rates of children and youth with type 1 diabetes in Rwanda. Pediatric Diabetes 14(3):217â226. Martin, A. N., K. M. Kaneza, A. Kulkarni, P. Mugenzi, R. Ghebre, D. Ntirushwa, A. M. Ilbawi, L. E. Pace, and A. Costas-Chavarri. 2019. Cancer control at the district hospital level in sub-Saharan Africa: An educational and resource needs assessment of general practitioners. Journal of Global Oncology 2019(5):1â8. Mbanjumucyo, G., M. A. Barry, N. Karim, and A. C. Levine. 2017. Impact of implementation of emergency medicine training on emergency department mortality in Kigali, Rwanda: An interrupted time-series quasi-experimental design. Annals of Emergency Medicine 70(4 Suppl 1):S78.
248 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA McPake, B., A. Maeda, E. C. Araujo, C. Lemiere, A. El Maghraby, and G. Cometto. 2013. Why do health labour market forces matter? Bulletin of the World Health Organization 91(11):841â846. MSH (Management Sciences for Health) and WHO (World Health Organization). 2006. Tools for planning and developing human resources for HIV/AIDS and other health services. Medford, MA: Management Sciences for Health. Muhirwa, E., C. Habiyakare, B. Hedt-Gauthier, J. Odhiambo, R. Maine, N. Gupta, G. Toma, T. Nkurunziza, T. Mpunga, J. Mukankusi, and R. Riviello. 2016. Non-obstetric surgi- cal care at three rural district hospitals in Rwanda: More human capacity and surgical equipment may increase operative care. World Journal of Surgery 40(9):2109â2116. Murebwayire, M., H. Ewing, and I. Biroli. 2015. Use of e-learning to advance nursing educa- tion in Rwanda. Rwanda Journal 2(2). Nahimana, E., M. Ngendahayo, H. Magge, J. Odihiambo, C. Amoroso, E. Muhirwa, J. N. Uwilingiyemungu, F. Nkikabahizi, R. Habimana, and B. Hedt-Gauthier. 2015. Bubble CPAP to support preterm infants in rural Rwanda: A retrospective cohort study. BMC Pediatrics 15(1):135. NASEM (National Academies of Sciences, Engineering, and Medicine). 2017. Global health and the future role of the United States. Washington, DC: The National Academies Press. Ntakiyiruta, G., E. G. Wong, M. C. Rousseau, L. Ruhungande, A. L. Kushner, A. S. Liberman, K. Khwaja, M. Dakermandji, M. Wilson, T. Razek, P. Kyamanywa, and D. L. Deckel- baum. 2016. Trauma care and referral patterns in Rwanda: Implications for trauma system development. Canadian Journal of Surgery 59(1):35â41. ODI (Overseas Development Institute). 2016. Doing Development Differently: Who we are, what weâre doing, what weâre learning. London, UK: Overseas Development Institute. OECD (Organisation for Economic Co-operation and Development). 2008. Paris Declara- tion on Aid Effectiveness (2005) and the Accra Agenda for Action (2008). Paris, France: Organisation for Economic Co-operation and Development. Patton, M. Q. 2010. Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press. Pavalam, S. M., M. Jawahar, and F. K. Akorli. 2010. Blended learning at National University of Rwanda. Paper read at the International Conference on Education and Management Technology, Proceedings. PEPFAR (Presidentâs Emergency Plan for AIDS Relief). 2011a. PEPFAR blueprint: Creating an AIDS-free generation. Washington, DC: Presidentâs Emergency Plan for AIDS Relief. PEPFAR. 2011b. FY 2012 country operational plan (COP) guidance. Washington, DC: Office of the U.S. Global AIDS Coordinator and Health Diplomacy. PEPFAR. 2014. PEPFAR 3.0: Controlling the epidemic; delivering on the promise of an AIDS-free generation. Washington, DC: Office of the U.S. Global AIDS Coordinator and Health Diplomacy. PEPFAR. 2019. PEPFARâs five-year strategy. https://www.pepfar.gov/about/strategy/docu- ment/133251.htm (accessed February 21, 2019). Petroze, R. T., A. Nzayisenga, V. Rusanganwa, G. Ntakiyiruta, and F. Calland. 2012. Compre- hensive national analysis of emergency and essential surgical capacity in Rwanda. British Journal of Surgery 99(3):436â443. Phillips, J. M., and A. M. Stalter. 2018. Using systems thinking to advance global health engagement in education and practice. Journal of Continuing Education for Nurses 49(4):154â156. Puri, L., J. Das, M. Pai, P. Agrawal, J. E. Fitzgerald, E. Kelley, S. Kesler, K. Mate, M. Mohanan, A. Okrainec, and R. Aggarwal. 2017. Enhancing quality of medical care in low income and middle income countries through simulation-based initiatives: Recommendations of the Simnovate Global Health Domain Group. BMJ Simulation and Technology Enhanced Learning 3:S15âS22.
RECOMMENDATIONS 249 Ramanadhan, S., S. Kebede, J. Mantopoulos, and E. H. Bradley. 2010. Network-based social capital and capacity-building programs: An example from Ethiopia. Human Resources for Health 8:17. Ripley, M., and S. Jaccard. 2016. The science in adaptive management. The Lab: Market Systems Development for Decent Work. Geneva, Switzerland: International Labour Organization. Samb, B., T. Evans, M. Dybul, R. Atun, J. P. Moatti, S. Nishtar, A. Wright, F. Celletti, J. Hsu, J. Y. Kim, R. Brugha, A. Russell, and C. Etienne. 2009. An assessment of interactions be- tween global health initiatives and country health systems. Lancet 373(9681):2137â2169. Scheffler, R. M., C. H. Herbst, C. Lemiere, and J. Campbell. 2016. Health labor market analyses in low- and middle-income countries: An evidence-based approach. Washington, DC: World Bank. Sewankambo, N., J. K. Tumwine, G. Tomson, C. Obua, F. Bwanga, P. Waiswa, E. Katabira, H. Akuffo, K. E. Persson, and S. Peterson. 2015. Enabling dynamic partnerships through joint degrees between low- and high-income countries for capacity development in global health research: Experience from the Karolinska Institutet/Makerere University partner- ship. PLoS Medicine 12(2):e1001784. Sousa, A., R. M. Scheffler, J. Nyoni, and T. Boerma. 2013. A comprehensive health labour market framework for universal health coverage. Bulletin of the World Health Organiza- tion 91(11):892â894. Stulac, S., A. Binagwaho, N. Tapela, C. Wagner, M. A. Muhimpundu, F. Ngabo, S. Nsanzi- mana, L. Kayonde, J. B. Bigirimana, A. J. lessard, L. lehmann, L. N. Shulman, C. Nutt, P. Drobac, T. Mpunga, and P. E. Farmer. 2015. Capacity building for oncology programmes in sub-Saharan Africa: The Rwanda experience. Lancet Oncology 16(8):e405âe413. Swanson, C., and M. Widmer. 2018. Transforming health education to catalyze a global paradigm shift: Systems thinking, complexity, and design thinking. In Putting systems and complexity sciences into practice: Sharing the experience, edited by J. P. Sturmberg. Springer International. THET (Tropical Health and Education Trust). 2019. Case studies. thet.org/our-impact/case- studies (accessed October 11, 2019). UN (United Nations). 2015. Addis Ababa Action Agenda of the Third International Confer- ence on Financing for Development. Addis Ababa, Ethiopia: United Nations. USAID (United States Agency for International Development). 2017. Discussion note: Co-creation additional help. Washington, DC: United States Agency for International Development. USAID. 2018. Discussion note: Adaptive management. Washington, DC: United States Agency for International Development. Vasan, A., M. Anatole, C. Mezzacappa, B. Hedt-Gauthier, L. Hirschhorn, F. Nkikabahizi, M. Hagenimana, A. Ndayisaba, F. Cyamatare, B. Nzeyimana, P. Droback, and N. Gupta. 2013. Baseline assessment of adult and adolescent primary care delivery in Rwanda: An opportunity for quality improvement. BMC Health Services Research 13:518. WHO (World Health Organization). 2009. Systems thinking for health systems strengthening. Geneva, Switzerland: World Health Organization. WHO. 2010. WISN: Workload indicators of staffing needâuserâs manual. Geneva, Switzer- land: World Health Organization. WHO. 2016. Health workforce requirements for universal health coverage and the Sus- tainable Development Goals. https://apps.who.int/iris/bitstream/handle/10665/250330/ 9789241511407-eng.pdf;jsessionid=E8A436D08D1AC52C7395E8959E612DAE? sequence=1 (accessed April 1, 2020). WHO. 2019a. Human resources for health (HRH) tools and guidelines. https://www.who.int/ hrh/tools/planning/en (accessed December 19, 2019).
250 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA WHO. 2019b. Twinning partnerships for improvement: Tools and resources. https://www.who. int/servicedeliverysafety/twinning-partnerships/tools/en (accessed October 11, 2019). Woolcock, M. 2018. Enhancing public health outcomes in developing countries: From good policies and best practices to better implementation. Scandinavian Journal of Public Health 46(22 Suppl):10â18.