8

Recommendations

This report spans from the Human Resources for Health (HRH) Program’s vision of increased institutional capacity to produce highly qualified health workers, to this evaluation’s specific charge to assess the Program’s effects on outcomes for people living with HIV (PLHIV). It thus reflects a balancing act that is widespread in the global health landscape. Most countries have both broad health system needs, including health workforce needs, and deep unmet needs with respect to specific disease burdens.

In the context of President’s Emergency Plan for AIDS Relief (PEPFAR) funding, this balancing act takes the form of decision making about how to meet HIV-specific needs, which is the core of PEPFAR’s mission, in the context of health systems that lack sufficient capacity to meet either HIV-specific or broader population health needs. Over time and across different investments, PEPFAR has exhibited both relatively siloed (or “vertical”) approaches, focused on HIV-specific efforts, which might have spillover effects on the health system as a whole, and broader (or “horizontal”) approaches, focused on systems-based efforts that also meet the needs of PLHIV who are served within that system (PEPFAR, 2011a,b, 2014, 2019; Samb et al., 2009). The vision for the HRH Program framed its impact through the latter horizontal perspective. The charge of this evaluation framed impact from a more vertical perspective.

OVERARCHING EVALUATION CONCLUSIONS

The HRH Program had many successes with respect to its goal to expand the quantity and quality of the health workforce in Rwanda, with

particular examples in the value it added to the quality of health professional education and training for different cadres of health professionals, especially in nursing, and improvements in the overall preparation and motivation of new professionals entering the workforce. The Program was seen by those in both health professional education and health service delivery roles as contributing to improving the quality of care for all Rwandans, including PLHIV, through direct and indirect pathways. Some of these successes resulted from the original design, as intended, whereas others were more unexpected, resulting from adaptations that were made in response to operational realities or challenges encountered. Given the complexity of the health system and HRH within that system, these successes were accompanied by challenges that offer opportunities for learning.

There were mixed results with respect to the ambitious goals of the HRH Program to increase institutional capacity for health professional education, resulting from a truncated time frame, operational challenges in its implementation, and insufficient design and planning around the intended mechanisms of change and the complexity of structural changes needed to achieve improvements in health professional education.

With respect to outcomes for PLHIV, it was much more difficult to assess the Program’s effects. There are indications that it contributed to improved quality of care, and the evaluation found no indications to suggest that the allocation of funds to this Program undermined Rwanda’s continuing progress in the HIV response. However, the evaluation was constrained in being able to fully assess the Program’s contribution to impact on PLHIV-specific health outcomes.

The HRH Program represented what was, at the time, a relatively uncommon (although not unique) donor-funded approach to strengthening HRH capacity in low- and middle-income countries by focusing on a large investment, at a foundational level, for capacity building in institutions for health professional education. For PEPFAR, it also represented a departure from the usual operational model between funder and government. When seen in light of this committee’s charge, the exceptional nature of the Program ended up being a missed opportunity to learn from what could have been a more intentionally designed approach that could have added new insights to the knowledge base not only for how to strengthen HRH capacity, but also, more broadly, for how to navigate the balancing act between disease-specific priorities and broader health system needs.

IMPLICATIONS FOR HIV AND HUMAN RESOURCES FOR HEALTH PROGRAMMING

Lessons learned from the experience of this Program point to broader implications for how to undertake this balancing act in future HRH pro-

gramming. For HIV in particular, as Rwanda and other countries make laudable progress toward targets related to epidemic control and improved coverage of antiretroviral therapies, more PLHIV are living longer, with health needs that increasingly lie at the intersections of managing HIV and its complications over time, managing comorbid conditions, and attending to quality of life. Increasingly, supporting the needs of PLHIV and sustaining comprehensive control of the HIV epidemic rely on the foundation of a strong and comprehensive health system. It is therefore in the interests of PEPFAR and other disease-specific funders to contribute to HRH and other health systems efforts that, to be most effective, are not designed around a specific disease, but meanwhile need not interfere with progress related to that disease. Yet, it is also reasonable for funders to expect that those investments in broader efforts will have effects that contribute, albeit not exclusively, to disease-focused outcome goals. Programs or initiatives can be designed in ways that optimize and monitor that disease-specific effect, without interfering with the broader systems effects. Such investments have the greatest potential to yield sustainable results when aligned with long-term systems strengthening strategies and coordinated with concurrent investments from governments and other donors.

RECOMMENDATIONS

The committee was tasked with making recommendations “to inform future HRH investments that support PLHIV and to advance PEPFAR’s mission.” The committee’s recommendations are therefore framed with the intention of helping to make the balancing act between disease specificity and systems strengthening more achievable, and more measurable, for future investments in HRH. The recommendations presented here posit that a future role for PEPFAR, or any other funder with a disease-specific mandate, would be to make investments in HRH that take a more explicit, intentional, and longer-term “diagonal” approach that focuses on finding the intersections between the vertical and the horizontal.

These recommendations provide a framework for how future efforts could build on the lessons learned from this Program, both reinforcing its successes and making accommodations to address its challenges, with a design that more fully accounts for needs and feasibility at baseline and ensures more of the needed information will be available to learn about the effects on the system and the effects on the response to HIV. Although the primary audiences for this evaluation’s findings and conclusions are PEPFAR and the government of Rwanda, the committee hopes that the conclusions and recommendations will inform other funders and other institutions contributing to strengthening the health workforce, such as medical

and health professional training institutions, professional societies, patient advocacy groups, and other civil society organizations. Furthermore, the hope is that the lessons learned from this Program and the committee’s recommendations might inform not only future efforts in Rwanda but also elsewhere in the region.

Recognizing the inherently complex interactions among many factors, stakeholders, institutions, and sectors when it comes to HRH, these recommendations emphasize the following aspects of an integrated approach to improving HRH through health professional education:

• Codesign among relevant stakeholders;
• A systems approach with adequate needs assessment and planning time;
• Operational planning that emphasizes adaptive management;
• Selection of a tailored set of components from models for improving health professional education;
• A prospective and multifaceted approach to monitoring, evaluation, and learning; and
• In all of the above, an explicit connection from disease-specific elements to interrelated broader systems elements.

Program Codesign

To ensure a robust and feasible programmatic design, an effective approach is to employ a collaborative design process at the level of key decision makers representing funders and government leadership across relevant sectors, while including implementers and beneficiaries (in the case of health professional education, faculty, trainees, the public and private health systems that will employ program graduates, and ultimately, patients). When embarking on a health systems strengthening program, it is important to engage all relevant government entities beyond the Ministry of Health, including the Ministries of Finance, Labor, and Civil Service, and other government bodies, to ensure the national budget and policies support the programmatic objectives. This inclusive, multilayered design process can ensure that the effort responds to the need, reflects contextual realities, and has the potential to be executed effectively.

The 2008 Accra Agenda for Action and the 2011 Busan Partnership for Effective Development and Co-operation both endorse collaborative approaches throughout the program cycle. The Accra Agenda noted that partnership efforts for development should “fully harness the energy, skills, and experience of all development actors—bilateral and multilateral donors, global funds, CSOs [civil society organizations], and the private sector” as only then are development efforts most effective (OECD, 2008). The Busan Partnership elaborated, with development partners pledging to encourage parliaments and local governments to strengthen their role “in the oversight of development processes” and to “[f]urther support local governments to enable them to assume more fully their roles above and beyond service delivery, enhancing participation and accountability at the subnational levels” (Fourth High Level Forum on Aid Effectiveness, 2011). Similarly, Busan acknowledged civil society organizations’ key roles in “shaping development policies and partnerships, and in overseeing their implementation” and urged development partners to create an enabling environment that allows these organizations to maximize their contributions (Fourth High Level Forum on Aid Effectiveness, 2011). In addition, Busan strongly encouraged the expansion of South-to-South partnerships and “triangular cooperation,” noting that these “have the potential to transform developing countries’ policies and approaches to service delivery by bringing effective, locally owned solutions that are appropriate to country contexts” (Fourth High Level Forum on Aid Effectiveness, 2011).

Busan also supported “participation of the private sector in the design and implementation of development policies and strategies” and encouraged “representatives of the public and private sectors and related organizations to play an active role in exploring how to advance both development and business outcomes so that they are mutually reinforcing” (Fourth High Level Forum on Aid Effectiveness, 2011). When it comes to strengthening

___________________

¹ Later recommendations that actions be taken by “designers of programs to strengthen human resources for health” refer to this group of diverse stakeholders: funders (e.g., multilateral donors, bilateral external government donors, philanthropic donors, national governments, and private payors), program administrators (e.g., government leadership in relevant sectors and nongovernmental leadership), implementers (e.g., program managers and health professional education program leaders), regulatory bodies (e.g., accreditors and professional councils), and those who will use or benefit from the funded activities (e.g., faculty, trainees, public or private health systems, and patients, including people living with HIV).

HRH, the role of the private sector is particularly salient, given the increasing presence in low- and middle-income countries of both private-sector health care delivery and private medical, nursing, and other health professional schools.

As one example of a version of codesign, the United States Agency for International Development (USAID) has begun applying cocreation methods to the design of its funded activities (USAID, 2017). During this process, diverse stakeholders (such as implementing partners, host-country governments, private-sector representatives, and local organizations and experts) are invited to a participatory workshop, in which they contribute to the design and structure of an upcoming activity. The aims of this approach are to enhance local ownership of the activity and to increase the probability that the activity will achieve its intended results, because its design is informed by diverse stakeholders.

For complex and long-term initiatives such as the HRH Program, which sought to strengthen HRH through building health professional education, including a process whereby different stakeholders formally articulate their commitments over different time spans can reduce longer-term risk. These risks include the donor’s ceasing funding of the program, as occurred with the HRH Program, or the host government’s not following through on commitments to recruit and retain the health care workers who have graduated. These eventualities can always occur, but any steps to minimize this risk should be considered.

Design with a Complex Systems Thinking Lens

The World Health Organization (WHO) has described the application of systems thinking to health systems strengthening, recognizing that “every intervention, from the simplest to the most complex, has an effect on the overall system, and the overall system has an effect on every intervention” (WHO, 2009). The interactive, nonlinear, and often unpredictable relationship among parts of a health system warrants approaches to strengthening HRH that are designed around its inherent complexity.

Applying complex systems thinking can change how program designers conceive of the challenges in the health system, the questions they ask

about how to improve the system, and their understanding of the environment that either supports or hinders improvement (Swanson and Widmer, 2018). A systems approach to strengthening the health system should also recognize that the health system is nested within a larger government, and the health workforce is nested within regional health labor markets. This necessitates multisectoral collaboration and coordination across the health, education, labor, and finance sectors and among governmental and nongovernmental institutions.

Other features of the system also need to be recognized as additional layers of complexity, such as cultural considerations and concurrent shifts in standards and norms. The HRH Program, for example, accompanied a transition in Rwanda from French- to English-language health professional education and a transition in the region away from the Francophone model of medical education. Global health movements should take such local transitions into account in the design and implementation of programs that aim to affect systemwide changes. Similarly, the quality of primary and secondary education need to be considered to ensure that students progressing to health professional education programs have adequate foundational knowledge to be successful. Therefore, complex systems thinking needs to be an integral part of the codesign process described under the first recommendation.

Time Frame Considerations for Strengthening HRH

Effective systems strengthening for HRH, especially in relation to faculty production and building university infrastructure, requires decades. Investments in health professional education and training should be expected to take many years to yield effects, given the time required for students to complete training programs, for trainees to make their way as fully qualified health professionals into the service delivery system, and for new or newly upgraded faculty to produce ongoing cohorts of providers.

There are examples of longer-term projects that have yielded systems changes, such as a nearly 20-year collaboration between Uganda and the Swedish International Development Cooperation Agency, executed via a partnership between Makerere University and the Karolinska Institute of Sweden, that has produced a joint Ph.D. program and a wealth of research products contributing to policy and program design (Sewankambo et al., 2015). This example indicates that the duration required to build institutional capacity is on the order of decades, whereas the HRH Program, as laid out in the 2011 proposal, was planned for 9 years and faced a significant drop in funding following the cessation of PEPFAR investment after 5 years, which had implications for addressing sustainability, institutionalization, and other issues that surfaced in the midterm review.

Although it may be more feasible to invest in shorter, concrete projects with observable benefits, such as infrastructure and commodities distribution, doing so may overlook investments that require more time but have more widespread effects. There is value in being strategic about the investments, “with an eye toward making long-term investments in global health instead of focusing on short-term expenditures” (NASEM, 2017). With respect to HIV/AIDS, these investments need to be made in response to the anticipated future of the epidemic, strengthening a health system to be able to care for an aging PLHIV community with an increasing burden of comorbid conditions. Building systems thinking approaches into health professional education is one mechanism for moving toward a “culture of health,” positioning clinical staff as leaders (Phillips and Stalter, 2018).

Funding Considerations for Strengthening HRH

Because systems strengthening takes decades and the HRH pipeline spans multiple stages—from recruitment of students to preservice training through specialization and continuing professional development of the workforce, to longer-term issues such as health worker motivation and retention—designers of HRH programs should articulate and work toward comprehensive long-term goals and outcomes. This will require local governments and funders to collaboratively develop funding strategies that can outlast political terms and agendas, as well as typical donor funding cycles, and enable a built-in transition to sustained country-led ownership and financing.

Governments should focus on assembling diversified funding sources and partners for HRH programming, recognizing that some donors have adopted a more broad-based approach to development assistance, while others take an approach focused on specific outcomes or interventions. Reliance on a single donor can jeopardize broader goals for sustained systems change if donor priorities shift over time. Governments are also positioned to identify and bring together public- and private-sector actors with vested interests in national HRH goals in order to coordinate financing initiatives. When aligned with a comprehensive HRH strategy based on a decades-long time frame, a diversified approach would enable governments to assemble a portfolio of shorter-term investments and programs with disease-specific, activity-specific, or time-bound parameters that are all coordinated toward achieving the defined overall strategy.

Donors should accommodate this longer-term, coordinated funding approach by being open to the explicit integration of their investments into a broader strategy that catalyzes sustainable change. This can be achieved through participation in a codesign process that encompasses not only a specific donor-funded program, but also how it ties into the broader national

HRH strategy and building sustainable local capacity. This coordinated funding and design approach would also provide an opportunity to attend to the extent to which donor funding adheres to the Addis Ababa Action Agenda. This agenda calls for shifting the balance toward domestic funding and away from donor funds that ultimately go to external parties, as was the case in the design of the HRH Program, where the majority of the budget was dedicated to U.S. institutions for faculty contracts (UN, 2015).

Once committed to contributing to a longer-term approach, it is important for donors to recognize that subsequent shifts in their priorities that affect funding midcourse, as occurred with PEPFAR’s funding for the HRH Program, will have broader consequences. Mutual expectations for a transparent process around potential revisions in funding and programming should be clearly outlined from the outset, and if anticipated shifts in funding arise, they should be considered and planned collaboratively and in relation to the broader coordinated strategy.

Donors should also, to the extent feasible, offer greater flexibility in shaping and adapting program budgets and processes to more readily accommodate a role in a broader funding strategy. U.S. government funders should build on USAID practices that have begun to address program funding constraints. Some of that agency’s procurement processes have incorporated an “inception period” to flexibly yet systematically revisit initial objectives, targets, and outputs, instead of requiring program proposals to commit rigidly to achievements and outputs within an established time frame. Additionally, USAID has allowed “windows of opportunity” to dedicate a portion of project budgets for adapting strategies and development programming based on changing conditions and enhanced understanding of needs (Brinkerhoff et al., 2018).

Systematic Approach in the Context of the Labor Market

Although programs and policies that focus on boosting health professional education are key to addressing the health workforce shortage, adopting a labor market lens can both leverage health professional education investments and redress factors that undermine the capacity of the health workforce (Evans et al., 2016; Sousa et al., 2013). Health service delivery is highly labor intensive and requires the appropriate number and mix of trained and motivated health professionals to provide high-quality health services. A health labor market is a dynamic relationship between the supply of health workers and the demand for health workers (McPake et al., 2013; Scheffler et al., 2016). When supply and demand are placed in the context of national goals for access and coverage, the need for health services and the health needs of the whole population come into play as a third important and dynamic factor. It is important to identify the optimal

mix of cadres and their geographic distribution in response to different needs and goals, such as addressing the HIV epidemic, having a broader impact on health, and teaching and workforce professionalization. This mix will need to be adjusted as goals and needs change over time.

In a health system, prevailing demand-side forces, especially in the absence of universal health coverage, can skew the supply of health workers away from population health needs; the private sector can fuel such imbalances with competing wages, working conditions, and regulations for health professionals (Evans et al., 2016; Sousa et al., 2013). Governments should develop production policies in the education sector in tandem with policies that address how the new supply of health workers can be absorbed into the labor market (Sousa et al., 2013). Furthermore, career progression, which differs according to cadres of health workers, needs awareness and consideration to ensure that existing health workers have opportunities for growth that can be absorbed by the health system and that sufficient numbers of health workers are being produced to take on the duties of those who have advanced.

Governments also have a role in regulating the private sector to ensure the quality of training, given the rise of private health training institutions, and the equitable distribution of health care workers (Evans et al., 2016; Sousa et al., 2013). In turn, the private sector can drive innovative public–private models for financing growth in health worker education in response to market opportunities and other areas where governments are unable to respond (Evans et al., 2016).

Planning and Adaptive Management

Because it relies on complex systems change, strengthening HRH requires not only visionary leadership and effective program activities, but also an appropriate and sufficient management structure to shepherd a program through an inevitably multifaceted and complicated implementation process. The experience of the HRH Program points to the need for strong management structures and processes that allow for continuous learning and improvement as a means of moving toward the defined programmatic goals, even in the face of policy pivots such as PEPFAR’s shift from its 2.0 to 3.0 strategy, which resulted in a determination to cease funding.

Often, the reality of implementation after program design necessitates changes to the proposed mechanisms to ensure progress toward the original vision and goals. This is particularly true in projects aimed at making changes in complex systems, such as human resources and the health system. Adaptive management, or problem-driven iterative adaptation, enables an intentional approach to making decisions and adjustments to programmatic activities in response to emerging information, unintended consequences, unexpected challenges, or changes that take place in the context in which the program is being implemented (USAID, 2018; Woolcock, 2018).

An adaptive approach can begin as early as the design stage—with a focus on adapting best practices and external solutions to create a “best fit” version for the context (Woolcock, 2018)—and continue throughout the life cycle of the program. The key principles of adaptive management include

• Reframing project design and implementation from a linear project trajectory to a more flexible sequencing;
• Building in management structures that are capable of being flexible;
• Creating explicit, periodic windows for assessing and reconsidering implementation decisions; and
• Linking adaptation to learning by creating a feedback loop between decision making and real-time information on the program’s progress and its struggles (Brinkerhoff et al., 2018).

The intent is not to find the best action, but rather to identify which of the available and feasible options move closer to realizing the systems-level changes the program envisioned (Ripley and Jaccard, 2016).

Adaptive management is also underpinned by robust and continuous data collection for real-time information and decision making (Brinkerhoff et al., 2018). Processes for reviewing the data generated and documenting the rationale behind decisions taken should also be institutionalized. As discussed under Recommendation 6, there is a need for HRH programs to include a comprehensive approach to monitoring, evaluation, and learning as an integrated responsibility not only for designated staff but also for other technical and operational staff.

Adaptive management is only possible if the people implementing the program are open to critically engaging with and learning from rapidly collected and analyzed data to make programmatic adjustments. This requires a culture of improvement at the programmatic and organizational levels. Debating the decisions being made, including consideration of effects on other aspects of the system, both within and beyond the program, can strengthen the quality of decisions while fostering a culture of learning and improvement (Allana, 2016). These conversations need to happen not

just within the program, but in consultation with a department within a Ministry of Health that holds the broader HRH strategy, comprising not only administrators and managers, but also economists and finance experts, political scientists, and other scientific and technical personnel. Although there are various potential modalities to achieve this, it is the coordinated consultation across functions that is essential for effective adaptive management. Simultaneously, defined roles and responsibilities around decision making are necessary to move beyond debate and data review and into action. One lesson from the HRH Program is that it could have benefited from a more clearly defined and more robustly supported decision-making structure to facilitate timely and data-driven adaptive management to enable improved implementation.

For adaptive management to succeed in the context of reliance on multiple funding sources, external donors and governments that fund HRH programs need to embrace the approach of expecting well-executed implementation to include clarity of rationale and specificity of design at the outset, and learning-based adjustments as implementation proceeds. Program assessment and accountability mechanisms should be based partly on the process of implementation and on achieving reasonable progress toward goals, in the context of the realities encountered during implementation, rather than basing accountability narrowly on adherence to the original design. Where incremental changes are made as part of an adaptive management plan, sufficient time needs to be given to allow the changes to yield improvements. Donors are increasingly recognizing this need for adaptability to make effective investments, as evidenced by initiatives such as the World Bank’s Global Delivery Initiative and the Doing Development Differently manifesto (ODI, 2016; USAID, 2018).

Models for Improving Health Professional Education

Varied models and approaches have been developed for improving health professional education (including twinning, interprofessional education, and technology-enabled education) and have been established as options that could be considered in the design phase of a program, depending on the program’s needs and goals and the country context.

Crosscutting Considerations Across Models

Selection of a health professional education model should be based on the goals and vision of the program and the needs of the health workforce. There should also be a focus on future institutionalization through:

• Ensuring structures are in place to support faculty in the longer term (career progression, time to commit to the program);
• Ensuring faculty have time built into their schedules to commit to additional health professional education development;
• Ensuring time and funding are provided to focus on accreditation, research skills, and other elements that are not a direct transfer of teaching skills;
• Emphasizing longer-term institutional partnerships; and
• Teaching models that require less time from faculty (blended learning).

Technology is used in a wide variety of ways for health professional education throughout low- and middle-income countries. E-learning, distance learning, web-based training, and m-learning (using mobile devices) enable remote education and training interactions and can be key strategies for reaching the workforce in resource-limited environments (Ballew et al., 2013; Buabeng-Andoh, 2018; Murebwayire et al., 2015). Simulation-based tools are another use of technology to provide training and assessment to improve quality of care and reduce medical errors (Puri et al., 2017).

Blended learning—combining technology tools with traditional face-to-face teaching approaches (Pavalam et al., 2010)—has been put forward as a way to create a more learner-centered environment, build student engagement, and relieve overcrowding (Frantz et al., 2011). It is a growing model owing to its effective pedagogy, cost-effectiveness, and increased faculty time for student mentorship (Geoffrey, 2014). In Rwanda, students and staff have reported positive attitudes toward the integration of blending learning (Pavalam et al., 2010), and the implementation of a blended e-learning approach has been used for nursing education to make access to training and classes more feasible for those living in rural areas (Murebwayire et al., 2015). When blended learning is adopted, however, it is important to consider the potential implications and additional needs it can create, such as requiring additional training to understand how to implement blended learning, designing appropriate curricula, and the need for adequate access to reliable technology (Geoffrey, 2014).

Optimizing the Use of Twinning

When twinning models are used as part of efforts to improve health professional education, this evaluation offers several lessons for potential improvements to the process and its effectiveness, depending on the time frame, the goals, and the desired type of skills transfer. Under the HRH Program, the objective was to transfer teaching and clinical skills to University of Rwanda faculty.

There is wide variation in the use of terminology to describe the types of activities that could fall under the concept of twinning. There are a few categorical distinctions that can be mapped to different strengths and to different considerations for implementation.

Effective institutional twinning needs clear objectives that can be operationalized in the context of available funding and country environment. Broadly speaking, however, programs should focus on longer-term institutional partnerships and should consider different models of twinning and adapt to what best fits the context and need, including forming South-to-South partnerships, when possible. Institutional twinning partnerships require clearly communicating objectives, aligning interests across institu-

___________________

² The organization’s original name was Ensemble pour une solidarité thérapeutique hospitalière en réseau (ESTHER), or Network for Therapeutic Solidarity in Hospitals against AIDS, as it was known in English.

tional partners, and taking into account the inherent cultural and historical dynamics of involved institutions.

All types of peer-to-peer support should be considered as partnerships. However, there is a need to clearly define roles and relationships within these partnerships to enable effective and efficient outcomes. For example, roles could include exchanging knowledge while dividing or sharing responsibilities (clinical or teaching duties or shared curriculum development), mentorship (coaching and sharing of experience), training (teaching of new clinical, teaching, or research skills), or mixed roles. These roles should be predefined, shared transparently, and formally agreed upon prior to initiating a twinning relationship. Although emergent twinning may have an advantage in terms of the initial quality of the relationship between twins, mandated twinning, when done well under the umbrella of an institutional partnership, may have the advantage of better delineating expectations about roles and responsibilities before the relationship begins.

It is imperative to consider the inherent cultural and linguistic dynamics involved in any type of individual twinning relationship. In mandated twinning, a matching process is needed to align skills, language, and interests of both twins as closely as possible. This matching can be enhanced by using regional experts or incorporating opportunities for emergent twinning. Additionally, offering similar conditions to the twins with respect to compensation and incentives, where possible, could facilitate greater engagement in twinning relationships. Preparing and coaching twins through their relationships can be a mechanism for building skills such as cross-cultural understanding, communication, and conflict management and resolution.

Operationalizing peer-to-peer twinning support should also consider methods such as mixed distance learning or bidirectional international placements of shorter durations. This could be particularly effective for deans of institutions, or for medical or surgical subspecialists for whom longer-term placements are impractical. Using ratios greater than one to one for the partnering between external and local twins could also be an effective approach to optimize capacity building for health professional education.

develop robust plans for learning, as discussed further in Recommendation 6, will have important and much needed opportunities to contribute to the knowledge base.

Monitoring, Evaluation, and Learning

Although the importance of monitoring, evaluation, and learning in health systems strengthening programming is recognized, operationalizing an effective and adaptive monitoring, evaluation, and learning system is often undervalued and underresourced.

For future investments in HRH, a low-cost but rigorous monitoring, evaluation, and learning plan and system will be most effective if it is included in the design phase and incorporates ongoing mixed-methods monitoring, with pause points for learning throughout the program, and the resources and staff to achieve realistic and actionable learning. Monitoring and evaluation capacity among in-country program managers and implementers should be strengthened to support ongoing monitoring and data use for decision making. The advantages and disadvantages of also using an external third party for evaluation should be weighed and considered as part of the design. In addition to planning for learning about implementation processes and program outcomes, it would be valuable for future efforts to prospectively plan for analysis that would allow program designers, implementers, and others looking to learn from such programs to understand the costs of program implementation and select, plan for, and carry out assessments of return on investment.

Below is a breakdown of monitoring, evaluation, and learning options at different points in a program’s time line.

Design and Start-Up

It is important for the program to draw on a wide base of evidence to increase relevance and effectiveness in the country’s current context. This could be done in a multitude of ways. One is to conduct background research on other approaches, especially models in the region, and how they would need to be adapted for the Rwandan (or other country’s) context,

as well as what the trade-offs and opportunity costs would be in choosing from among different approaches to achieve the desired objectives. In the HRH Program, although the designers referenced other regional twinning programs, there was not a clear process for how to better understand the operational infrastructure and overhead needed to operationalize such programs in the Rwandan context. Another option is to conduct a landscape mapping of existing actors (nongovernmental organizations, government, private sector) working to improve HRH. To deepen this mapping, a network analysis to investigate the social and organizational structures, relationships, and interactions can help designers understand the roles of different actors in an existing system and how they influence each other and work together toward a common goal, such as strengthening HRH (Ramanadhan et al., 2010).

The design phase should include a priori development of indicators to evaluate the program’s effectiveness, efficiency, and outcomes within the overarching structure of the partnership, and a plan, including funding, for analysis and dissemination of findings to improve the knowledge base for similar endeavors in the future. Designers and program administrators should also conduct a baseline assessment that maps the country’s HRH needs to the design of the program, helping to assess how to balance specialized care and primary care, the cadres required to address HIV and to have a broader health impact, and the teaching cadres who can continue building the needed workforce. A baseline assessment also gives the program a starting point for midpoint or endpoint learning. As described below, global tools and local evidence can inform a baseline assessment of this kind.

Several tools are available specifically to assess HIV-specific workforce needs and the intersection between HRH and the response to HIV (MSH

___________________

³ WHO’s collection of HRH tools and guidelines can be accessed at https://www.who.int/hrh/tools/planning/en (accessed December 19, 2019).

⁴ See https://www.who.int/hrh/resources/WISN_Eng_UsersManual.pdf (accessed December 19, 2019).

⁵ See https://www.who.int/hrh/documents/hurst_summary.pdf (accessed December 19, 2019).

and WHO, 2006).⁶ These tools can be used according to their relevance to the specific context, and in combination, as needed, with locally tailored needs assessments.

There are also more generalized and unifying workforce and capacity-building needs. Rwanda faces barriers of unmet need and educational and resource gaps that affect the quality of care its health workforce can provide. For example, significant variability and fundamental gaps have been described in adult and adolescent primary care delivery (Vasan et al., 2013).

An additional, widespread capacity-building and workforce development challenge lies in the availability of information. For example, Nahimana and colleagues (2015) describe limited information on technology scale-up in rural health facilities, and Egziabher and colleagues (2015) describe a lack of data on quality of care for obstetric fistula management. A similar issue exists in emergency medicine, where there are no data on

___________________

⁶ Questionnaires to measure the effect of HIV/AIDS on HRH and tools for planning and developing human resources for HIV/AIDS and other health services can be accessed at https://www.who.int/hrh/tools/tools_planning_hr_hiv-aids.pdf (accessed December 19, 2019).

the effects of training programs on patient-centered outcomes in resource-limited settings (Mbanjumucyo et al., 2017).

These studies provide assessments of need that span a variety of specialties and diseases. While they indicate varied and evolving treatment needs throughout Rwanda as the burden of disease shifts over time, they also reveal overarching commonalities regarding workforce and capacity needs. Future HRH investments would benefit from a more comprehensive and coordinated assessment of these evolving needs, in light of both the common barriers and opportunities and those that are specific to diseases and specialties.

Program Implementation

Throughout the life of a program, it is imperative for an adaptive monitoring system to build on existing government data systems and represent a low burden to those implementing and benefiting from the project. Ensuring monitoring plans move beyond collecting outputs and conceptualize implementation outcomes and mechanisms of change, including creating prospective measurement architecture, is necessary for systematically understanding the potential impact of a program and for implementation to be responsive and reactive to the real-world context. Similarly, it is important to include early process indicators to determine whether the initiative is on the right track and make quick course corrections. To ensure data for decision making are accurate, it is helpful to include data quality assessments to examine the quality of implementers’ monitoring data at various points throughout the program.

Building in time for capacity-building opportunities for implementers can improve on findings from data quality assessments. This could include existing courses on standard health and health systems data (such as the District Health Information System, national health accounts, national AIDS accounts), as well as project- or program-specific capacity building. For an HRH program that uses a twinning model, monitoring the twinning process and interactions and adapting recruitment and onboarding based on this evidence could improve implementation and achievement of results. Finally, including a mapping mechanism that tracks where trainees are placed and what their roles are following the program could facilitate comparison analysis to determine the program’s impact on patient outcomes.

Pause Points for Accountability, Learning, and Adaptation

Adaptive management and implementation require pause points for evaluating or assessing the program to identify opportunities for improvement. A mix of national and international evaluation mechanisms in HRH

and academic programs can provide rich data for informed decision making. Options include hiring an external evaluation team or funding an internal monitoring, evaluation, and learning team to conduct learning at pause points throughout the implementation life cycle. This could include more traditional baseline, midline, and end line assessments that use mixed methods and assess not only what progress has occurred, but also why it has or has not occurred. Designers could also build in special research studies between baseline and midline, midline and end line, or after a program’s completion. Special studies enable designers and implementers to learn in real time and improve implementation. This could include looking at a particular aspect of the program, such as conducting an assessment of curricula development in year 2 or an ex-post evaluation 5 to 7 years after a program has ended.

If designers are interested in learning about what it takes to establish and maintain an effective HRH system, they could also build in a developmental and/or process evaluation. Process evaluations allow funders to learn in real time, examining outputs and processes of why interventions are or are not working. Process evaluations provide a good platform to teach other designers what they should consider during the design of HRH programs. A developmental evaluation is particularly well suited to stimulating innovative program design in response to dynamic realities, possible desire for program replication, and complex issues and environments (Patton, 2010). Developmental evaluations ask program administrators to work closely with evaluators as partners, sharing data and lessons throughout the program cycle. Another evaluation approach is social network analysis; in addition to being useful for the design phase, as described above, social network analysis has been applied as a tool for evaluating partnerships (Kamya et al., 2016; Kenis, 2017).

Whatever approach is taken, all future HRH programs should avoid repeating this HRH Program’s missed opportunity for greater learning. This can be achieved by ensuring programs are funded with a sufficient amount of design, planning, management, and staffing with the capabilities to work with implementers to monitor, evaluate, and learn what it takes to successfully build, implement, and sustain an effective HRH program.

REFERENCES

Atijosan, O., V. Simms, H. Kuper, D. Rischewski, and C. Lavy. 2009. The orthopaedic needs of children in Rwanda: Results from a national survey and orthopaedic service implications. Journal of Pediatric Orthopaedics 29(8):948–951.

Ballew, P., S. Castro, J. Claus, N. Kittur, L. Brennan, and R. C. Brownson. 2013. Developing web-based training for public health practitioners: What can we learn from a review of five disciplines? Health Education Research 28(2):276–287.

Brinkerhoff, D. W., S. Frazer, and L. McGregor-Mirghani. 2018. Adapting to learn and learning to adapt: Practical insights from international development projects. Research Triangle Park, NC: RTI International.

Buabeng-Andoh, C. 2018. New technology in health education: Nursing students’ application of mobile technology in the classroom in Ghana. Interactive Technology and Smart Education 15(1):46–58.

Egziabher, T. G., N. Eugene, K. Ben, and K. Fredrick. 2015. Obstetric fistula management and predictors of successful closure among women attending a public tertiary hospital in Rwanda: A retrospective review of records. BMC Research Notes 8(1):774.

European ESTHER Alliance Secretariat/GIP ESTHER. 2018. Bringing together our expertise for capacity building and quality health care, edited by E. France. Paris, France: ESTHER.

Evans, T., E. Araujo, C. Herbst, and O. Pannenborg. 2016. Addressing the challenges of health professional education: Opportunities to accelerate progress towards universal health coverage: A special report of the Health Professional Education Advisory Board. Doha, Qatar: World Innovation Summit for Health 2016.

Fourth High Level Forum on Aid Effectiveness. 2011. Busan Partnership for Effective Development and Co-operation: Fourth High Level Forum on Aid Effectiveness, Busan, Republic of Korea, 29 November-1 December 2011. Paris, France: Organisation for Economic Co-operation and Development.

Frantz, J. M., S. Himalowa, W. Karuguti, A. Kumurenzi, D. Mulenga, and M. Sakala. 2011. Challenges and opportunities related to postgraduate evidence-based practice module using blended learning. African Journal of Health Professions Education 3(1):15–18.

Geoffrey, N. M. 2014. Challenges of implementing quality assurance systems in blended learning in Uganda: The need for an assessment framework. Huria Journal 18.

Hurst, K. 2002. Selecting and applying methods for estimating the size and mix of nursing teams. Leeds, UK: Nuffield Institute for Health.

Kamya, C., J. Shearer, G. Asiimwe, E. Carnahan, N. Salisbury, P. Waiswa, J. Brinkerhoff, and D. Hozumi. 2016. Evaluating global health partnerships: A case study of a Gavi HPV-vaccine application process in Uganda. International Journal of Health Policy and Management 6(6):327–338.

Kenis, P. I. 2017. Passed the age of puberty: Organizational networks as a way to get things done in the health field comment on “Evaluating global health partnerships: A case study of a Gavi HPV vaccine application process in Uganda.” International Journal of Health Policy and Management 6(11):677–679.

Marshall, S., D. Edidin, V. Sharma, G. Ogle, V. C. Arena, and T. Orchard. 2013. Current clinical status, glucose control, and complication rates of children and youth with type 1 diabetes in Rwanda. Pediatric Diabetes 14(3):217–226.

Martin, A. N., K. M. Kaneza, A. Kulkarni, P. Mugenzi, R. Ghebre, D. Ntirushwa, A. M. Ilbawi, L. E. Pace, and A. Costas-Chavarri. 2019. Cancer control at the district hospital level in sub-Saharan Africa: An educational and resource needs assessment of general practitioners. Journal of Global Oncology 2019(5):1–8.

Mbanjumucyo, G., M. A. Barry, N. Karim, and A. C. Levine. 2017. Impact of implementation of emergency medicine training on emergency department mortality in Kigali, Rwanda: An interrupted time-series quasi-experimental design. Annals of Emergency Medicine 70(4 Suppl 1):S78.

McPake, B., A. Maeda, E. C. Araujo, C. Lemiere, A. El Maghraby, and G. Cometto. 2013. Why do health labour market forces matter? Bulletin of the World Health Organization 91(11):841–846.

MSH (Management Sciences for Health) and WHO (World Health Organization). 2006. Tools for planning and developing human resources for HIV/AIDS and other health services. Medford, MA: Management Sciences for Health.

Muhirwa, E., C. Habiyakare, B. Hedt-Gauthier, J. Odhiambo, R. Maine, N. Gupta, G. Toma, T. Nkurunziza, T. Mpunga, J. Mukankusi, and R. Riviello. 2016. Non-obstetric surgical care at three rural district hospitals in Rwanda: More human capacity and surgical equipment may increase operative care. World Journal of Surgery 40(9):2109–2116.

Murebwayire, M., H. Ewing, and I. Biroli. 2015. Use of e-learning to advance nursing education in Rwanda. Rwanda Journal 2(2).

Nahimana, E., M. Ngendahayo, H. Magge, J. Odihiambo, C. Amoroso, E. Muhirwa, J. N. Uwilingiyemungu, F. Nkikabahizi, R. Habimana, and B. Hedt-Gauthier. 2015. Bubble CPAP to support preterm infants in rural Rwanda: A retrospective cohort study. BMC Pediatrics 15(1):135.

NASEM (National Academies of Sciences, Engineering, and Medicine). 2017. Global health and the future role of the United States. Washington, DC: The National Academies Press.

Ntakiyiruta, G., E. G. Wong, M. C. Rousseau, L. Ruhungande, A. L. Kushner, A. S. Liberman, K. Khwaja, M. Dakermandji, M. Wilson, T. Razek, P. Kyamanywa, and D. L. Deckelbaum. 2016. Trauma care and referral patterns in Rwanda: Implications for trauma system development. Canadian Journal of Surgery 59(1):35–41.

ODI (Overseas Development Institute). 2016. Doing Development Differently: Who we are, what we’re doing, what we’re learning. London, UK: Overseas Development Institute.

OECD (Organisation for Economic Co-operation and Development). 2008. Paris Declaration on Aid Effectiveness (2005) and the Accra Agenda for Action (2008). Paris, France: Organisation for Economic Co-operation and Development.

Patton, M. Q. 2010. Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press.

Pavalam, S. M., M. Jawahar, and F. K. Akorli. 2010. Blended learning at National University of Rwanda. Paper read at the International Conference on Education and Management Technology, Proceedings.

PEPFAR (President’s Emergency Plan for AIDS Relief). 2011a. PEPFAR blueprint: Creating an AIDS-free generation. Washington, DC: President’s Emergency Plan for AIDS Relief.

PEPFAR. 2011b. FY 2012 country operational plan (COP) guidance. Washington, DC: Office of the U.S. Global AIDS Coordinator and Health Diplomacy.

PEPFAR. 2014. PEPFAR 3.0: Controlling the epidemic; delivering on the promise of an AIDS-free generation. Washington, DC: Office of the U.S. Global AIDS Coordinator and Health Diplomacy.

PEPFAR. 2019. PEPFAR’s five-year strategy. https://www.pepfar.gov/about/strategy/document/133251.htm (accessed February 21, 2019).

Petroze, R. T., A. Nzayisenga, V. Rusanganwa, G. Ntakiyiruta, and F. Calland. 2012. Comprehensive national analysis of emergency and essential surgical capacity in Rwanda. British Journal of Surgery 99(3):436–443.

Phillips, J. M., and A. M. Stalter. 2018. Using systems thinking to advance global health engagement in education and practice. Journal of Continuing Education for Nurses 49(4):154–156.

Puri, L., J. Das, M. Pai, P. Agrawal, J. E. Fitzgerald, E. Kelley, S. Kesler, K. Mate, M. Mohanan, A. Okrainec, and R. Aggarwal. 2017. Enhancing quality of medical care in low income and middle income countries through simulation-based initiatives: Recommendations of the Simnovate Global Health Domain Group. BMJ Simulation and Technology Enhanced Learning 3:S15–S22.

Ramanadhan, S., S. Kebede, J. Mantopoulos, and E. H. Bradley. 2010. Network-based social capital and capacity-building programs: An example from Ethiopia. Human Resources for Health 8:17.

Ripley, M., and S. Jaccard. 2016. The science in adaptive management. The Lab: Market Systems Development for Decent Work. Geneva, Switzerland: International Labour Organization.

Samb, B., T. Evans, M. Dybul, R. Atun, J. P. Moatti, S. Nishtar, A. Wright, F. Celletti, J. Hsu, J. Y. Kim, R. Brugha, A. Russell, and C. Etienne. 2009. An assessment of interactions between global health initiatives and country health systems. Lancet 373(9681):2137–2169.

Scheffler, R. M., C. H. Herbst, C. Lemiere, and J. Campbell. 2016. Health labor market analyses in low- and middle-income countries: An evidence-based approach. Washington, DC: World Bank.

Sewankambo, N., J. K. Tumwine, G. Tomson, C. Obua, F. Bwanga, P. Waiswa, E. Katabira, H. Akuffo, K. E. Persson, and S. Peterson. 2015. Enabling dynamic partnerships through joint degrees between low- and high-income countries for capacity development in global health research: Experience from the Karolinska Institutet/Makerere University partnership. PLoS Medicine 12(2):e1001784.

Sousa, A., R. M. Scheffler, J. Nyoni, and T. Boerma. 2013. A comprehensive health labour market framework for universal health coverage. Bulletin of the World Health Organization 91(11):892–894.

Stulac, S., A. Binagwaho, N. Tapela, C. Wagner, M. A. Muhimpundu, F. Ngabo, S. Nsanzimana, L. Kayonde, J. B. Bigirimana, A. J. lessard, L. lehmann, L. N. Shulman, C. Nutt, P. Drobac, T. Mpunga, and P. E. Farmer. 2015. Capacity building for oncology programmes in sub-Saharan Africa: The Rwanda experience. Lancet Oncology 16(8):e405–e413.

Swanson, C., and M. Widmer. 2018. Transforming health education to catalyze a global paradigm shift: Systems thinking, complexity, and design thinking. In Putting systems and complexity sciences into practice: Sharing the experience, edited by J. P. Sturmberg. Springer International.

THET (Tropical Health and Education Trust). 2019. Case studies. thet.org/our-impact/case-studies (accessed October 11, 2019).

UN (United Nations). 2015. Addis Ababa Action Agenda of the Third International Conference on Financing for Development. Addis Ababa, Ethiopia: United Nations.

USAID (United States Agency for International Development). 2017. Discussion note: Co-creation additional help. Washington, DC: United States Agency for International Development.

USAID. 2018. Discussion note: Adaptive management. Washington, DC: United States Agency for International Development.

Vasan, A., M. Anatole, C. Mezzacappa, B. Hedt-Gauthier, L. Hirschhorn, F. Nkikabahizi, M. Hagenimana, A. Ndayisaba, F. Cyamatare, B. Nzeyimana, P. Droback, and N. Gupta. 2013. Baseline assessment of adult and adolescent primary care delivery in Rwanda: An opportunity for quality improvement. BMC Health Services Research 13:518.

WHO (World Health Organization). 2009. Systems thinking for health systems strengthening. Geneva, Switzerland: World Health Organization.

WHO. 2010. WISN: Workload indicators of staffing need—user’s manual. Geneva, Switzerland: World Health Organization.

WHO. 2016. Health workforce requirements for universal health coverage and the Sustainable Development Goals. https://apps.who.int/iris/bitstream/handle/10665/250330/9789241511407-eng.pdf;jsessionid=E8A436D08D1AC52C7395E8959E612DAE?sequence=1 (accessed April 1, 2020).

WHO. 2019a. Human resources for health (HRH) tools and guidelines. https://www.who.int/hrh/tools/planning/en (accessed December 19, 2019).

WHO. 2019b. Twinning partnerships for improvement: Tools and resources. https://www.who.int/servicedeliverysafety/twinning-partnerships/tools/en (accessed October 11, 2019).

Woolcock, M. 2018. Enhancing public health outcomes in developing countries: From good policies and best practices to better implementation. Scandinavian Journal of Public Health 46(22 Suppl):10–18.