Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
2 Evaluation Design, Methods, and Limitations This chapter details the evaluationâs operational design and methodol- ogy, building on the discussion of the theoretical framework in Chapter 1 and followed by a discussion of the limitations encountered.1 DESIGN To address the Statement of Task, the evaluation applied a retrospective, concurrent mixed-methods design with embedded in-depth examinations and contribution analysis. The committee approach focused on the Human Resources for Health (HRH) Programâs potential contributions to observed outcomes by understanding how the Program and its components were implemented and by examining the contextual factors that may have en- hanced, moderated, or otherwise influenced outcomes (Moore et al., 2014). Mixed-methods designs provide the flexibility to capture trends regard- ing what results have occurred, while enabling a deeper understanding of how gains were achieved and why change has (or has not) happened (Cre- swell and Plano Clark, 2007). Such designs also provide insight into how different populations might have experienced the intervention. Drawing on multiple data sources and approaches, this evaluation yields an understand- 1â The protocol for this evaluation underwent a scientific and technical review by the U.S. Centers for Disease Control and Prevention, Office of the Associate Director of Science, and was approved on April 25, 2019. Ethical approvals for this evaluation were also provided by the Rwanda National Ethics Committee (April 3, 2019) and the U.S. National Academiesâ Committee to Review Studies on Human Subjects (April 23, 2019). 37
38 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA ing of both breadth (via quantitative data) and depth (via qualitative data). Using mixed methods and drawing on data across diverse data sources is of critical importance when evaluating complex interventions for which the pathway between activities and outcomes is nonlinear (Creswell and Plano Clark, 2007; Patton, 1987). This evaluation draws from the following types of data: â¢ Literature and document review â¢ In-depth examinations of the University of Rwanda and a facility microsystem â¢ Qualitative interview data â¢ Quantitative secondary data Figure 2-1 illustrates the link between the evaluation objectives and the design elements (e.g., document review, in-depth examinations, qualitative interviews, and secondary quantitative data) used to address the objectives. The lower part of the figure shows that although the design was concurrent, some elements occurred sequentially. The document review preceded the con- current qualitative interviews, in-depth examinations, and secondary quanti- tative data. Integrated interpretation was the final phase. Integrated analysis and interpretation occurred during July, August, and September 2019. The data sources had complementary uses for the evaluation. The document review drew on policy and program documents, reports, and published literature. It situated the HRH Program within the broader HRH and HIV context in Rwanda and across the globe to understand the context 2018 2019 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Literature and Document Review Qualitative Interviews In-depth Examinations Secondary Quantitative Data Collection Integrated Analysis and Interpretation FIGURE 2-1 Evaluation process.
EVALUATION DESIGN, METHODS, AND LIMITATIONS 39 and landscape in which the Program was designed and implemented and to inform an understanding of the extent to which the Program implemented activities and produced results as planned. Data from interviews with key respondents with specialized knowledge of the HRH Programâits incep- tion, implementation, management, and transition after the close of funds from the Presidentâs Emergency Plan for AIDS Relief (PEPFAR)âwere used to further address the evaluation objectives. These data provided insight into the Programâs implementation and achievements, and into its perceived impact on HRH capacity and HIV service delivery. These data also facili- tated the interpretation of findings from the document review and from secondary quantitative data for HRH and HIV outcome indicators, which were analyzed for trends over time. The two in-depth examinations were conducted at the University of Rwanda and a âfacility microsystem,â a facility that receives referral pa- tients from a lower-level facility. These deeper examinations provided a more holistic understanding of the effects of the HRH Program on the capacity to produce a workforce of sufficient quantity and quality to meet the needs of the Rwandan population, and of the HRH Programâs role in af- fecting health care management and the provision of HIV and other health services. In an effort to assess the potential causal impact on HIV outcomes, the intention was to treat HRH Program graduates as an intervention, characterizing each districtâs dose based on quantity and type of graduate. The committee could then estimate the Programâs pooled effect on the HIV outcomes of interest. However, unavailability of data at the required level of detail hindered the committeeâs ability to perform this type of analysis. Evaluation Team A team of expert evaluators from EnCompass LLC collaborated with the committee and study staff in the design phase, carried out the primary qualitative and secondary quantitative data collection and initial analysis, and provided synthesized findings and initial interpretations to the commit- tee. See Appendix A for more information about the EnCompass evaluators. Sample A mixed-methods design involves collecting data from diverse popula- tions using an appropriate sampling methodology. Literature and Document Review The evaluation team conducted a thorough search to identify the docu- ments for review, including HRH Program reports; relevant global and
40 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA Rwandan policies, guidelines, plans, and strategies; and peer-reviewed and gray literature on topics related to HRH and HIV service delivery in Rwanda, the region, and worldwide. Key respondents shared additional documents during the qualitative data collection. In-Depth ExaminationâUniversity of Rwanda The University of Rwanda, as the primary institution responsible for educating health professionals, was purposively selected to understand the institutional capacity for health professional education. The evaluation team identified and invited key respondents to participate in interviews. Key respondents were individuals with knowledge of the HRH Program design and implementation and with experience with the Program in the university, including administration, leadership, faculty, and students across relevant schools (medicine, nursing and midwifery, and public health). In-Depth ExaminationâFacility Microsystem The intention of the facility microsystem analysis, which included at least a district hospital and a teaching/referral hospital, was to construct a bridge between the outputs of the HRH Program (i.e., a larger and well-trained health workforce, including greater availability of specialists for referral services) and patient-level outcomes for HIV. The evaluation aimed to designate facilities by degrees of exposure to the Program, as determined by having or not having HRH trainees on staff. However, as the evaluation team learned more about the Program, it became evident that all facilities had HRH trainees on staff, including hospital administrators, physician specialists, nurses, and midwives. It was also important to gather experiences and perceptions from HRH trainees and frontline health workers working outside of Kigali, prompting the selection of the Centre Hospitalier Universitaire de Butare (CHUB), which is a university teaching hospital in Butare in the Huye district (Southern Region), and Bushenge Hospital in the Nyamasheke district (Western Region). These districts were chosen because they had similar HIV prevalence to the 2014 to 2015 Rwandan national average of 3 percent (NISR et al., 2016), making them typical cases with respect to HIV-related indicators. The Nyamasheke district had higher antiretroviral therapy (ART) coverage (85 percent) than the 2015 national average (74 percent) (PEPFAR Rwanda, 2016),2 while the Huye district was comparable to the national average in both categories (see Table 2-1). 2â While more recent national-level estimates of ART coverage are available and indicate greater coverage; for selection purposes, it was necessary to use coverage estimates from the same year.
EVALUATION DESIGN, METHODS, AND LIMITATIONS 41 TABLE 2-1 HIV Prevalence and ART Coverage in Facility Microsystem Districts Huye District Nyamasheke District HIV Prevalence Among People Aged 15â49 2.9% 2.2% ART Coverage 73% 85% NOTE: ART = antiretroviral therapy. SOURCE: PEPFAR Rwanda, 2016. Qualitative Interviews Interviews (qualitative and in both of the in-depth examinations) involved a predominantly purposive sampling approach, mixed with snowball sampling, in which information-rich respondents were selected during the first round. These respondents were then invited to suggest other potential respondents. Respondents were selected from the follow- ing categories: â¢ HRH Program administration, both within and outside the Gov- ernment of Rwanda â¢ Faculty from U.S. institutions that were members of the HRH Programâs academic consortium â¢ Professional associations and people living with HIV (PLHIV) groups â¢ Other key stakeholders, such as representatives and staff of other donors and international organizations working in HRH and HIV in Rwanda Current employees of the U.S. Centers for Disease Control and Pre- vention (CDC) and PEPFAR at both the Rwanda country team and head- quarters levels declined to participate as interview respondents, owing to a determination made by CDCâs Associate Director for Science that there could be an actual or perceived conflict of interest. The final sample of interview respondents across the three evalua- tion components (facility microsystem in-depth examination, University of Rwanda in-depth examination, and qualitative interviews) included 87 interviews (see Table 2-2). Several respondents fell into multiple stakeholder types (e.g., a student who had graduated and gone on to serve as a university faculty member, or a university faculty member who also held a leadership position in a professional association).
42 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA TABLE 2-2 Interview Respondent Sample Stakeholder Type Sample Qualitative Interviews Government of Rwanda program administration 12 Nongovernment (of Rwanda) program 4 administration U.S. institution faculty 12 Professional associations and professional councils 2 Others (donors, international NGOs, experts, 5 PLHIV groups) University of Rwanda Faculty 5 In-Depth Examination Students 20 Administration 7 Facility Microsystem CHUB 12 In-Depth Examination Bushenge Hospital 8 NOTE: CHUB = Centre Hospitalier Universitaire de Butare/University Teaching Hospital, Butare; NGO = nongovernmental organization; PLHIV = people living with HIV. Data Collection The evaluation team conducted a literature and document review and collected publicly available quantitative data between October 2018 and October 2019. Primary qualitative and requested secondary quantitative data were collected between May and September 2019. Literature and Document Review The evaluation team conducted a thorough literature and document re- view of materials on the HRH Program and related topics, as well as docu- ments from other projects and programs related to HRH in Rwanda. The evaluators gathered publicly available documents from relevant websites dating from 1996 through 2019, beyond the end of PEPFARâs investment in the HRH Program. This time frame includes health system rebuilding following the genocide against the Tutsi, decentralization of the health sys- tem, and initiation of the HRH Program in 2012, and it extends beyond the implementation and closeout of PEPFARâs support of the HRH Program in 2017. The review comprised 4,267 documents: â¢ Program documentation: Request for proposals, proposals, de- signs, regular reporting to CDC, financial information provided by the Ministry of Health (MOH), and closeout documents
EVALUATION DESIGN, METHODS, AND LIMITATIONS 43 â¢ Peer-reviewed, gray, and unpublished literature on the HRH Pro- gram, health workforce, health professional education, and the HIV epidemic in Rwanda more broadly, as well as HRH labor market analyses â¢ Government of Rwanda documents, including policies, strategies, plans, and guidelines on HRH and HIV service delivery at the national and subnational levels; labor market reports and other documentation on labor market dynamics; and insurance scheme documentation â¢ Performance-based and results-based financing documents â¢ Global documents, including policies, priorities, strategies, plans, and guidelines on HRH and HIV service delivery from sources such as PEPFAR; the Global Fund to Fight AIDS, Tuberculosis and Ma- laria; the World Health Organization; and the Joint United Nations Programme on HIV/AIDS (UNAIDS) â¢ Program documents and publications from other HRH-related pro- grams in Rwanda, including those from donors and implementing partners Qualitative Data Collection All qualitative data collection conducted as part of the two in-depth examinations and other qualitative interviews used Appreciative Inquiry, an asset-based approach that captures current strengths and the most sig- nificant changes identified by respondents, as well as desired realities and steps needed to reach those realities. Appreciative Inquiry actively engages respondents and increases the likelihood of obtaining rich information about sensitive topics (Preskill and Catsambas, 2006). Data were gathered via semistructured individual interviews and group discussions. In keeping with the evaluationâs appreciative and utilization-focused principles, interview guides emphasized collecting participantsâ views of what worked well in the HRH Program,3 factors that facilitated or inhib- ited success, and opportunities for continued success in HRH in Rwanda. The guides also focused on areas where respondents saw potential for improvement in future HRH and health systems strengthening activities, programs, and initiatives. These approaches enabled the evaluation team to 3â Rwanda-based staff reviewed the interview guides for translation errors in Kinyarwanda and English and to ensure a clear understanding by all those interviewed. The semistructured interview guides were adjusted for relevance to respondent type. For trainee respondents, probes were adjusted as relevant. However, as is the standard when using semistructured in- terview approaches, the key questions were kept consistent to reveal trends across respondent groups.
44 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA gather a variety of perspectives on HRH Program successes and challenges and to ensure relevant, realistic findings and recommendations. A central tenet of qualitative methods is flexibility; in this case, the evaluators adapted and adjusted lines of inquiry and interview guides as new information and insights emerged. Data collection and analysis was iterative; data were coded and initial analyses were performed during the data collection phase to inform subsequent data collection. During regu- larly scheduled meetings, the evaluation team collectively determined new insights to examine more thoroughly and how to gather this information. This process involved minor modification to the interview guides to bet- ter understand the nuances of the HRH Program and context, but these changes were not substantive, in that they did not involve new topics or respondent groups. Initial analysis of the transcripts was another means of evaluating and enhancing the quality of the data. Transcripts were evaluated for fidelity to the data collection instruments, appropriateness and depth of probing, detail of descriptive notes from the data collectors, and qual- ity of the rapport between data collectors and key respondents. When issues arose, data collectors were retrained, focusing on improving areas of weakness. Data quality assessment took place throughout the data collection phase. Preference was for individual interviews to provide the space and confidentiality to examine individual experiences and potentially sensitive knowledge. All but one interview was individual. Respondents in Rwanda were invited to participate in person in Kigali and other locations; if sched- uling did not allow for this, the data collection team offered a virtual interview option. Data were gathered in English, French, or Kinyarwanda, according to the respondentâs preference. Respondents based in the United States or a third country were invited to participate in virtual interviews. Virtual data collection was conducted in English by U.S.-based evaluation team members using Zoom or Skype, depending on the quality of the con- nection and the respondentâs preference. All interviews and group discussions (in-person and virtual) were audio recorded, transcribed verbatim, and, when necessary, translated to English for analysis. Virtual interviews were recorded using an external device, not through the virtual platformâs recording option, to avoid a third party retaining a copy of the conversation. In-Depth ExaminationâUniversity of Rwanda Interview guides for the in-depth examination with the University of Rwanda were tailored to those in administrative roles, those in faculty
EVALUATION DESIGN, METHODS, AND LIMITATIONS 45 roles, and current and former students. Interviews explored the following topics: â¢ Experience with the administration of the HRH Program â¢ Process of, and experience with, building faculty capacity to train health care workers and managers, including via the twinning pro- gram and curriculum development â¢ Perception of systemwide effects of the HRH Program on health care service delivery in Rwanda: production, distribution, and management of HRH, including HIV service; planning and man- agement at health facilities; and referrals between facilities â¢ Perception of the HRH Programâs impact on the health education system infrastructure at the university In-Depth ExaminationâFacility Microsystem One Rwandan data collector made three site visits to collect data from the two facilities in the facility microsystem. Interviews were conducted with health care workers and health managers, those who had undergone training under the HRH Program and those who had not. Interviews ex- plored the following topics: â¢ Experience with health professional education training (pre-service and in-service) â¢ Professional career path â¢ Job satisfaction and belief in their own capacity, morale, and motivation â¢ Perception of their job as an important, viable career â¢ (When applicable) Perception of systemwide effects of the HRH Program on health care service delivery in Rwanda: production, distribution, and management of HRH, with a focus on HIV ser- vice; planning and management at health facilities; and referrals between facilities â¢ (When applicable) Perception of the HRH Programâs impact on the quality of HIV and other health services Qualitative Interviews Qualitative interview guides were developed according to initially iden- tified relevant topics and then adapted iteratively, based on respondent type and what had been learned in previous data collection activities, following the approach described above. Table 2-3 lists the detailed interview topics by respondent type.
46 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA TABLE 2-3 Interview Topics by Respondent Type Interview Topic PEPFAR investments in HRH broadly since 2011 Plan and design of the HRH Program Experiences with health professional education in the HRH Program Management of the HRH Program Implementation of the HRH Program, including any changes that occurred Coordination across other investments in the HRH Program and other HRH activities in Rwanda Perceived influence of Rwandan contextual factors on the HRH Program Perceived influence of PEPFAR and global health contextual factors on the HRH Program Perceived influence of the HRH Program on infrastructure for professional health education, production, management, and distribution of HRH; quality of services; and service delivery Perceived impact of the early termination of the HRH Program Perceptions of institutionalization of HRH Program achievements Learning for future HRH investments in Rwanda NOTE: HRH = human resources for health; PEPFAR = Presidentâs Emergency Plan for AIDS Relief; PLHIV = people living with HIV. Quantitative Data The evaluation team sent an informal request for quantitative data to the MOH and Rwanda Biomedical Center (RBC) on May 7, 2019, followed by a formal request, sent on May 17, 2019. The RBC issued an approval letter on May 20, 2019, but with the stipulation that study findings be shared with the RBC before publication or any dissemination. To meet this condition, the National Academies of Sciences, Engineering, and Medicine (the National Academies) proposed that the MOH conduct a technical review of the analysis of MOH-provided data and that a predissemination briefing be held with the MOH. The Ministryâs agreement to this proposal was received on August 20, 2019, at which point the evaluation team began working directly with key individuals in the MOH to obtain these data. The following data were requested:
EVALUATION DESIGN, METHODS, AND LIMITATIONS 47 Respondent Type Program Rwandan Administrators Professional (Rwandan Twinned Faculty and Association and Staff from Other Government HRH Program Leads National-Level Donors and Partners Officials; from the PLHIV Group Working in HRH in PEPFAR Staff) United States Leadership Rwanda x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x â¢ HIV-related indicators at the facility level â¢ HRH Program trainee data: who was trained under the Program, in what specialty, graduation dates, and current place of employment â¢ Equipment procured under the HRH Program â¢ HRH Program budget and expenditures The evaluation team did not receive facility-level HIV data from the MOH, leaving the committee to rely on publicly available data, predomi- nantly from UNAIDS. Per communication with the MOH, there was not a readily available database of HRH Program trainees and their current places of employment; thus, it was not possible to understand which facili- ties had been exposed to the HRH Program. The MOH and the University of Rwanda did provide elements of the data on trainees alongside data from
48 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA professional councils responsible for licensures and other publicly available data from Government of Rwanda sources, such as the Master Facility List. However, these data varied in some instances, confounding interpretation. Because of the timing of receipt of the data, the committee was not able to reconcile these variations across sources. Data Analysis Literature and Document Review The evaluation team conducted text analysis of literature and docu- ments using a Microsoft Excel spreadsheet set up to capture and organize data according to topics represented in the evaluation objectives and ques- tions and reflecting the theoretical causal pathway discussed in Chapter 1. The team generated a time line to more readily observe the events that preceded and were concurrent to HRH Program implementation, including after the end of PEPFAR funding for the Program. Comparison matrices were generated to explore convergence and divergence between what was planned as described in the design documents and what was executed, as described in HRH Program reports through 2017. Text and policy analyses were performed to examine key elements of the context and the Program,4 in keeping with the theoretical causal pathway. In-Depth Examinations and Qualitative Data The evaluation team employed a utilization-focused and iterative data collection and analysis process, with initial data analysis occurring as data were still being gathered to ensure quality, refine questions, and identify new lines of inquiry. Any additional data added to the dataset were coded appropriately, with new findings integrated into existing findings and tested to confirm or disconfirm cases. A combination of deductive (theory-driven) and inductive (data-driven) coding approaches were applied to the qualitative data. The evaluation team developed an initial coding scheme, based on the evaluation objec- tives, and included the topics in the data collection instruments. Comple- mentary to the deductive approach, transcripts were read to enable open coding and identification of topics outside the content of the guides. Four members of the evaluation team led the coding, using an iterative process, reapplying additions or changes to the coding scheme to the entire dataset. 4â âPolicy analysisâ describes a type of analysis that examines economic, social, or other pub- lic issues through the formulation, adoption, and implementation of a principle or approach to address a problem. As this examination was of existing policies, it is primarily descriptive.
EVALUATION DESIGN, METHODS, AND LIMITATIONS 49 Frequent meetings with the evaluation team members participating in cod- ing facilitated a shared understanding of the coding scheme. Intercoder reli- ability testing was conducted to ensure at least 90 percent agreement across coders. Qualitative transcripts were coded in Dedoose, an online qualitative data analysis platform the team selected to allow multiple team members to code in the same platform and check and correct any discrepancies in real time. Dedoose enables encryption, so only those participating in data collection or analysis had access to these data. Coded data were synthesized into categories and themes, informed by the evaluation objectives. The team produced interpretive data summaries, reflecting the initial insights gener- ated in each category or theme and, where appropriate, links with other categories or themes. The evaluation team performed comparative analysis to identify points of convergence and divergence across key respondent groups. Comparative matrices were prepared by theme to generate visual representations of the points of convergence and divergence. Findings generated from these data sources were combined with insights generated through the comparative matrices to enrich understanding of the HRH Programâs planned activities and what had been implemented. The evaluation team conducted two main phases of data analysis, with the first round in preparation for the fourth committee meeting in July 2019. Insight and guidance from the committee was integrated into subsequent data collection, coding, and analysis. The second phase of data collection was completed in August 2019. Following data collection, the evaluation team conducted an internal, integrated 2-day data analysis, tri- angulation, and interpretation session. Quoted material throughout this report is redacted to protect respon- dentsâ confidentiality and anonymity, in alignment with the ethical guidance approved in the evaluation protocol. Direct quotations from respondents included in this report have been edited for clarity. Quantitative Data The evaluation team used a time series approach to examine publicly available HRH and population-level data for trends over time. Data were plotted in run charts and annotated with events and other descriptions to help interpret what was observed. Where appropriate, a maximum likeli- hood event count time series analysis was performed. A maximum likeli- hood event count time series model assesses the probability of a maximum parameter value in an ordered sequence of observations through time. It is generally a stochastic method, and when analyzing the variability of HRH program data, this approach was useful to assess whether the intervention had an effect significantly greater than the underlying trends in HRH and
50 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA population-level data. Wherever possible, this analysis was performed at national and subnational levels. First, data was collated and cleaned in Microsoft Excel and subsequently analyzed in Stata 14 using the estima- tion function. Analysis of financial data was descriptive, exploring changes in the level of funding for the HRH Program across sources. Other CDC and do- nor funds in HRH and health systems strengthening in Rwanda were also examined for change over time, as a means of understanding the financial landscape. Analysis and Interpretation with the National Academiesâ Committee To ground findings and conclusions in utilization-focused approaches, following initial independent analysis of data by type, the evaluation team used a participatory process for integrating and synthesizing the data. This was conducted at three key points in the data collection and analysis period. First, during the fourth meeting of the National Academiesâ committee (July 9â10, 2019), the EnCompass team presented initial insights into the data that had been collected by that time. The committee worked through an affinity analysis process, in which they organized insights into categories and then queried the data for what was missing, what needed further data collection, and what needed additional data analysis. The evaluation team also held a 2-day internal data analysis, triangula- tion, and interpretation workshop on August 20â21, 2019. This workshop included only those on the evaluation team who had collected or analyzed data. Participants examined evidence across data sources, shared emerging themes, and developed initial findings and interpretations. The result of the workshop was an evidence-based narrative addressing the first three evaluation objectives, as well as thematic matrices that presented qualitative and quantitative data side by side, facilitating observation of convergence and divergence and explanation of evidence that supported each finding and conclusion. The evaluation team then supported a process of further analysis and interpretation by the committee in September 2019. After providing com- mittee members with draft findings and interpretations, backed by the thematic matrices, the evaluation team facilitated a participatory, apprecia- tive session with the committee to further examine, interpret, and validate findings and to arrive at consensus on conclusions and recommendations. Ethical Considerations Approvals for this evaluation were provided by the Rwanda National Ethics Committee (April 3, 2019), the U.S. National Academiesâ Commit-
EVALUATION DESIGN, METHODS, AND LIMITATIONS 51 tee to Review Studies on Human Subjects (April 23, 2019), and the CDC Office of the Associate Director for Science (April 25, 2019). Permissions were obtained from the necessary national, subnational, and institutional authorities to carry out the in-depth examinations at the University of Rwanda and the facility microsystem. All potential respondents for in-depth examinations and interviews re- ceived a written information sheet and a certificate of consent in English or Kinyarwanda, as appropriate. The information sheet described the purpose of the evaluation; how the respondent(s) had been selected; the data collec- tion method, procedure (individual interview or group discussion), and du- ration; risks and benefits; reimbursements; confidentiality; planned sharing of findings; and voluntariness (right to refuse or withdraw). Respondents also received contact information for the Rwanda National Ethics Com- mittee. Signed consent was the preferred method of documentation. Virtual interview respondents received the consent information during the recruit- ment and scheduling process, and their electronic signatures were obtained. LIMITATIONS In an effort to address the first two objectives, which focused on de- scribing the HRH Program and the context in which it was implemented, the evaluation team conducted qualitative interviews with individuals who were involved in the administration of the Program. These included respon- dents from the Government of Rwanda and other organizations involved in the design and early implementation. However, CDCâs Associate Director for Science determined that the participation of current staff would pres- ent a conflict of interest, and staff from CDC, PEPFAR, and the Office of the U.S. Global AIDS Coordinator declined invitations for interviews. The absence of PEPFARâs perspective, as the funder of the portion of the Pro- gram under evaluation, presents a significant gap in the data and has likely resulted in skewed findings. The donorâs perspective would have provided balance and nuance to findings, conclusions, and recommendations around program design, management, and sustainability. The absence of clear con- clusions regarding the donor is the result of these missing data and should not be interpreted as an indication of donor performance. Not all specialties under the HRH Program are represented equally in the qualitative interviews. The sample is skewed toward pediatrics, obstetrics and gynecology, and nursing, facilitating deeper examination of these specialties relative to others, such as surgery, internal medicine, and midwifery. With respect to the third objective of the evaluation, described in more depth in Chapter 1, it was not possible to design an evaluation that as- sessed the attributable impact of the Program on HRH outcomes or HIV-
52 EVALUATION OF PEPFARâS CONTRIBUTION TO RWANDA related morbidity and mortality. The lack of facility-level HIV indicator data decreased the committeeâs ability to assess meaningful differences in HIV care and outcomes between facilities with and without HRH trainees. However, even if the committee had these data, identifying an appropriate comparatorâwhich would be the most appropriate approach for deter- mining attributionâwas not possible. Instead, this evaluation uses the lens of contribution analysis and presents regional data alongside Rwandan indicator data to situate Rwandaâs progress in context, without making analytical comparisons. Drawing conclusions about plausible contribution was also challeng- ing, because of the multitude of interacting factors and concurrent programs that could reasonably be expected to contribute to the same outcomes of interest specified in the Statement of Task. Furthermore, the timing of this evaluation, just 2 years following the end of PEPFAR funding, also pre- sented an obstacle to determining changes in HIV-related mortality and morbidity, as well as other HRH outcomes. Producing a specialized health workforce that has entered the health system and is providing high-quality services for PLHIV takes years, and observing any patient-level effects would require even more time. This challenge was similarly important for evaluating the Programâs sustainability and degree of institutionalization. In addition, the integration of financial investments from diverse sources in the Rwandan governmentâs administration and management of the HRH Program meant it was difficult to disentangle PEPFARâs invest- ments and related activities from other sources (Government of Rwanda and the Global Fund). It was also challenging to assess the plausible contri- bution of the activities that were predominantly funded by PEPFAR without more detailed disaggregation of Program expenditures, which would have enabled a more thorough description and analysis of the relative contribu- tion of different programmatic activities. Finally, the request for data on where HRH trainees were placed fol- lowing graduation had to be extracted from multiple sources, as Rwandaâs Human Resources Information System did not include information on health worker training. This placed a significant burden on the MOH for extracting and generating the information. Much of the data used in this analysis was therefore extracted from publicly available data, which was generally aggregated data that did not allow for effective subnational analysis. Other data were not up to date; for example, the most recent An- nual Health Statistics Booklet published by the MOH includes data up to 2016. Finally, the data that were received from the MOH raised questions when triangulating and interpreting them alongside data received from other sources. A technical review was conducted in October 2019, in which the MOH was provided with tables and figures, as well as descriptive text (methods
EVALUATION DESIGN, METHODS, AND LIMITATIONS 53 applied and description of data sources), but excluding findings, interpreta- tions, or any information that could be interpreted as committee delibera- tions. The purpose of this review was to seek guidance from the MOH and prevent misinterpretations that were technical in nature. In accordance with the National Academiesâ policy, comments and written feedback provided by the MOH as part of this technical review are included in the studyâs Public Access File. Documents and other information provided for the committeeâs consideration are available upon request from the National Academiesâ Public Access Records Office (firstname.lastname@example.org). REFERENCES Creswell, J. W., and V. L. Plano Clark. 2007. Designing and conducting mixed methods re- search. Thousand Oaks, CA: Sage Publications. Moore, G., S. Audrey, M. Barker, L. Bond, C. Bonell, C. Cooper, W. Hardeman, L. Moore, A. OâCathain, T. Tinati, D. Wight, and J. Baird. 2014. Process evaluation in complex public health intervention studies: The need for guidance. Journal of Epidemiology and Community Health 68(2):101â102. NISR (National Institute of Statistics of Rwanda), MOH (Ministry of Health), and ICF In- ternational. 2016. Rwanda demographic and health survey 2014â15. Rockville, MD: National Institute of Statistics of Rwanda, Ministry of Health, and ICF International. Patton, M. Q. 1987. How to use qualitative methods in evaluation. Newbury Park, CA: Sage Publications. PEPFAR (Presidentâs Emergency Plan for AIDS Relief) Rwanda. 2016. Rwanda country op- erational plan FY 2016. Kigali, Rwanda: Presidentâs Emergency Plan for AIDS Relief. Washington, DC: Office of the U.S. Global AIDS Coordinator and Health Diplomacy. Preskill, H., and T. T. Catsambas. 2006. Reframing evaluation through appreciative inquiry. Thousand Oaks, CA: Sage Publications.