Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 605
Appendix C Evaluation Methods OVERVIEW The evaluation employed a mix of methods and layers of investigation and analysis involving a range of primary and secondary data sources, taking into account the methodological design considerations described in Chapter 2. This included mapping of investments using financial data, assessing trends over time using program monitoring indicators and clinical data from OGAC and PEPFAR implementing partners, benchmarking progress against stated programmatic targets and goals, reviewing extensive documents, and analyzing primary data collected through more than 400 semi-structured interviews with a range of stakeholders on visits to 13 PEPFAR partner countries, at the U.S. headquarters of PEPFAR, and at other institutions and multilateral agencies. Primary and secondary data were analyzed, using appropriate methodologies, by the members of the evaluation committee, the Institute of Medicine (IOM) study staff, and consultants with specialized knowledge in both qualitative and quantitative methodologies. The committee, staff, and consultants took steps to assess and ensure the quality and completeness of the data used for the evaluation, and took these factors into account during data interpretation. The methods used to assure the quality of the primary data collected and the secondary data received through data requests are described in more detail in the sections that follow. When data analyzed externally were used, the committee, staff, and consultants reviewed and assessed the methodology and quality of the data. The mandate of the committee was to draw conclusions and make recommendations across the whole program. Wherever possible, data were gathered and data analyses and interpretation were conducted and presented across all 31 PEPFAR partner countries defined as the focus of the evaluation; however, only very limited data were comparable and comprehensive across all countries. In order to not limit the committee’s findings to data consistently available across the whole of the program and all of these countries, which would have been a significant constraint, the evaluation draws on subsets of countries, programmatic areas, or intervention components implemented within PEPFAR for which sufficient data could be gathered to contribute to the assessment. Therefore, data presentations and analyses representing these subsets were interpreted with care to inform conclusions about the whole of the program. For example, analysis of country visit interview data was limited to the countries selected for visits by the committee. In addition, some analyses drew on existing data sources that are only PREPUBLICATION COPY: UNCORRECTED PROOFS C-1

OCR for page 605
C-2 EVALUATION OF PEPFAR available for some countries, programs, and partners, such as Track 1.0 partner data. Some evaluation questions were most applicable only for a subset of countries, such as countries with concentrated epidemics driven by injecting drug use. Finally, the time and resources available limited the scope of some analyses, such as those involving review of Country Operational Plans for which the sheer volume of the documents over all countries and years limited the feasibility of comprehensive review across all countries. Throughout the report, where data analyses that do not represent the whole of the program are presented, the scope of these data is described. Because the committee was not charged to draw conclusions or make recommendations at the level of specific countries, partners, or programs, analyses of data from subsets of countries or partners are presented in a manner designed to maintain anonymity. By applying this mix of methods and layers of investigation and analysis using a range of available primary and secondary data sources, the committee arrived at findings that could be triangulated to draw conclusions about the performance and impact of PEPFAR, even when any one data source was not sufficient or any one methodological approach was not feasible. Building on the interpretation of the available data, the conclusions and recommendations presented in this report represent the consensus reached through the deliberations of the evaluation committee. Over the course of the evaluation, the full committee met six times in person, with participation of the staff and consultants. One additional meeting was conducted using Web-based conferencing. In addition, working groups within the committee that were focused on specific content areas held additional meetings by teleconference as needed for ongoing deliberations as well as for data analysis and interpretation. These committee activities were augmented by ongoing communications by telephone and e-mail among the committee members, staff, and consultants. The following sections describe some of the overarching processes the committee used to frame and shape the evaluation. Subsequent, more detailed sections describe the methods for each of the data sources used in the evaluation. Development of Evaluation Questions and Mapping of Data Sources Through working groups comprised of a subset of committee members, the evaluation committee identified proposed evaluation questions based on major content areas, the Statement of Task (see Appendix A), the Program Impact Pathway (PIP) framework (see Chapter 2), and the preliminary work reported in the Strategic Approach (IOM and NRC, 2010). Once the working groups established their initial questions and subquestions, IOM staff and consultants developed and provided to the committee the following information pertaining to each of the more than 300 questions: The domains of the PIP to which the question belonged (i.e., input, activity, output, outcome, or impact) The type of data necessary to answer the question (e.g., financial data; program monitoring, surveillance, and clinical data; interview data; literature and document review) A description of potential data sources that had been identified Limitations associated with the data sources, such as issues related to availability, feasibility of accessing the data, and any other relevant issues that could inform considerations for formulating data requests and for the utility of the data PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
APPENDIX C: METHODS C-3 Mapping of Potential Data Sources The IOM staff and consultants carried out an extensive data-mapping effort, expanding on the preliminary work conducted during the strategic planning and operational planning phases. The data-mapping process relied on document review, stakeholder interviews, information obtained from preliminary data requests, and information gathered during 2 pilot country visits. The data mapping served to assess the feasibility of collecting and using data from each source, taking into consideration the burden that data requests would place on each source’s resources and staff time. In addition, this data mapping assessed whether data from each source would require new data analysis in order to answer the evaluation questions posed by the committee. The categories of available data sources that were mapped and ultimately used for the evaluation included financial data; program monitoring, surveillance, and clinical data; interview data; and literature and document review. The sources included central OGAC data, data from multilateral organizations, data from implementing partners, and data from publicly available documents and other sources. The data sources used for the evaluation are described in more detail in subsequent sections of this appendix. Priority Evaluation Questions Committee members then worked with IOM staff and consultants to finalize a set of priority evaluation questions (PEQs) based on relevance to the Statement of Task and related evaluation considerations, relative importance among subquestions, and feasibility of answering each question with the time, resources, and data available. The ultimate relative contribution of data sources to different content areas and evaluation questions, and, ultimately, to the committee’s conclusions and recommendations, varied depending on availability and appropriateness. Overview of Data Collection A summary of the data request and data collection processes for each major data source is provided in the sections that follow, along with a description of the analyses for which the data were used. Requests for interviews and requests for secondary data not readily available publicly were made by the IOM independently, with OGAC and in country Mission Teams serving as a liaison only when necessary. Participation in the evaluation was voluntary. Except when reference is made to existing published materials, findings, examples, and comments are not attributed to individuals and the identities of individuals, programs, partners, and countries are protected. FINANCIAL DATA Global Financial Data To contextualize PEPFAR’s financial contribution within the broader donor funding landscape for HIV/AIDS, the committee examined disbursement data on official development assistance for HIV/AIDS as reported to the Organization for Economic Cooperation and Development (OECD) Creditor Reporting System. Disbursements represent the sum of two PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-4 EVALUATION OF PEPFAR OECD sector codes: STD control including HIV/AIDS and Social mitigation of HIV/AIDS. The committee examined data for the 31 PEPFAR countries that were writing COPs when the IOM evaluation study process began in 2009. PEPFAR Financial Data: Available, Obligated, and Outlaid On a quarterly basis, OGAC submits summary financial status reports to Congress on “the allocation, obligation and expenditure of funds appropriated for [PEPFAR]” (PEPFAR, 2012). These reports are publicly available. The committee used the fourth-quarter report from each fiscal year to calculate annual appropriations, obligations, and outlays for the PEPFAR program. PEPFAR Financial Data: Annual Expenditure Data Calculated from Agency Reporting In May 2012, in response to a committee data request, IOM consultants received from OGAC PEPFAR funding obligations and outlays for fiscal years (FYs) 2004 through 2011 for all countries receiving PEPFAR funding. Upon review of the data and through clarifications with OGAC, IOM staff and consultants realized that these financial data corresponded to the cumulative amount of funding available, obligated, and outlaid from each budget year rather than the actual annual amount of funding available, obligated, and outlaid. Another request was made to OGAC for funding data that would clearly distinguish funding by budget year and reporting year and would represent actual annual expenditures, regardless of the year in which the money was appropriated or obligated. In July and August 2012, IOM consultants received from OGAC cumulative agency-specific funding for each reporting year. Annual expenditures were derived as described below. Data Description OGAC sent IOM consultants 78 Excel spreadsheets containing financial data for the 6 agencies that received PEPFAR funding between FY 2004 and FY 2011: United States Agency for International Development (USAID) Peace Corps (PC) Department of Defense (DOD) Department of Labor (DOL) Department of State (STATE) Department of Health and Human Services (HHS) With the exception of STATE, each agency reported all of its financial information to OGAC in a consolidated format. STATE, however, reported its PEPFAR funding through 5 distinct offices/bureaus: Bureau of African Affairs (AF) Bureau of East Asian and Pacific Affairs (EAP) Office of the Global AIDS Coordinator (OGAC) Bureau of Population, Refugees and Migration (PRM) Bureau of Western Hemisphere Affairs (WHA) PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
APPENDIX C: METHODS C-5 Each file contained cumulative available, obligated, and outlay budget information, by country, corresponding to a single fiscal year’s budget. As discussed in Chapter 4, most PEPFAR funding does not have an annual use-or-lose requirement (i.e., unspent funding from one fiscal year can often be carried over to be spent in subsequent years). Therefore, the money spent during a particular year had the potential to come from the budgets of multiple, prior fiscal years. Consolidating Data into Consistent Files Each agency provided funding to a different group of PEPFAR countries, many of which received funding from more than one agency. Therefore, each agency’s set of budgetary files was first consolidated into a single data file for a total of 10 unique datasets—one set per agency or bureau. Each country’s annual funding was retained within each data file to enable potential analyses that would require subgroups of PEPFAR countries based on country attributes. Second, these datasets were harmonized into a single dataset to allow for data to be used together to comprehensively represent PEPFAR spending across agencies, in total and by country. Documenting Discrepancies, Notes, and Comments The data extraction process revealed embedded comments within spreadsheet cells and footnotes explaining data nuances; this information was recorded in a separate file. Additionally, some funding numbers changed from one reporting year to the next. Increases in funding amounts were expected over time as more of the funding from a particular fiscal year was expected to be obligated or outlaid. Decreases, however, were not expected from one year to the next; i.e., the amount of available funding from a specific fiscal year budget was not expected to decrease in subsequent reporting years. Therefore, these unexpected changes in the funding data were documented. For all three of these scenarios—embedded comments, footnotes, or unexpected changes in funding—the following information was recorded corresponding to each instance: Agency/Bureau—which agency’s spreadsheets contained the comment, footnote, or inconsistency Country/Region—the country or region affected by the comment, footnote, or inconsistency Reporting Year—the reporting year with the observation Budget Year—the year during which the budget was issued Comment, footnote, or inconsistency—verbatim comments and footnotes from the spreadsheet; inconsistencies were described as clearly as possible Detected by agency or IOM—an indicator variable reflecting whether the comment or footnote was already in the spreadsheet or whether the inconsistency was encountered by IOM staff during the data extraction process To further assess the most notable discrepancies in the available totals by country by year, these were compiled in a separate spreadsheet and compared from their inception year to the proceeding year across annual reports through 2011. In particular, major discrepancies occurred when the dollar amounts reported as available for a given budget year changed (both increases and decreases were observed) in subsequent reporting years, although one would expect the amount to be a fixed constant for a budget year after that year in which it was made PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-6 EVALUATION OF PEPFAR available. These discrepancies ranged in magnitude with a maximum of a $214 million difference between two reports for one budget year for one agency. As a result, it was difficult to assess the correct figures for the total amounts made available. Overall, the number of discrepancies and magnitude of changes from year to year diminished in later reporting years, and the same degree of discrepancy was not seen in the reporting of outlays. Calculation of Annual Expenditures Once the funding data were completely extracted into a single data file, serial subtractions were performed of each reporting year’s cumulative outlay data to obtain the amount of money actually spent (outlaid) during each reporting year, regardless of the fiscal year’s budget from which the money came. To get the annual expenditure for within a given fiscal year, all prior year outlays were subtracted from the cumulative total outlays reported for that year. Given the data discrepancies described above, for the calculations for annual expenditures the data from the FY 2011 reports were used for the data for all of the fiscal years to have one consistent source that reflected the most recently available data. Quality Control When all of the data had been extracted into consistent data files, an IOM consultant compared all the extracted data files against the raw data files sent from OGAC. The validator worked with the original data extractor and reconciled all inconsistencies uncovered within the extracted data files. This independent validator also verified the serial equations used to calculate the amount of funding spent during each reporting year. The validated datasets were not reconfirmed with OGAC. Data Presentation Once all of the data had been validated, IOM consultants imported the data into SAS to generate financial presentations of the annual expenditure over time. These presentations were provided by the consultants to the committee in November 2012. PEPFAR Financial Data—Planned/Approved Funding for All PEPFAR Countries Planned/approved funding reflects how OGAC and PEPFAR Mission Teams plan to obligate and outlay funds. Each year, OGAC releases an operational plan for PEPFAR that includes summary budget information regarding the planned and approved use of PEPFAR funding, including which activities will be implemented by which agencies, as determined during the interagency planning process. The operational plans report planned/approved funding for four technical areas that correspond to the primary categories of HIV/AIDS services and systems strengthening efforts: Prevention, Care, Treatment, and Other. PEPFAR funding is planned through budget codes which capture funding information about more specific activities within these categories. Data Extraction Process The planned/approved funding was extracted by year and by budget code. Planned/approved funding data were extracted independently by two IOM staff into identical spreadsheets. Each staff member extracted data on total PEPFAR funding by implementing agency and year, as well as total PEPFAR funding by budget code and year. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
APPENDIX C: METHODS C-7 Data presentations The data extraction was validated and IOM consultants converted the data into constant 2010 USD to allow for a consistent interpretation of funding over time. The consultants then generated a final dataset to be used for data presentations showing funding by agency, type of program, and budget code. PEPFAR Financial Data—Planned/Approved Funding for Subset of 31 Countries Data on planned/approved funding from the subset of 31 countries that were the focus of this evaluation were gathered through a separate data extraction. These data were used for the committee’s analysis of funding by country characteristic. Data Extraction Process The planned/approved funding was extracted according to the following classifications: By country (31 countries in total) By year (FY 2005 through FY 2011, for the years during which a country was completing a COP) This funding information was extracted from the following publically available data sources, which were determined to be the most comprehensive across the classifications for the data extraction: o FY 2005 through FY 2007—Focus countries only: Operational Plans o FY 2008—Focus countries: Operational Plan; non-Focus countries: individual Country Operational Plans (COPs) o FY 2009—All countries: individual COPs o FY 2010 and FY 2011—All countries: Operational Plans IOM consultants developed specifications corresponding to the variables necessary for the PEPFAR financial data extraction process and developed dataset specifications for two separate extraction processes. The first data extraction compiled annual, country-specific funding by agency; the second data extraction compiled annual, country-specific funding by technical area and budget code. Data were not extracted by both agency and technical area, but rather either by agency or technical area. During the extraction process, any funding corresponding to regions was omitted (e.g., Central America, Central Asia, and Caribbean) and the process was limited to the 31 countries that were preparing COPs at the time this evaluation was initiated. Funding amounts were rounded to the nearest whole dollar. During the extraction by technical area, some budget codes switched from one technical area to another across reporting years; however, these differences were tracked in an effort to make consistent comparisons over time. Data Extraction Quality Control Two IOM consultants extracted the data independently into comparably formatted spreadsheets. Each consultant extracted a spreadsheet of funding data by year, country, and agency, as well as a second spreadsheet by year, country, technical area, and budget code. Once all of the data had been extracted across all budget years, one of the consultants developed a tool PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-8 EVALUATION OF PEPFAR to compare individual records across datasets and flag inconsistent values. This comparison tool flagged every instance of a record with inconsistent information, whether it corresponded to how the extractors recorded a particular budget code or whether the budget amounts differed. Together, the consultants then reconciled the inconsistent records. Once their datasets matched 100 percent, a third, independent consultant imported the data into SAS and used a random number generator to select 50 (about 4 percent) of the 1,302 records that summarized the financial information by agency. The consultant also randomly selected 80 (about 2 percent) of the 4,123 records that summarized the financial information by budget code and crosschecked these 130 values against the information written in the HQ operational plans and COPs. Note that some of these records corresponded to countries/agencies/budget codes that were not specified during a particular year. Therefore, this selection of records also confirmed that particular combinations of years/countries/agencies/budget codes were not inadvertently incorporated into the datasets. All 130 validation records matched the operational plans exactly, thus, confirming the quality of the data extraction process. Data Presentations Once the validation process was complete, IOM consultants generated a final dataset to be used, along with publicly available data from global sources, for data presentations showing PEPFAR funding by HIV prevalence, average funding per person living with HIV, and country income level. Planned/Approved Funding by Prime Partner The committee examined planned/approved funding extracted from a range of publically available data sources. The process of extracting and compiling these data was time intensive, so to be feasible within the resources and time available for the study, the committee’s analysis had to be limited to a subset of partner countries. The committee chose to compile these data for the same 13 countries purposefully selected for country visits, as described later in this Appendix. Within this subset of countries, the committee was able to compare partner data and planned/approved PEPFAR funding for the focus countries for FY 2004 through FY 2010 and for non-focus countries for FY 2008 through FY 2010. Data Extraction Data were extracted according to the following characteristics: By country (13 countries in total) By year (FY 2004 through FY 2011) By prime partner This funding information was extracted from the following data sources, which were determined to be the most comprehensive data available across the classifications for the data extraction. For FY 2004 to FY 2006, the prime partner funding data were extracted from a Center for Global Development (CGD) dataset in order to present the most complete data consistently for those years. The CGD publicly released this dataset that “was originally obtained from the State Department by the Center for Public Integrity through several Freedom of Information Act requests and a lawsuit against the U.S. Government, settled out of court” (CGD, PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
APPENDIX C: METHODS C-9 2008). This dataset contains PEPFAR data on country funding obligated to prime partners in focus countries in FYs 2004, 2005, and 2006. Before releasing the dataset, CGD added data on central funding that is obligated from OGAC HQ to partners to implement programs in countries. For FY 2004 and FY 2005, CGD obtained central funding information from the Center for Public Integrity; FY 2006 funding was estimated based on previous funding amounts and the total allocation of PEPFAR funding for focus countries in FY 2006. Partner lists that provide the amount of funding obligated to prime partners within a country are also available on PEPFAR’s website. For focus countries, these lists are available for FY 2005 to FY 2008; for non-focus countries, these lists are only available for FY 2008. By comparing most of the PEPFAR partner lists and the CGD dataset for FY 2005 and FY 2006, it appears that the PEPFAR partner lists for 2005 include country and central funding, but the PEPFAR partner lists for 2006 include only country funding. Therefore, the CGD dataset was determined to be the most complete data for FY 2004 to FY 2006 and allowed the presentation of both country and central funding consistently for these years. For FY 2007 and FY 2008, data were extracted from PEPFAR partner lists. PEPFAR partner lists provide funding amounts but do not include information about the type of funding (i.e., central or country funding). There is no equivalent dataset to the CGD dataset for FY 2007 and FY 2008 so the committee was unable to determine whether the FY 2007 and FY 2008 partner lists report country and central funding, or only country funding. The only source of partner data for FY 2009 and FY 2010 are the COPs; these data are limited to planned partner funding (not obligations). Since not all partners have been chosen by the time the COPs are submitted, these are incomplete sources of partner funding. For example, 2 percent of total funding data extracted from the FY 2009 COPs was labeled as “To Be Determined”, which means that a partner had not yet been chosen or contracted with to provide planned activities. Two IOM staff members extracted the data independently into identical spreadsheets. After the data extraction, staff carried out additional research as needed to determine the type (multilateral, government, nonprofit, for-profit, academia) and origin (U.S.-based, partner country–based, multilateral, other) of each prime partner. The type and origin of each partner was also recorded into the same spreadsheets with the funding information. Data Extraction Quality Control After all the data and supplementary information had been extracted, one of the IOM staff developed a tool to compare individual records across spreadsheets and flag inconsistencies. This comparison tool flagged every instance of a record with inconsistent information. Independently, the staff went back to the original sources to confirm or edit the inconsistencies and the spreadsheets were compared again. Any inconsistencies that persisted were reconciled together until all records matched. Data Limitations The prime partner data the committee compiled was limited by incomplete data sources, as described above. Overall, the total amount of partner funding compiled for this analysis reflects only 77 percent of the total planned/approved funding for this subset of countries for FY 2005 to FY 2010 (as reported in the operational plans). The gap between the data used for the analysis and the total planned funding represents expenses not expended through the COP prime partner mechanism, To Be Determined (TBD) funding, and any central funding not reported in PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-10 EVALUATION OF PEPFAR partner lists or COPs. Funding may have been reported as TBD if prime partners had not been identified prior to reporting or in situations where partners had been identified, but contracts had not been finalized. Given the nature of these gaps, the funding not represented in the dataset would be distributed across types of partners, therefore the committee determined that the dataset represented adequate information for a reasonable descriptive analysis to help understand the distribution of prime partner funding. Data Presentations Once all of the data were validated, a final Excel dataset was used to produce data presentations showing funding over time by type and origin of prime partner using data aggregated across all 13 countries. PEPFAR PROGRAMMATIC INDICATOR DATA Background PEPFAR indicators are used to monitor and assess progress in the HIV/AIDS response within and across PEPFAR-funded technical areas. Indicators are grouped into two distinct categories: those that were collected by partner countries during FY 2004–FY 2009 and those that are being collected starting in FY 2010, after a revision of the indicators (OGAC, 2009). Prior to FY 2010, there were two reporting levels of PEPFAR indicators. Indicators were either required and reported to OGAC or were recommended but not reported to OGAC. Starting in FY 2010, PEPFAR classified indicators according to three reporting levels: essential reported, essential not reported, and recommended. 1 During FY 2004–FY 2009, focus countries were required to report to OGAC 47 “essential” country-level and programmatic indicators2. Seven of these 47 indicators corresponded to overall country-level indicators (see Table C-1). These seven country-level indicators were comprised of two components, direct and indirect, “where direct results represented counts for PEPFAR prevention, care, or treatment support provided at PEPFAR- supported delivery sites, while indirect results represented PEPFAR contributions to national, 1 OGAC defines these 2009 classifications in the NGI guidance as follows (OGAC, 2009): Essential/Reported to HQ: Indicators that are aggregated and reported to PEPFAR Headquarters. Essential/Not Reported to HQ: Indicators that do not need to be aggregated and reported to PEPFAR Headquarters; however, partners are required to report applicable indicators to the PEPFAR country teams. In addition, PEPFAR country teams are expected to support and encourage intermittent surveillance required to monitor indicators not routinely captured through programs. The intent of these indicators is to highlight critical program areas that PEPFAR country teams should be monitoring and provide teams increased flexibility to work within the context of the national system. Recommended: These are additional indicators for partners and program managers who need information for program management beyond the minimum set reported to OGAC Headquarters. The PEPFAR interagency TWGs selected and recommended these indicators as important areas for program managers to monitor, but they are not considered indispensable to program tracking. The intent of these indicators is to encourage comprehensive monitoring of programs, provide additional recommendations on indicators, and give PEPFAR country teams increased flexibility to work within the context of the national system. These indicators are not subject to audit. 2 Programmatic refers to PEPFAR indicators that only have ‘direct’ counts. Country-level refers to PEPFAR indicators that have ‘direct’ and ‘indirect’ counts. Phase 1 had only seven country-level indicators. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
APPENDIX C: METHODS C-11 regional, or local systems strengthening activities, all resulting to the PEPFAR contributions to the national program (total)” (OGAC, 2011). OGAC issued an indicator guidance document for FY 2006 reporting and revised indicator guidance for FY 2008 reporting, which did not change the number of essential indicators reported to OGAC but did include some new and revised indicators. (Note: No indicator guidance was made available for indicators reported during FY 2004 and FY 2005; for this evaluation the FY 2006 guidance definitions were referenced for interpretation of FY 2004 and FY 2005 indicators.) TABLE C-1 Country-Level Indicators Reported During FY 2004–FY 2009 Indicator Number Indicator Label Prevention 1.2 1.3 Care 6.2 7.2 8.1 9.2 Treatment 11.4 In 2009, OGAC developed the Next Generation Indicators (NGIs) to reduce the number of PEPFAR-specific reporting requirements and, where possible, to align with globally harmonized and reported indicators in partner countries (OGAC, 2009). The NGIs “reflect PEPFAR’s strategy to increase country ownership of HIV/AIDS efforts and ensure that partner countries and regions are at the center of decision-making, leadership, and management of their HIV/AIDS programs” (OGAC, 2011). As a result, the number of “essential reported” indicators decreased, and countries are now only required to report 25 programmatic indicators to OGAC. If a partner country has a signed Partnership Framework, the country is required to report a 26th programmatic indicator to OGAC (H6.1.D). Five additional indicators are routinely reported to OGAC but these are national-level indicators and are not PEPFAR-specific. In addition, many of the indicators corresponding to care and support were redefined to make definitions easier to understand and information easier to collect. The NGIs consist of eight primary indicators, a subset of the 25 programmatic indicators required to be reported to OGAC (see Table C-2). Table C-3 presents how the overlapping country-level indicators before and after the NGI revision map to one another. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-28 EVALUATION OF PEPFAR systematically interview stakeholders serving in particular roles in every country visited. The roles of these individuals or groups of individuals included U.S. Country Mission leader(s) PEPFAR all-staff mission team members PEPFAR country coordinator or equivalent PEPFAR technical staff/ work group members (with varying numbers of staff and kinds of workgroups depending on the country) Partner country stakeholders including, at minimum, health- and HIV-related government or government agency personnel, partner country HIV-related NGO directors, staff, volunteers, and others. U.S.-based stakeholders including, at minimum, HIV-related NGO directors/chiefs of party, staff, or volunteers, and others HIV-related civil society organization leaders or members, including faith-based programs, human rights programs, and other organizations that provided programs and services for populations at elevated risk or other vulnerable populations Personnel having direct experience with a particular focus for data collection within a given country (e.g., services for populations at elevated risk, services for orphans and vulnerable children, the health care system, PMTCT, or others). Such personnel might be from different levels of a partner country government, NGO, and U.S.-based NGOs or other organizations. In any case, interviewee selection was based on those who had the most direct experience with the area of focus and who were voluntarily willing to be interviewed. For example, interviewees who directed or provided services or programs that addressed the needs of HIV- related vulnerable children were sought in countries designated as target countries for such information. Based on an IOM Institutional Review Board review and exemption, for the protection of their personal health information, individual beneficiaries were not included in the interview sample, except when individuals serving in one of the roles described above were also beneficiaries of PEPFAR-supported programs (for example, some peer educators were also patients or clients of PEPFAR-supported programs). During the process of sampling and consequent data collection, team members assiduously protected individual’s confidentiality and anonymity. At no time did the evaluation team share with others external to the team the identity of anyone contacted by team members, scheduled for an interview, or interviewed. Team members provided no feedback to individuals who proffered the names of potential interviewees and, with the exception of drivers and interpreters hired by the team, divulged no information of any kind to anyone external to the team. Thus, the team kept the identity of interviewees confidential during all country visits. Over the 13 country visits, the IOM delegations conducted a total of 383 interviews; 68 of these included a visit to a service delivery facility or program site. The interviewees included individuals or groups representing partner country government, U.S. government mission staff including the Department of State and the PEPFAR implementing agencies, multilateral organizations, nongovernmental organizations (NGOs), academia, and the private sector. Table C-4 summarizes the number of total interviews completed by stakeholder type and subtype. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
APPENDIX C: METHODS C-29 TABLE C-4 Country Visit Interviews by Stakeholder Type Stakeholder Type and Sub-Type Number of Interviews U.S. Government Stakeholders 147 Mission Leadership In-Briefings and Exit Meetings 26 PEPFAR All-Staff Mission Team Briefings 16 PEPFAR Country Coordinator 13 Agency Leadership 25 Technical Staff and Working Groups 67 U.S.-Based Stakeholders with Operations in Partner Country 62 NGO 41 Academia 11 Private Sector (for-Profit) 10 Partner Country Stakeholders 156 Government, National 53 Government, Subnational (Province, District, Facility) 40 NGO 51 Academia 6 Global Fund CCM 4 Private Sector (for-Profit) 2 Other Stakeholders 16 NGO (Other Country-Based) 4 Other Bilateral Government Donors 1 Multilateral 11 Mixed (Stakeholders from USG, Mutilateral Organizations, Other 2 Bilateral Donors, Partner Country Government, US Private Sector) TOTAL 383 NOTE: This does not represent the total number of interviewees, as the majority of interviews were with groups of interviewees. In some cases, the same interviewees participated in multiple interviews. For example, there was usually participant duplication between the PEPFAR all-hands interview and subsequent USG interviews. Repeat participation also happened occasionally across multiple interviews with partner country governments. In-country data collection As advocated in the conduct of credible qualitative studies (Creswall, 2007; Merriam, 2002; Patton, 2002), this evaluation involved the investigators in extended time in the field, which enabled in-depth data collection opportunities that extended well beyond a “snapshot view” (Rossman and Rallis, 2012) of the PEPFAR program. Prolonged engagement afforded the evaluation team an opportunity to gain an in-depth understanding of the PEPFAR program in the context of each country visited. Country visits typically spanned two weeks, and qualitative data collection encompassed an average of 25 in-depth interviews with key stakeholders as well as several site visits in each of the selected countries. Qualitative data collection during country visits involved the active participation of committee members. Each committee member actively participated in data collection and preliminary analysis processes for at least one visit to a partner country, with the majority engaging in 1-week data collection and analysis efforts while some participated in a full 2 weeks and a few participated in two country visits. At least one and sometimes both IOM study directors led the collection of data and data analysis processes during each country visit. Other PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-30 EVALUATION OF PEPFAR IOM research staff members also participated as delegation team members in collecting data and engaging in preliminary analysis during typically the full 2-week visit. Consultants participated in the data collection and analysis process for either 1 or 2 weeks during visits to 7 countries. Data collection toolkits were developed and provided for use by IOM evaluation teams on country visits. Additional preparation was provided through staff trainings, an initial briefing on methods and processes at a committee meeting, and an in-country orientation briefing at the beginning of each country visit. For each country visit delegation, briefing materials were prepared that included background information on the country, the national HIV response, and the PEPFAR country program, as well as basic financial, program monitoring, and surveillance data. A subset of evaluation team members was present and engaged in qualitative data collection typically for a period of 12-14 days during each visit to a partner country, with a total in-country visit time across the selected countries of about 180 days. Thus, the team had a total of approximately 6 months residence (or approximately 140 person-weeks for all of the committee, IOM staff, and committee consultants) in a total of 13 partner countries from the initiation of onsite data collection in November 2010 to its completion in February 2012. Each day in-country involved the evaluation team in some aspect of the evaluation process, including logistical planning for collecting data, as well as data collection, transcription of notes, team debriefings, and data analysis and interpretation. Through semi-structured interviews, delegation team members learned about the national HIV/AIDS response, interviewees’ experiences with PEPFAR, and the role of PEPFAR in the national response currently and over time. These interviews were conducted using interview guides tailored for each interview. Development of each guide was informed by interviewee role and level, agency type, and program area. Guides were developed by selecting and tailoring a subset of interview questions and follow-up prompts from a pre-established set of key country visit interview questions. Development of these pre-established questions was based on the Strategic Approach to the Evaluation (IOM and NRC, 2010) and the evaluation committee’s priority evaluation questions. The questions covered eight primary evaluation areas: Knowledge Management, Resource Flow, PEPFAR Implementation, Programmatic Context, PEPFAR Effects, Health System Strengthening, and PEPFAR Transition to Sustainability and Country Ownership, as well as opening and final questions. In addition to the questions, follow-up prompts were developed both within these evaluation areas and for the following programmatic areas: Prevention, Treatment, Care and Support, Laboratory, Children and Adolescents, and Gender. Interviews were conducted by an interview team generally comprised of a lead facilitator, co-facilitator, and note taker. The typical duration of an interview was 60-90 minutes. During the interview, the facilitator’s role was to build rapport and facilitate the interview using the semi- structured guide and the co-facilitator’s role was to ask follow-up or clarifying questions, serve as time keeper, write notes, and provide an end-of-interview review of the main points heard during the interview. The purpose of the co-facilitator’s summary of main points was to provide an opportunity for participant validation which is discussed below. Additionally, the review of main points served as a starting point for the peer debriefings described below. The note-taker’s role was to capture and record the data by means of handwritten or typed field notes as well as electronically, via an audio recorder, with interviewee consent. Additional team members who were present during an interview served as additional note takers; if fewer team members were present, one individual served as both note taker and either facilitator or co-facilitator. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
APPENDIX C: METHODS C-31 The evaluation delegation team participated in 68 field visits to a variety of sites in the 13 partner countries visited. The purpose of the site visits was to gain a contextual understanding of PEPFAR programs and resources. Each site visit included an interview, some of which were in- depth open-ended interviews while others were informal interviews conducted during walking tours of the sites. At least one designated delegate team member visiting the site took handwritten field notes during the visit; these were reviewed and reconciled by team members using the same procedures as interview notes. The team conducted the majority of interviews in English. In the case of interviewees who preferred conducting the interview in a language other than English, the delegation team hired professional interpreters from the partner country and oriented them to the purpose and process of qualitative data collection and their role in the process. Participant validation of data summaries A commitment to anonymity and confidentiality and a focus on cross-country data reporting precluded the sharing of country-specific findings with interviewees and their agencies or organizations during or after individual partner country visits. Interviewees were able to assess the scope and content of their key messages, however, in response to an end-of-interview summary of key messages that the co-facilitator offered at the conclusion of every interview. Following the summary of key points or messages, co-facilitators explicitly invited interviewees to convey any additions, corrections, or additional information that they wished to offer. Thus, all interviewees had an opportunity to affirm, modify, or extend their key messages, a process that not only affirmed that their viewpoints had been clearly understood and documented by the interview team, but also verified the accuracy and completeness of key messages shared with the team. Researcher reflexivity Because delegation team members served as “instruments” of qualitative data collection, they were aware of a need to be reflexive or “simultaneously aware of self and other and of the interplay between the two” (Rossman and Rallis, 2012). In other words, engagement in reflexivity facilitated individuals’ emergent self-awareness of personal predilections, assumptions, biases and beliefs so that each individual could potentially recognize and thus minimize her or his impact on interviewees and the research environment as well as the impact of the research environment on them (Patton, 2002). Team leaders and consultants urged all team members to engage in reflection and reflexivity throughout the evaluation by using at least one of two primary strategies: maintaining a private reflective and reflexivity journal or engaging in verbal reflexivity during any of the interview or team debriefings. Members of the evaluation team frequently, openly, and voluntarily shared their self-awareness of personal assumptions, biases, and beliefs verbally during one or more of the multiple peer debriefings and synthesis processes associated with data collection. At times, peers encouraged a team member to be reflexive when that individual’s personal assumptions or biases emerged during discussions and debriefings related to the evaluation. During discussions, it was not unusual for team members to reference a personal need for self-reflexivity regarding some topic. Thus, the need for all investigators to become increasingly self-aware about their personal beliefs, assumptions, values, and biases that could impact the research or the research environment and vice versa was frequently reinforced during every country visit. Audit trail The maintenance of an audit trail served as a means by which the evaluation team sought to establish study credibility and confirmability (Wolf, 2003). Evaluation team members PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-32 EVALUATION OF PEPFAR were charged with organizing and maintaining various electronic and hardcopy audit trail evidence related to the evaluation. Evidentiary documents related to the process of the evaluation included the following: An agenda log maintained electronically for each country visit chronicled interview scheduling and contact information, evaluation-related contacts, and information on the participants and questions covered in completed interviews. An activity log maintained electronically throughout the evaluation process chronicled process and methodological decisions and action items both within and across country visits. Analysis and interpretation notations were indicated on flip chart paper and electronic notes during facilitated team debriefings and the mid-week and exit synthesis process. When evaluation team members recounted interviewees’ viewpoints and experiences related to evaluation topics, they not only reported the content of interviewees’ perspectives (“what they said”) gleaned during interviews, but also differentiated interviewees’ narratives from how they as team members interpreted what interviewees shared with them. Team members also discussed emerging linkages among participants’ interview data and other data such as documents and observations. A codebook was initially developed and then revised based on evaluation topics and, to a lesser degree, data that emerged from the interviews and site visits. The codebook fostered team members’ ability to consistently label or code segments of the narrative data. Evaluation team debriefings The evaluation team engaged in a multistage process of data debriefings that were instrumental in verifying and communicating interview content, facilitating reflection and personal reflexivity, and synthesizing data findings according to evaluation topic. The types and content of the peer debriefings are outlined below: Individual Interview Debriefings o Using the co-facilitator’s end-of-interview summary as a basis, interview team members’ documented interviewees’ key points or messages, reflected on the interview process, engaged in and acknowledged personal reflexivity, and participated in a preliminary analysis and interpretation of the data collected during the given individual in-depth interview or group interview. Daily or Every-Two-Day Interview Debriefings o All team members convened to share key points that emerged from the interviews of which they were a part, their perspectives about and interpretations regarding the data, and their personal reflections/reflexivity. Synthesis (End of Week 1) o All team members engaged in a midpoint synthesis of interview findings, the process of which was facilitated by the team leader and structured according to evaluation topic. o Committee members often participated during the first week of the 2-week country visits. The end-of-week 1 synthesis process was thus critical in PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
APPENDIX C: METHODS C-33 eliciting committee member insights into country visit data and interpretation before they exited the country. Exit Synthesis (End of Week 2) o All team members engaged in an 8 to 10 hour process of verbally synthesizing the findings associated with data collection prior to exiting the country. Each team member received a copy of the interview debriefings that had been conducted. Similar to the end-of-week 1 synthesis process, the team leader facilitated the exit synthesis process that was structured according to evaluation topics and that included data documentation, reflection and reflexivity regarding the data collected, and verbal analysis and interpretation notations. Across country debriefings and discussions o Periodically in between clusters of country visits, IOM study staff participated in a discussion and synthesis of the qualitative findings according to evaluation topic and identified consonance or differences in these findings across a number of countries. o At committee meetings that occurred periodically in between clusters of country visits, committee members, either as a whole committee or in working groups focused on specific content areas, participated in discussions of the analysis and interpretation of interview data, including review of draft data presentations. Accuracy of data collection Critical in this evaluation was accuracy in documenting the data collected. With participants’ permission, interviews were digitally recorded in conjunction with handwritten notes taken by members of the interview team. Professional transcriptionists ultimately transcribed the digitally recorded interviews, but the need for timeliness, efficiency, ease of comprehension, and engagement in data analysis from the onset of data collection led the evaluation team to rely on their own typed transcription of handwritten interview notes as the primary source of interview data for analysis. Interview team members later engaged in an independent, detailed review and edification of the note-taker’s transcribed handwritten notes to ensure completeness and accuracy which involved an initial round by the assigned note taker, a second round by another team member who participated in the interview, and a final resolution round by the original note taker. During the end-of-interview summary provided by the co-facilitator (or the facilitator when there was no co-facilitator), interviewees addressed the accuracy of main end-of interview points that co-facilitators shared with them by affirming, correcting, or adding to the end-of- interview summary. In addition, the interview team debriefed each interview shortly after it occurred to affirm the accurate documentation of main points using the co-facilitators’ potentially revised summary as a foundation and contributing additional details. An additional accuracy check was afforded team members who could reference the digital recording of the interview when clarifying segments of narrative or resolving issues of disagreement regarding the content of a particular interview. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-34 EVALUATION OF PEPFAR FIGURE C-1 Country visit qualitative data collection process PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-35 EVALUATION OF PEPFAR Collection of Non-Country Visit Interview Data As part of the data collection effort for the evaluation, IOM staff and consultants also conducted a series of 32 non-country visit interviews with key stakeholders. These interviewees included the U.S. government at PEPFAR headquarters level (including OGAC, CDC, and USAID) and U.S.-based implementing partners at headquarters level, as well as other organizations who work in the global response to HIV, including multilateral organizations, nongovernmental organizations, and another bilateral donor. As with the country visit interviews, non-country visit interviewees were selected through purposeful sampling, prioritized on the basis of targeted focus areas within the evaluation, and on the process of mapping data sources for evaluation questions. Semi-structured interviews were conducted using the same methodology as the country visit interviews, using interview guides with questions and prompts adapted as appropriate for each interview. Analyses of Qualitative Data In-country data analysis process In-country data review and preliminary analysis occurred at various levels and several times during country visits. As soon as possible after each interview, team members conducted a post-interview debriefing to discuss and document the main points shared by the interviewee(s). Delegation members also convened routinely as an entire team during the country visit to engage in debriefings to share with each other the main points from the data across all the interviews that were conducted. At the close of the first week of each 2-week country visit and again at the close of the country visit, the team conducted an end-of-week debriefing and exit synthesis debriefing (respectively) that utilized an inductive analysis approach for the purpose of identifying dominant themes that emerged from the data. Both of these processes began with delegation members individually reading the debriefing notes from interviews conducted during the week to review key data from the interviews and to identify concepts and themes emerging from the data. Delegation members then collectively discussed the data and dominant themes that arose from the interviews, systematically using categories that were pre-selected based on the evaluation objectives. The delegation team differentiated between evidence (E), the responses heard during the interviews, and analysis and interpretation (A&I), which reflected the delegation’s interpretations of what the evidence meant, focusing on the meaning in relation to the evaluation objectives. The output from these processes was an Exit Synthesis document capturing the key evidence and analysis and interpretation from the interviews grouped by evaluation category, and a Key Messages document capturing the main themes that emerged across the interviews. These documents were then included as part of the country visit summary, which was reviewed by the members of the trip delegation and then posted on the committee portal. The country visit summary also includes other information provided to the delegation in advance of the trip in the form of a country brief, including background research on the country context and PEPFAR program as well as basic financial data and OGAC and other programmatic or indicator data (including UNAIDS data). The country visit summary is a compilation of the data from these multiple sources, not a triangulated analysis of the data and evidence available for each country. The goal was to provide a “snapshot” overview to inform the rest of the committee about the visit and the country and provide a centralized source for country data. Synthesis of exit syntheses To provide the committee with a sense of the overall current findings emerging from the interview data, for some of the evaluation categories the IOM staff PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-36 EVALUATION OF PEPFAR and consultant conducted a synthesis to identify and present the dominant themes that emerged in the exit syntheses across countries. This synthesis was conducted and presented in a variety of ways, ranging from analytical synthesis presented in narrative form to data grouped in bulleted form by sub-themes, which offers less synthesis and analysis but is closer to the “raw” data. Additional analysis of interview data Additional data summaries, syntheses, and analyses from both the country visit and non-country visit interview data were generated using methods detailed below. Members of the IOM staff used NVivo software (version 9.0) to conduct macro-level coding of the data using detailed interview notes generated by IOM staff and consultants or transcripts generated by contracted professional transcriptionists using audio recordings of interviews. The subset of data coded in NVivo comprised more than half of the interviews, purposefully selected for representation across countries and stakeholder types. This coding was based on a standardized project code book with each code reflecting important data concepts with inclusion and exclusion criteria. The data concepts represented in the code book were based on the Strategic Approach (IOM and NRC, 2010), the evaluation committee’s development of priority evaluation questions, and the exit synthesis process and review of initial data collected from the pilot country visits and other early country visits. For synthesis and analysis, these coded data were separated and extracted by querying for a single code or combinations of the macro-level codes across interviews. In some cases, data were also extracted from the NVivo dataset using targeted word search queries. Building on this initial thematic identification, IOM staff or consultants then conducted a more in-depth and refined analysis through repeated reading, reflection, and continued micro- level coding of the data for narrower subconcepts. This led to inductive identification of themes, patterns, and categories that emerged as findings from the data. This was followed by deductive confirmation and disconfirmation of those findings and determinations of data saturation for topics and themes (i.e., whether any new data has emerged). Prolonged engagement in data collection also led team members to affirm data saturation. Delegation evaluation teams recognized data saturation, through multiple iterations of individual and group analyses and discussions described below, as the repetition of information to the point of redundancy (Merriam, 2002; Patton, 2002), which thus indicated that data collection could be reasonably concluded (Lincoln and Guba, 1985). As the next iteration of the analytical process, drafts of data analysis outputs were read for discussion and revision by members of the project staff, consultants, and evaluation committee members who were familiar with the interview data and had participated in data collection and in-country data analyses. In addition, interview debriefing and exit synthesis documents from all interviews, including those not in the initial coded dataset, were used to carry out supplementary deductive confirmation and disconfirmation of findings that emerged from the coded data, and to identify specific additional interview notes and transcripts for enrichment of the analysis of coded data. These interview data findings and analyses were presented in a number of ways, including in narrative form with accompanying illustrative quotations, in summary tables, or in bulleted groupings by subconcepts. The presentation of quotes was used when one person’s words provided a memorable description of an issue that was resonant with multiple interviewees or perspectives, or in some cases when one person’s words represent a meaningful disconfirming perspective. Single quotations denote an interviewee's perspective with wording PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
APPENDIX C: METHODS C-37 extracted from transcribed notes written during the interview. Double quotations denote an exact quote from an interviewee either confirmed by listening to the audio-recording of the interview or extracted from a full transcript of the audio-recording. Interview data presented in the report are accompanied by a citation key. Typically interviews in qualitative research are cited with a brief descriptive demographic phrase; however, this was not feasible for an evaluation of this scope, with more than 400 interviews and the frequent citations for multiple interviews. Therefore, a citation tag was developed to allow the reader to identify the key characteristics relevant for the analysis and interpretation of the data for this evaluation, including the range of countries and interviews represented and the stakeholder type. The interview citation key is shown in Box C-1. BOX C–1 Interview Citation Key Country Visit Exit Synthesis Key: Country # + ES Country Visit Interview Citation Key: Country # + Interview # + Organization Type Non-country Visit Interview Citation Key: “NCV” + Interview # + Organization Type Organization Types: United States: USG = US Government; USNGO = US Nongovernmental Organization; USPS = US Private Sector; USACA = US Academia Partner Country: PCGOV = Partner Country Government; PCNGO = Partner Country NGO; PCPS = Partner Country Private Sector; PCACA = Partner Country Academia Other: CCM = Country Coordinating Mechanism; ML = Multilateral Organization; OBL = Other (non-US and non-Partner Country) Bilateral; OGOV = Other Government; ONGO = Other Country NGO PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 605
C-38 EVALUATION OF PEPFAR REFERENCES Bendavid, E., and J. Bhattacharya. 2009. The President's Emergency Plan for AIDS Relief in Africa: An Evaluation of Outcomes. Annals of Internal Medicine 150(10):688-695. Bendavid, E., C. B. Holmes, J. Bhattacharya, and G. Miller. 2012. HIV development assistance and adult mortality in Africa. JAMA 307(19):2060-2067. Bouey, Paul and De Leon, Jordana. Personal Interview. 27 April 2011. Caulley, D. N. 1983. Document Analysis in Program Evaluation. Evaluation and Program Planning 6:19-29. CGD. 2008. Pepfarfundingdata-fulldataset.xls. http://www.cgdev.org/section/initiatives/_active/hivmonitor/funding_data/pepfardata (accessed July 31, 2012). Creswall, J. W. 2007. Qualitative Inquiry and research Design: Choosing Among five Approaches. Thousand Oaks, CA: SAGE. Denzin, N. K. 1978. The Research Act: A Theoretical Introduction to Sociological Methods. 2nd ed. New York: McGraw-Hill. IOM, and NRC. 2010. Strategic Approach to the Evaluation of Programs Implemented under The Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008. Washington, DC: The National Academies Press. Lincoln, Y. S., and E. G. Guba. 1985. Naturalistic Inquiry. Thousand Oaks, CA: SAGE. McCullough, R., and M. L. 2011. Surveying the Global HIV Landscape. http://ftguonline.org/ftgu- 232/index.php/ftgu/article/view/2050/4096 (accessed June 10, 2011). Merriam, S. B. 2002. Assessing and Evaluating Qualitative Research. Edited by In Sharon B. Merriam and Associates, Qualitative Research in Practice. San Francisco, CA: Josey-Bass. OGAC. 2009. The President’s Emergency Plan for AIDS Relief: Next Generation Indicators Reference Guide. Version 1.1. OGAC: Washington, DC. ———. 2011. IOM Request for PEPFAR Results. Washington, DC. Patton, M. Q. 2002. Qualitative Research and Evaluation. 3rd ed. Thousand Oaks, CA: SAGE. PEPFAR. 2012. Obligation and Outlay Reports. http://www.pepfar.gov/about/c24880.htm (accessed November 29, 2012). Rossman, G. B., and S. Rallis. 2012. Learning in the Field: An Introduction to Qualitative Research. Thousand Oaks, CA: SAGE. Sessions, M. 2011. Overview of the President's Emergency Plan for AIDS Relief (PEPFAR). http://www.cgdev.org/doc/HIVAIDSMonitor/OverviewPEPFAR.pdf (accessed June 10, 2011). Wolf, Z. R. 2003. Exploring the Audit Trail for Qualitative Investigations. Nurse Educator 28(4):175- 178. PREPUBLICATION COPY: UNCORRECTED PROOFS