Strategic Approach to the Evaluation of Programs Implemented Under the Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008 (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix D Glossary Source Responsibility for the use of resources and the (UNAIDS MERG, Accountability decisions made, as well as the obligation to 2010, p. 2) demonstrate that work has been done in compliance with agreed-upon rules and standards and to report fairly and accurately on performance results vis-à-vis mandated roles and/or plans. Actions taken or work performed through which (UNAIDS MERG, Activities inputs such as funds, technical assistance, and other 2010, p. 2) types of resources are mobilized to produce specific outputs. The extent to which countries or people can bear the Adapted from Affordability cost of programs or services. (Merriam-Webster, 2010) The ascription of a causal link between observed (UNAIDS MERG, Attribution changes and a specific intervention. 2010, p. 2) A systematic process for evaluating the products, Adapted from Benchmarking services and work processes of organizations that are (Spendolini, 1992, recognized as representing best practices for the p. 2 ) purpose of organizational improvement. A methodological approach that describes a situation, (UNAIDS MERG, Case study individual, or the like and that typically incorporates 2010, p. 2) data-gathering activities (e.g., interviews, observations, questionnaires) at selected sites or programs/projects. Case studies are characterized by purposive selection of sites or small samples; the 163 

164 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS expectation of generalizability is less than that in many other forms of research. The findings are used to report to stakeholders, make recommendations for program/project improvement, and share lessons learned. A series of statements that link the causes of a (Belausteguigoitia, Causal Chain problem with its effects (i.e., a causal chain maps the 2004, p. 2) relations between inputs, activities, outputs, outcomes and impact). When HIV actively multiplies, it infects and kills CD4 (CDC, 2008, p. 13) CD4 Count T cells, a specific type of white blood cell, that are the immune system’s key infection fighters. The effects of HIV are measured by the decline in the number of CD4 cells. The CD4 count is the number of CD4 cells in the blood and reflects the state of the immune system. The normal count in a healthy adult is between 600 and 1,200 cells/mm3. The extent to which donors implement, where (High-level Forum, Coordination & feasible, common arrangements at country level for 2005, p. 6) Harmonization planning, funding (e.g., joint financial arrangements), disbursement, monitoring, evaluating and reporting to government on donor activities and aid flows." A summary measure of linear association between two (Woodward, 2005, Correlational quantitative variables. The correlation coefficient p. 456–457) analysis takes the value-1 for perfectly negatively correlated data. For perfectly positively correlated data it takes the value unity. The nearer to zero is the correlation coefficient, the less linear the association there is between the two variables. A measure of how economically resources/inputs (DAC Network on Cost- (funds, expertise, time, etc.) are converted to results. Development effectiveness/ Evaluation, 2002, Efficiency p. 21) The ability of government, the private sector, and civil Partially adapted Country society to “plan, manage, implement, and account for from Capacity results of policies and programs.” (High-level Forum, 2005, p. 5) 

165 APPENDIX D A situation in which “partner countries exercise Partially adapted Country effective leadership over their development policies from Ownership and strategies, and coordinate development actions.” (High-level Forum, Partner countries expect donors to respect country 2005, p. 3; Accra priorities and to invest in country human resources Agenda for Action, and institutions with the goal of improving or 2008) maximizing the use of the country’s systems to deliver aid for more rapid effectiveness with developmental aid “[…] to achieve their own economic, social, and environmental goals.” As a part of country ownership, partner countries will be more transparent and accountable to donors, their governing bodies, and their citizens by translating their actions into positive impacts on the lives of their citizens and populations. The extent to which a program/intervention is being (UNAIDS MERG, Coverage implemented in the right places (geographic coverage) 2010, p. 2) and is reaching its intended target population (individual coverage). An intensive literature search, review and synthesis of (Association of Document all relevant documents concerning the program being Qualitative review evaluated. In the case of HIV programs, documents to Research, 2010a; be considered may typically include national Operario, 2008, epidemiological reports, official government or other p. 13) state-level public health and human rights policy reports, non-governmental organization reports, and any other academic or scientific papers related. Findings are usually integrated into the overall findings of the study and/or used to help design a main project. The extent to which a program/intervention has (UNAIDS MERG, Effectiveness achieved its objectives under normal conditions in a 2010, p. 2) real-life setting. Results or changes from the program such as changes (Taylor-Powell and Effects in knowledge, awareness, skills, attitudes, opinions, Henert, 2008) aspirations, motivation, behavior, practice, decision making, policies, social action, condition, or status. Effects may be intended and/or unintended: positive and negative. Effects fall along a continuum from proximal (immediate; initial; short-term) to distal effects (ultimate; long-term), often synonymous with impact. 

166 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS The extent to which an intervention produces the (UNAIDS MERG, Efficacy expected results under ideal conditions in a controlled 2010, p. 3) environment. A measure of how, economically, inputs (resources (UNAIDS MERG, Efficiency such as funds, expertise, time) are converted into 2010, p. 3) results. The study of the magnitude, distribution and (UNAIDS MERG, Epidemiology determinants of health-related conditions in specific 2010, p. 3) populations, and the application of the results to control health problems. The rigorous, scientifically-based collection of (UNAIDS MERG, Evaluation information about program/intervention activities, 2010, p. 3) characteristics, and outcomes that determine the merit or worth of the program/intervention. Evaluation studies provide credible information for use in improving programs/interventions, identifying lessons learned, and informing decisions about future resource allocation. A survey of a representative sample of facilities that (UNAIDS MERG, Facility Survey generally aims to assess the readiness of all elements 2010, p. 3) required to provide services and other aspects of quality of care (e.g., basic infrastructure, drugs, equipment, test kits, client registers, and trained staff). The units of observation are facilities of various types and levels in the same health system. The content of the survey may vary but typically includes a facility inventory and, sometimes, health worker interviews, client exit interviews, and client-provider observations. The stage of a research project in which data is (Association of Field Work collected, whether in the form of interviews, group Qualitative discussions, observations, or materials for cultural Research, 2010b) analysis. Qualitative methodology used to obtain information (McKenzie, 2005, Focus Group about the feelings, opinions, perceptions, insights, p. 76) beliefs, misconceptions, attitudes and receptivity of a group of people concerning an idea or an issue. They include only 8–12 participants relatively homogeneous though unfamiliar to each other. They share in a guided discussion which is informal and last approximately 2 hours. 

167 APPENDIX D The extent to which findings can be assumed to be (UNAIDS MERG, Generalizability true for the entire target population, not just the 2010, p. 3) sample of the population under study. Note: To ensure generalizability, the sampling procedure and the data collected need to meet certain methodological standards. A computer mapping and analysis technology (O’Carroll, 2003, Geographic consisting of hardware, software, and data allowing p. 432) Information large quantities of information to be viewed and System (GIS) analyzed in a geographic context. It has nearly all of the features of a database management system, with a major enhancement. Every item of information in a GIS is tied to a geographic location. “Health workers are all people engaged in actions (WHO, 2007, p. Health whose primary intent is to protect and improve health. 16; IOM, 2008, Workforce A country’s health workforce consists broadly of p. 2) Strengthening health service providers and health management and support workers. This includes private as well as public sector health workers; unpaid and paid workers; lay and professional cadres. Countries have enormous variation in the level, skill and gender-mix in their health workforce. Overall, there is a strong positive correlation between health workforce density and service coverage and health outcomes.” Strengthening “involves the improvement in a range of capacities including training, supervision and job satisfaction.” A data system, usually computerized, that routinely (UNAIDS MERG, Health collects and reports information about the delivery and 2010, p. 3) Information cost of health services, and patient demographics and Systems health status. All organizations, people and actions whose primary Partially adapted Health System intent is to promote, restore or maintain health. WHO from (WHO, 2007, divided health systems into six operational “building pp. v & 2) blocks” action framework: services, workforce, information, commodities and technologies, financing, and leadership and governance. Proportion of children that are alive and free of HIV- Adapted from HIV-Free Infant infection, which can be measured at 18 months of age. (Stringer et al., Survival The denominator of this outcome measure consists of 2008) the number of children born within the past two years, estimated through the survey component of the 

168 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS Demographic and Health Survey. The numerator is based on the same figure, minus the number of children found to be HIV-infected [determined via HIV antibody and deoxyribonucleic acid polymerase chain reaction testing] and the number of children reported to have died (derived via survey methodology). The long-term, cumulative effect of (UNAIDS MERG, Impact programs/interventions over time on what they 2010, p. 3) ultimately aim to change, such as a change in HIV infection, AIDS-related morbidity and mortality. Note: Impacts at a population-level are rarely attributable to a single program/intervention, but a specific program/intervention may, together with other programs/interventions, contribute to impacts on a population. A type of evaluation that assesses the rise and fall of (UNAIDS MERG, Impact impacts, such as disease prevalence and incidence, as 2010, p. 3) Evaluation a function of HIV programs/interventions. Impacts on a population seldom can be attributed to a single program/intervention; therefore, an evaluation of impacts on a population generally entails a rigorous design that assesses the combined effects of a number of programs/interventions for at-risk populations. The number of new cases of a disease that occur in a (UNAIDS MERG, Incidence specified population during a specified time period. 2010, p. 4) A quantitative or qualitative variable that provides a (UNAIDS MERG, Indicator valid and reliable way to measure achievement, assess 2010, p. 4) performance, or reflect changes connected to an intervention. Note: Single indicators are limited in their utility for understanding program effects (i.e., what is working or is not working, and why?). Indicator data should be collected and interpreted as part of a set of indicators. Indicator sets alone cannot determine the effectiveness of a program or collection of programs; for this, good evaluation designs are necessary. The financial, human, and material resources used in a (UNAIDS MERG, Inputs program/intervention. 2010, p. 4) 

169 APPENDIX D A specific activity or set of activities intended to bring (UNAIDS MERG, Intervention about change in some aspect(s) of the status of the 2010, p. 4) target population (e.g., HIV risk reduction, improving the quality of service delivery). Management tool used to improve the design of (UNAIDS MERG, Logic interventions. It involves identifying strategic 2010, p. 4) Framework elements (inputs, outputs, activities, outcomes, impact) and their causal relationships, indicators, and the assumptions of risks that may influence success and failure. It thus facilitates planning, execution, and monitoring and evaluation of an intervention. A multi-year implementation strategy for the (UNAIDS MERG, M&E collection, analysis and use of data needed for 2010, p. 4) (Monitoring and program/project management and accountability Evaluation) purposes. The plan describes the data needs linked to a Strategy specific program/project; the M&E activities that need to be undertaken to satisfy the data needs and the specific data collection procedures and tools; the standardized indicators that need to be collected for routine monitoring and regular reporting; the components of the M&E system that need to be implemented and the roles and responsibilities of different organizations/individuals in their implementation; how data will used for program/project management and accountability purposes. The plan indicates resource requirement estimates and outlines a strategy for resource mobilization. Note: A national HIV M&E plan is a multi-sectoral, 3–5 year implementation strategy which is developed and regularly updated with the participation of a wide variety of stakeholders from national, sub-national, and service delivery levels. Mathematical analysis that describes the association (Woodward, 2005, Modeling between exposure, outcome and confounders. p. 427) Numerous models have been developed and their use depends on a certain set of assumptions. It is generally use when it is necessary to control for many confounding variables. Routine tracking and reporting of priority information (UNAIDS MERG, Monitoring about a program/project, its inputs and intended 2010, p. 4) outputs, outcomes and impacts. 

170 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS For service delivery programs, the “systematic and (Adapted from Operational objective assessment of the availability, accessibility, UNAIDS MERG, Research quality, and/or sustainability of services designed to 2010, p. 5) improve service delivery. It assesses only factors that are under the control of program/project managers, such as improving the quality of services, increasing training and supervision of staff members, and adding new service components.” Protozoan, bacterial, fungal, and viral infections that (Adapted from Opportunistic are more frequent or more severe because of CDC, 2008, pp. Infection immunosuppression in people with HIV infection or 13–14) AIDS. When the CD4 count of an adult falls below 200 cells/mm3, the risk of opportunistic infection is high. Short-term and medium-term effect of an (UNAIDS MERG, Outcome intervention’s outputs, such as change in knowledge, 2010, p. 5) attitudes, beliefs, behaviours. The results of program/intervention activities; the (UNAIDS MERG, Outputs direct products or deliverables of program/intervention 2010, p. 5) activities, such as the number of HIV counseling sessions completed, the number of people served, the number of condoms distributed. A set of procedures used by planners/evaluators to try (McKenzie, 2005, Pilot Test out various processes during program development on p. 123) a small group of subjects prior to actual implementation. A type of survey which is statistically representative (UNAIDS MERG, Population- of the target population, such as the AIDS Indicator 2010, p. 5) based Survey Survey and the Demographic and Health Survey. The total number of persons living with a specific (UNAIDS MERG, Prevalence disease or condition at a given time. 2010, p. 5) An overarching national or sub-national response to a (UNAIDS MERG, Program disease. A program generally includes a set of 2010, p. 5) interventions marshaled to attain specific global, regional, country, or sub-national objectives; involves multiple activities that may cut across sectors, themes and/or geographic areas. 

171 APPENDIX D “A study that intends to control a health problem or (UNAIDS MERG, Program improve a public health program or service. The 2010, p. 6) Evaluation intended benefits of the program are primarily or exclusively for the study participants or the study participants’ community (i.e., the population from which the study participants were sampled); data collected are needed to assess and/or improve the program or service, and/or the health of the study participants or the study participants’ community. Knowledge that is generated does not typically extend beyond the population or program from which data are collected.” Also referred to as summative evaluation. Data collected using qualitative methods, such as in- (Adapted from Qualitative Data depth, open-ended interviews, direct observation, and UNAIDS MERG, analysis of written documents. Qualitative data can 2010, p. 6; Patton, provide an understanding of social situations and 2002, p. 4, 5) interactions, as well as people’s values, perceptions, motivations, and reactions. Qualitative data are generally expressed in narrative form, pictures or objects (i.e., not numerically). When using mixed methods, “findings may be presented either alone or in combination with quantitative data.” There are many different theoretical perspectives for qualitative inquiry, but they all attempt to describe or explain phenomena. Include individual and group approaches to data (Sankar et al., Qualitative collection. They feature in-depth and extended 2006, p.s55) Interview discussions guided by an interviewer. They vary in the Methods degree of structure involved. On one end of the continuum are semi-structured interviews using predefined questions that allow open-ended responses. Open-ended interviews are less structured and use a list of discussion topics to cover in each interview. Least structured are the informal interviews. Data collected using quantitative methods, such as (UNAIDS MERG, Quantitative surveys. Quantitative data are measured on a 2010, p. 6) Data numerical scale, can be analyzed using statistical methods, and can be displayed using tables, charts, histograms and graphs. The aim of a quantitative study is to classify features, count them, and construct statistical models in an attempt to explain what is observed. 

172 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS Study in which comparisons are made between Epidemiology Quasi- nonequivalent groups that are not randomly assigned GORDIS Experimental to intervention and control groups. Study Study where subjects are allocated to intervention and (Woodward, 2005, Randomized control groups according to some chance mechanism. p. 343) Controlled Trial A study which intends to generate or contribute to (UNAIDS MERG, Research generalizable knowledge to improve public health 2010, p. 6) practice, i.e., the study intends to generate new information that has relevance beyond the population or program from which data are collected. Research typically attempts to make statements about how the different variables under study, in controlled circumstances, affect one another at a given point in time. The outputs, outcomes, or impacts (intended or (UNAIDS MERG, Results unintended, positive and/or negative) of an 2010, p. 6) intervention. A person, group, or entity that has a direct or indirect (UNAIDS MERG, Stakeholder role and interest in the goals or objectives and 2010, p. 7) implementation of a program/intervention and/or its evaluation. A type of evaluation conducted at the end of an (UNAIDS MERG, Summative intervention (or a phase of that intervention) to 2010, p. 7) Evaluation determine the extent to which anticipated outcomes were produced. It is designed to provide information about the merit or worth of the intervention. The ongoing, systematic collection, analysis, (Adapted from Surveillance interpretation, and dissemination of data regarding a UNAIDS MERG, specific disease or behavior for use in public health 2010, p. 7) action to reduce morbidity and mortality and to improve health. These kinds of surveillance data can help predict future trends and target needed prevention and treatment programs. The continuation of benefits from a development (DAC Network on Sustainability intervention after major development assistance has Development been completed. Evaluation, 2002, p. 36) “A collation of all empirical evidence that fits pre- (Adapted from Systematic specified eligibility criteria to answer a specific Liberati, et al., Literature 

173 APPENDIX D research question. It uses explicit, systematic methods 2009, p. 2) Review that are selected with a view to minimizing bias, thus providing reliable findings from which conclusions can be drawn and decisions made. The key characteristics of a systematic review are: (a) a clearly stated set of objectives with an explicit, reproducible methodology; (b) a systematic search that attempts to identify all studies that would meet the eligibility criteria; (c) an assessment of the validity of the findings of the included studies, for example through the assessment of risk of bias; and (d) systematic presentation, and synthesis, of the characteristics and findings of the included studies.” These methods can be applied to various types of literature for research. Statistical techniques used for analyzing data over Adapted from Trend Analysis time. The primary step is to plot the observed numbers (Rosenberg, 1997) or rates of interest by study period, followed by further inspection of the data which provides the basis for subsequent analysis including data transformation and smoothing, and more complex statistical procedures (regression analysis, time series analysis). The analysis of data from three or more sources (Adapted from Triangulation obtained by different methods. Findings can be UNAIDS MERG, corroborated, and the weakness or bias of any of the 2010, p. 7, Patton, methods or data sources can be compensated for by 2002, p. 247.) the strengths of another, thereby increasing the validity and reliability of the results. Data triangulation, investigator triangulation, and theory triangulation are three other types of this analytic technique. While ideal, the technique can be expensive—“a study’s limited budget and time will affect the amount of triangulation that is practical.” The extent to which a measurement or test accurately (UNAIDS MERG, Validity measures what is intended to be measured. 2010, p. 7) GLOSSARY REFERENCES Association of Qualitative Research. 2010a. “Desk Research.” http://www.aqr.org.uk/glossary/?term=deskresearch (accessed June 7, 2010). Association of Qualitative Research. 2010b. “Fieldwork.” http://www.aqr.org.uk/glossary/?term=fieldwork (accessed June 7, 2010). Belausteguigoitia JC. 2004. Causal chain analysis and root causes: the GIWA approach. Ambio 33(1–2):7–12. 

174 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS CDC (Centers for Disease Control and Prevention). 2008. PMTCT training programme trainer manual. Module 1: Introduction to HIV/AIDS. Washington, DC: CDC. DAC (Development Assistance Committee) Network on Development Evaluation. 2002. Glossary of key terms in evaluation and results based management, evaluation and aid effectiveness. Paris: Organisation for Economic Co-operation and Development. High-Level Forum. 2005. Paris declaration on aid effectiveness: Ownership, harmonization, alignment, results, and mutual accountability, Paris, France, February 28-March 2. High-Level Forum. 2008. Accra agenda for action, Third High-Level Forum on Aid Effectiveness, Accra, Ghana, September 2–4. IOM (Institute of Medicine). 2008. Design considerations for evaluating the impact of PEPFAR. Washington, DC: The National Academies Press. Liberati, A., D. G. Altman, J. Tetzlaff, C. Mulrow, P. C. Gotzsche, J. P. Ioannidis, M. Clarke, P. J. Devereaux, J. Kleijnen, and D. Moher. 2009. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Med 6(7):e1000100. MacKenzie, J. F., B. L. Neiger, J. L. and Smeltzer. 2005. Planning, implementing & evaluating health promotion programs—A primer. 4ed. San Francisco, CA: Benjamin Cummings. Merriam-Webster. 2010. “Affordability.” http://mw2.merriam-webster.com/dictionary/affordability. (accessed June 16, 2010). O’Carroll, W. Yasnoff, and M. E. Ward. 2003. Public health informatics and information systems. New York: Springer-Verlag. Operario, D. 2008. Guideline for a qualitative research methodology to assess the social impacts of HIV and identify priorities for advocacy action. Bratislava, Slovakia: United Nations Development Programme Bratislava Regional Centre. Patton, M. Q. 2002. Qualitative research & evaluation methods, 3rd edition. Thousand Oaks, CA: Sage Publications, Inc. Rosenberg, D. 1997. Trend analysis and interpretation: Key concepts and methods for maternal and child health professionals. Rockville, MD: Health Resources and Services Administration. Sankar, A., C. Golin, J. M. Simoni, M. Luborsky, and C. Pearson. 2006. How qualitative methods contribute to understanding combination antiretroviral therapy adherence. J Acquir Immune Defic Syndr 43(Suppl 1):S54–S68. Spendolini, J. M. 1992. The benchmarking book. New York: American Management Association. Stringer, E. M., B. H. Chi, N. Chintu, T. L. Creek, D. K. Ekouevi, D. Coetzee, P. Tih, A. Boulle, F. Dabis, N. Shaffer, C. M. Wilfert, and J. S. Stringer. 2008. Monitoring effectiveness of programmes to prevent mother-to-child HIV transmission in lower-income countries. Bull World Health Organ 86(1):57–62. Taylor-Powell, E. and E. Henert. 2008. Developing a logic model: Teaching and training guide. Madison, WI: University of Wisconsin. UNAIDS MERG (United Nations Joint Programme on HIV/AIDS Monitoring and Evaluation Reference Group). 2010. Glossary: monitoring and evaluation terms. Geneva, Switzerland: UNAIDS. WHO (World Health Organization). 2007. Everybody’s business: Strengthening health systems to improve health outcomes-WHO‘s framework for action. Geneva, Switzerland: WHO. Woodward, M. 2005. Epidemiology—Study design and data analysis. 2nd ed. Boca Raton, FL: Chapman & Hall/CRC Press.