Measurement and Evaluation
The preceding chapters have outlined the scope and context of the global epidemic of cardiovascular disease (CVD), the extent of the resulting health and economic burden, and the challenge that lies ahead. This provides a compelling rationale for aggressively reducing risk factors that lead to CVD globally. Measurement is the basis for determining the scale of the global CVD epidemic and for understanding how best to intervene, and it will be critical to the success of efforts to reduce disease burden. While there is a need for CVD-specific measurement tools, existing global health efforts provide a robust foundation to draw upon and to avoid duplication as the global CVD community continues to develop and expand its evaluation of program and policy initiatives. Over the past several decades, advances in the field of global health have led to a wealth of measurement knowledge, tools, and techniques that have been developed for evaluating policy and program outcomes and impact on health status at all levels. Indeed, many of these national and global measurement initiatives are currently at risk of overlooking measurement of CVD and related chronic diseases, which will in fact be crucial in order to obtain a truly complete picture of national health needs.
This chapter1 first describes the functions and principles of measurement, monitoring, and evaluation. The chapter then addresses several critical cross-cutting considerations that affect measurement and evaluation. This is followed by a discussion of the potential for measurement ap-
proaches that can provide timely feedback and guide decision making at multiple levels to achieve reductions in cardiovascular disease, including a discussion of emerging technologies to improve measurement. Finally the chapter touches on the use of measurement at the global level to inform actions to reduce the burden of CVD.
FUNCTIONS AND PRINCIPLES OF MEASUREMENT
Measurement serves a number of critical roles in the effort to address any health problem. The use of measurement to inform the cycle of decision making in addressing a public health problem is outlined in Figure 4.1. This cycle applies to decision making at any level of stakeholder, from global to local, and at any scale of intervention, from a demonstration project to a global action plan. First, it is used to assess the magnitude of the problem at the level of the population and subpopulation and informs the mitigation of risk factors. When coupled with an assessment of capacity, these can inform priorities and the setting of realistic intervention goals. This in turn guides implementation of interventions, including policies, programs, and clinical interventions at the level of the population, the provider, and the individual. Measurement then can be used to assess the processes, outcomes, and impact of the implemented interventions. This feeds back into the cycle to encourage adaptations that help ensure sustainable progress. Thus, measurement is not simply an endpoint to determine the value of an intervention; it is also the foundation for an ongoing cycle of planning, prioritizing, and operationalizing interventions.
Ultimately, measurement strategies have the potential to lead to changes in health outcomes by changing the decisions and behavior of policy makers, providers, and individuals. This derives from the fundamental purpose of measurement: to create awareness that increases understanding and motivates change. In this way, as illustrated in Figure 4.2, measurement can be viewed as a critical component of any effort to result in an impact on health outcomes, serving to guide those efforts and to accelerate the pace of change to achieve the targeted outcomes. To serve as an instrument of change, measurement needs to be ongoing and cyclical. Transparent information can increase knowledge and change intentions throughout the process of implementing an intervention approach, just as it can lead to overall changes in baseline status and new policies or programs in response to achieving a new baseline.
A number of underlying principles drive measurement as a fundamental part of efforts to decrease CVD. First, in order to be effective measurement needs to be relevant to the context in which it is implemented (Majumdar and Soumerai, 2009). Contextual elements are typically local—occurring at the level of countries, regions, cities, and villages. Context includes local
elements of economics, financing, existing policies, existing capacity, population demographics, and social and cultural factors.
These in turn exist in a larger global context. A second principle is that measurement is most effective when it is transparent and when there are feedback mechanisms to ensure that the resulting data is widely available and widely used. Indeed, measurement alone is not sufficient—the data must actually be used by policy makers, providers, and individuals. Third, to truly document and maximize impact, measurement is needed at all levels, from individuals to providers to policy makers. Measurement is also needed across all kinds of interventions approaches, from clinical interventions and individual risk reduction to changes in the infrastructure to deliver interventions to policy changes and other population-based strategies. A fourth principle is that measurement needs to focus on the intermediate outcome of behavior change, for it is changes in the behavior of those at risk, of care providers, and of policy makers that will lead to lessening of the CVD burden. In order for measurement to be effective it must also be accurate, feasible, affordable, actionable, responsive, and timely (Majumdar and Soumerai, 2009). Finally, measurement outcomes should be able to be communicated clearly. Although there may be necessary complexity in the design of measurement systems, this complexity should be converted into relatively simple reporting of the data.
The number and variety of determinants that contribute to cardiovascular disease means that no single set of measures or data collection system will suffice for all goals or settings. Instead, this complexity necessitates the use of an array of measures and a variety of collection strategies, along with careful planning to set priorities for measurement and to balance feasibility with the need for comprehensive data that can be integrated and compared across countries, programs, and levels of measurement. As a final principle, it is critical in the planning and implementation of measurement strategies to make the following determinations:
who is expected to use the data;
what is to be measured;
what metrics or indicators should be used;
who will be collecting the data;
what tools will be used to collect the data;
who will analyze the data;
how the data will be reported and disseminated; and
how much it will cost to implement the measurement strategy and to analyze and disseminate the data.
CROSS-CUTTING CONSIDERATIONS IN MEASUREMENT
There are several critical cross-cutting considerations that affect measurement and evaluation that are important to discuss as the basis for interpreting the potential use of the methodologies described later in this chapter. These include standardization of indicators, data ownership and capacity for data analysis, and costs of measurement.
Standardizing Indicators for CVD Surveillance, Intervention Research, and Program Evaluation
To monitor the epidemic of CVD and ensure that there are effective intervention approaches that can be disseminated widely, it is critical to be able to gather data and make comparisons across countries, across sectors and systems, and across intervention and program evaluations. Therefore, while measurement efforts need to be tailored to the context, program, or intervention approach, some measurement strategies would benefit from standardization and global coordination of surveillance systems and evaluation systems.
The question of which indicators to use and how to prioritize them must be agreed upon by the relevant stakeholders in the international community. A number of key categories of metrics are crucial to measuring CVD and its breadth of determinants and would need to be considered. These include demographics; risk and risk mitigation including behaviors (e.g., smoking rates, physical activity, diet and nutrition) and biomedical measures (e.g., weight and height, blood pressure, cholesterol); disease outcomes (e.g., cardiovascular events); cause-specific mortality; health provider and quality improvement measures; health systems performance; economic measures; intersectoral policy measures (e.g., cigarette costs and sales data, agricultural trends, urbanization); and measures of global action. Some of these measures need to be disease specific, while others need to be harmonized and coordinated with measurement strategies for related chronic diseases and for other areas of health and development.
While there may already be consensus within a few of these indicator categories, far more are currently still being debated, and setting priorities within and across categories to balance comprehensive measurement with feasibility will not be simple. Although it was beyond the scope of this committee to do so, a minimum set of indicators with clear definitions with guidance on prioritization needs to be developed to allow for uniform and comparable data across countries and systems. Developing an indicator framework of this kind could be achieved through a consensus process involving key stakeholders such as researchers, practitioners, economists, funders, and representatives from national health and public health authorities from developing countries. This process would need to realistically con-
sider how to balance the need for comprehensive data collection with the practicalities of timeliness and resources. In addition, a critical component for any indicator framework is what the implementation and maintenance of each measurement system would cost. The World Health Organization (WHO) has convened an epidemiology reference group, drawing on headquarters and regional offices, to develop guidance for chronic disease surveillance systems and to agree on core indicators that will be used to monitor the major chronic diseases and their risk factors (Alwan, 2009, personal communication). If this effort takes into account the considerations described here, it could be a first step in achieving an implementable indicator framework.
This need for standardization and coordination has been recognized by the global HIV/AIDS community and is addressed in large part by the United Nations’ Joint Programme on HIV/AIDS (UNAIDS’s) Monitoring and Evaluation Reference Group (MERG) (UNAIDS, 2009a). Created in 1998 by the UNAIDS Secretariat, the MERG provides technical guidance for HIV monitoring and evaluation and is a key driver in the harmonization of HIV indicators at the global level (Global HIV M&E Information, no date). Working through a coordinated effort with individuals at the Global Fund and the U.S. President’s Emergency Plan for AIDS Relief, the MERG identified, collected, and defined high-quality indicators, making them freely accessible online (UNAIDS, 2009a, 2009b). In addition, while the indicator registry identifies which measures have been harmonized and endorsed by other stakeholders, it leaves the decision on determining the indicators that are most important to collect to the implementer, be it a national government or program manager (UNAIDS, 2009b). This use of online resources to lower the cost of use for developing countries as well as the leadership and coordination from a body with the capacity to also provide relevant technical support could provide a useful model for WHO during indicator standardization efforts for chronic diseases.
Once developed, coordinated support will be needed for the implementation of these globally comparable indicators. Technical assistance and training in surveillance, research, and evaluation will be needed to provide options for measurement tools that incorporate the uniform data from globally comparable indicators, but also to allow for national or local/ program-level choices on which tools to use and which indicators to collect (beyond the minimum set) based on local and project- or program-specific priorities, resources, and needs.
Data Ownership and Capacity for Data Analysis
The collection and reporting of data, regardless of how detailed, accurate, or comprehensive, is a potential waste of time and resources unless the information is appropriately processed, analyzed, and communicated
to relevant stakeholders. Currently there is a growing need to develop and maintain data analysis capacity at the local level, in an effort to help communities feel ownership of reported outcomes (Stansfield, 2009). Limits in local capacity to conduct both analysis and operations research have left some national governments hesitant to take on new measurement initiatives as they could overwhelm already fragile health information systems (Bennett et al., 2006). Thus, these absorptive capacity concerns must be kept in mind when determining how rapidly and to what degree to scale up measurement and evaluation initiatives.
Addressing these capacity needs will require a paradigm shift at the international, national, and local levels about the importance of developing locally relevant measurement solutions. Targeted funding from donors may be required not only for the development of sustainable health information systems but also to assist organizations with training of local individuals in data collection and analysis where there are shortages in this expertise, as well as with the retention of trained individuals. In order to be effective, these efforts need to be coupled with an assessment of the existing monitoring and evaluation capacity of local actors. Tools that could be used to improve measurement capacity include workshops and training sessions to instruct health authorities or program coordinators on how to set up and maintain data collection systems, implement core indicators, design evaluations, adapt preferred guidance documents to their unique situation, and analyze data. Centers of excellence in this area that are established within a developing country need not be disease specific and must have the potential to build capacity at both the national or regional level that would benefit multiple health sectors.
Expanding local analytic capacity could also potentially help to reduce the prevalence of unused data “piles” that amass in developing countries. The failure of both donors and national governments to invest in sustainable health information systems inhibits countries’ abilities to routinely process these data (Stansfield, 2009). It is important to note that building the capacity to collect and analyze data is not sufficient. There is also a need to strengthen the motivation and capacity for policy makers to interpret and act on the data. To achieve this, data collection strategies could be developed in consultation with policy makers and include mechanisms for timely reporting to inform policies and programs.
In addition, the proliferation of multilateral organizations, international and local nongovernmental organizations, and the expanding private sector all place their own, often redundant measurement and evaluation demands on local actors, which adds an additional burden to local and national measurement efforts and contributes to the accumulation of unused data. Following the completion of their individual evaluation processes, the information collected is typically analyzed and disseminated within
the organization itself, completely extracting it from the communities to which it refers. This practice has had two notable negative consequences: first, it limits the amount of community involvement in the measurement and evaluation process, missing an important opportunity to develop local analytic skills, and second, it propagates a culture of non-evidence-based policy making by failing to connect policy or program interventions with impact assessment results (Stansfield, 2009).
Costs of Measurement
The cost of measurement can pose an important limitation on feasibility. Along with capacity for data collection and analysis, costs must be taken into consideration when prioritizing, planning, and implementing any of the specific measurement approaches that will be described later in this chapter. Methods to collect population data, such as systematic surveillance and health information systems, can be very expensive and have required subsidization from external funders in many countries. Although there is limited publicly available information and analysis of the costs to implement population measurement strategies, some estimates for country spending on health data suggest that comprehensive measurement can be affordable for developing countries. For example, the Health Metrics Network (HMN) estimates a national health information system comprising six essential subsystems (health service statistics, public health surveillance, census, household surveys, vital events, and health resource tracking) would cost $0.53 per capita in a low income country (Stansfield et al., 2006). The health information system in Belize was implemented at an initial cost of approximately $2 per capita (Bundale, 2009). The Millennium Development Goals Africa Steering Group (2008) estimates that to support censuses, household surveys, and civil registration and vital statistics systems across Africa would cost $250 million annually (less than $1 per capita). In Tanzania, 11 information systems that generate health and poverty indicators were able to generate all but one of the indicators recommended by four major poverty reduction and reform programs, at an aggregate cost of $0.53 per capita in 2002/2003 (Rommelmann et al., 2005).
Program evaluation also requires an investment of a proportion of the project budget, but there is little publicly available information on the amount spent on measuring, monitoring, and evaluating health programs, and there is limited evidence to assess the costs, cost-effectiveness, benefit-cost ratio, or financial return on investment for different measurement strategies to evaluate these programs. Indeed, although measurement activities usually receive some funding as part of the implementation of a program, no empirical basis supports specific budget targets for measurement or monitoring and evaluation. The most explicit guidance regarding
the proportion of program activities that should be devoted to monitoring and evaluation comes from the Global Fund. The Global Fund’s 2009 Monitoring and Evaluation Toolkit: HIV, Tuberculosis, and Health Systems Straightening, which was co-sponsored by a number of major multilateral global health organizations, states that over the past years “global and national efforts have been made to increase financial resources for monitoring and evaluation to the widely recommended 5–10 percent of the overall program budget.” It endorses this amount and offers a framework on how to allocate these funds (The Global Fund to Fight AIDS, Tuberculosis and Malaria, 2009, p. 32).
APPLYING MEASUREMENT METHODS FOR GLOBAL CVD
The following sections describe methods and tools that can serve to improve measurement for global CVD by providing information for feedback and decision making from multiple sources (such as surveillance, intervention research and program evaluation, clinical practice data, and policy analysis) and at multiple levels (including national, subnational, health systems, communities, households, and individuals). Although a distinction in levels and sources is made in the discussion that follows, it is also ideal for measurement approaches to work across different levels—for example, by using nested measures with relevance to each other. For comparable use of data across sources and levels, there also needs to be agreement on what is to be measured and how it is disseminated. To address global CVD, the methods described here draw from successful CVD measurement strategies and programs from the developed and the developing world where available, as well as from significant advances in measurement in other areas of global health, especially HIV/AIDS.
Measurement to Inform Policy
For policy makers at all levels, measurement provides information that can motivate changes in priorities and policies, influence public opinion, help select and manage intervention approaches, and set priorities for the allocation of resources. The discipline of policy analysis strives to provide objective data and analyses to support rational policy decisions (i.e., “evidence-based policy”). There is an important distinction between evidence for policy and evidence on policy. Evidence for policy supports a rationale for prioritizing and implementing policies and programs and often comes from population-level evidence as well as from system- and program-level evidence. A well-supported rationale, however, involves uncertainty as to the actual benefit or relevance, especially when being translated from other contexts.
Evidence on policy attempts to address that uncertainty by actively assessing the impact of public health and prevention policies using measurement of population endpoints, such as smoking prevalence or clinically recognized myocardial infarctions (MIs). Such research is often described as health policy and systems research in the global health literature. For example, a policy change that is phased in allows experimental data to be gathered comparing population outcomes with and without the implemented policy; this can be especially valuable in informing future policies. Policy makers also often make implementation decisions based on evaluations that assess the effectiveness and implementation of particular clinical, organizational, or public health strategies. This evaluation approach is described in more detail later in this chapter.
A recent review of the literature indicates that health policy analysis in developing countries is quite limited, especially with regard to CVD (Gilson and Raphaely, 2008). Although spending on health policy and systems research is growing, it remains low and results remain limited in rigor and generalizability compared to the needs of policy makers and providers in developing countries (Anonymous, 2008; Bennett et al., 2008). The literature on implementation science—which addresses how interventions demonstrated to be effective can be implemented in a wider range of settings—is also limited for developing countries (Madon et al., 2007). However, there is an emerging movement to use more evidence-based policy at all levels in low and middle income countries, and it is crucial to be sure that this movement does not continue to develop without being applied to policies related to chronic diseases. The strength and mix of national, regional, and local policy measurement will depend on country-specific factors, such as the governance system, the size of the relevant population, and other local attributes.
Working to fill the evidence-based policy gap in low and middle income countries, the Evidence-Informed Policy Network (EVIPNet) aims to synthesize research results into products useful to developing-country health policy makers. EVIPNet teams have now been established in Africa (van Kammen et al., 2006), Asia, and the Americas (Corkum et al., 2008). However, these efforts remain limited in scope and applicability for cardiovascular disease as none of the policy briefs currently being developed by the nine-country coalition of EVIPNet Africa relate to policy decisions on CVD risk factors (EVIPNet, 2008). The Future Health Systems: Innovations for Equity consortium is another example of an active approach to making a research–policy linkage, by working with six developing countries to develop 5-year research plans whose results will address priorities identified by policy makers (Syed et al., 2008).
While significant progress can be made by engaging national governments around measurement, the use of evidence-based policy should not be
limited to national-level actors. For example, the Tanzania Essential Health Interventions Program, initiated in 1993, has attempted to identify simple tools to enable health planners at the district level to plan on the basis of evidence (IDRC, 2008). Population data and computerized decision tools were provided to two district health management teams. The goal of the program was for the data and decision support tools to assist decision makers in prioritizing their funding strategies to those diseases that cause the greatest burden. While the program focused primarily on communicable diseases, subsequent health gains were achieved in a severely low resource setting, and the potential to adapt these strategies to a chronic disease framework could be explored. To date the program has resulted in more than a 40 percent decrease in child mortality in each district (IDRC, 2008; Stansfield et al., 2006).
Cost-effectiveness analysis based on modeling is another approach that has been used to set general priorities among CVD prevention interventions in developing countries (Gaziano et al., 2007). While the broad findings on cost-effectiveness come primarily from modeling studies, countries with adequate measurement data can perform country-specific cost-effectiveness analyses of these interventions to help inform policy decisions with evidence that is more context specific. However, to date these evaluations have been limited. Cost-effectiveness analysis of CVD interventions in low and middle income countries is discussed further in Chapter 7.
A major consideration in developing better measurement for evidence on policy related to CVD is the complexity of the determinants of cardiovascular disease. Because of this complexity, policies in multiple sectors have the potential to affect CVD. Therefore, to comprehensively measure policy effects on CVD, there needs to be shared understanding of target outcomes as well as comparable indicators and integrated measurement approaches to determine the health impact on chronic diseases of policies in areas such as agriculture, urban planning, and development initiatives from donors and governments.
One of the key challenges hampering many low and middle income countries is the lack of baseline and trend data on population prevalence and incidence of cardiovascular disease. Without this knowledge, ministers of health and program directors cannot know what types of interventions will be necessary, whom should be targeted to ensure these initiatives are most effective, and how long programs should remain in place. Without baseline data, resources might be misdirected to programs targeting individuals or groups who either are already managing their risk factors well or do not comprise the majority of individuals in need. Baseline data that
are gathered systematically and regularly over time are also necessary in order to assess the long-term impacts and effects of policy changes across a range of sectors.
Population and subpopulation statistics on CVD and related risk factors can document outcomes that are typically easy to understand and communicate, making discussions with a wide variety of stakeholders, from policy makers to community health workers, more effective and compelling (Engelgau, 2009). Population data can also highlight opportunities to more effectively prioritize, target, or adapt CVD interventions if the data are collected in a way that allows quantification of variations in CVD outcomes, risk factors, or access to care between genders or among regions, ethnic groups, socioeconomic strata, or rural populations (Joshi et al., 2008; Kivimaki et al., 2008). In addition to informing national policies, surveys with standardized data collection methods can also be used to compare conditions and the effects of intervention approaches across countries. Showing that an intervention is effective in one country of a region, or at one level of economic development, can help convince similar countries to implement similar interventions.
There is some evidence that population-level assessments influence policy decisions in developed and developing countries, although CVD-related examples are limited. In Latin and South America, 30 countries have implemented the Pan American Health Organization’s (PAHO’s) Regional Core Health Data Initiative, which aims to monitor individual member states’ attainment of health goals and stated obligations. The Regional Core Health Data Initiative collects information on both communicable and noncommunicable diseases, including indicators relating to cause-specific CVD mortality, overweight and smoking prevalence, and select health systems measures. However, the vast majority of indicators are related to identifying communicable disease risk (PAHO Health Analysis and Statistics Unit, 2007). This may be indicative of the current health information needs of the region, but it could also contribute to underreporting of the contribution of CVD to local disease burdens.
Of the 30 countries that have implemented the Data Initiative, 21 now report using these data to make policy decisions (Ten-year evaluation of the regional core health data initiative, 2004). For example, Mexico, following the revelation of how out-of-pocket payments led to impoverishment among low income citizens, established Seguro Popular, a new insurance program, along with other mechanisms to aid poor families (Knaul et al., 2006). As part of the Data Initiative, PAHO also produced Basic Country Health Profiles for participating countries in 2002, describing both health status and trends in each nation as well as descriptions of both policy and health systems responses (PAHO, 2007). However, follow-up surveys are needed to continue to track any progress made within countries and to
identify, if possible, the level of responsiveness of national governments to changing trends in their disease burden data. Importantly, the use of standardized indicators in the Data Initiative allows comparisons of progress across countries and informs the potential transferability of effective interventions to reduce disease burden.
Over the past several decades there has been a steady growth in the number and types of methods available for collecting population data to monitor and evaluate national and global health. The following section describes methods that are currently either in use or available for population-based systematic surveillance and periodic surveys, both in developed and developing countries. These methods include vital registration systems, systematic surveillance sites and health information systems as well as periodic demographic and risk-factor surveys. Population-level methods can at minimum collect data on cause-specific mortality and cardiovascular events. To the extent feasible, these could also potentially be supplemented with more in depth analyses of demographic subpopulations and surveys of additional indicators such as behavioral and biological risk factors, disease outcome measures, biomarkers, economic measures such as national spending and household expenditures, and health-system measures such as service utilization.
Each method offers a unique opportunity for the global CVD community to draw upon existing models and lessons learned from their implementation, and in some limited cases to integrate with existing infrastructure to avoid duplication of resource expenditures. However, effective use of each of these measurement strategies is dependent on the establishment and prioritization of relevant metrics for cardiovascular disease and cardiovascular disease risk, as described earlier in this chapter.
Given the long time trends associated with CVD outcomes and the likelihood that funding for programmatic interventions will not include sustained, multiyear follow-up studies, the presence of active systematic surveillance mechanisms that analyze and report data is essential for identifying both program and policy impacts on CVD in developing countries. As described below, a number of surveillance systems are currently in use both in developed and developing countries that provide substantial information not only on health indicators, but also on successful methods of data collection and aggregation. However, these systems are often dominated by infectious disease data requirements, causing the unintended negative impact of constraining the amount of chronic disease-related information that can be gathered. If surveillance systems are to truly present an accurate picture of the disease and risk-factor burden in a particular country, integration with
or adaptation of these systems where feasible to provide chronic disease data will be required.
Vital Registration Systems One method of collecting essential population data is through the use of vital registration systems (also called civil registration systems or sometimes death registries). These systems provide the critical information necessary to monitor a country’s disease burden and associated needs through the collection of standardized cause of death statistics, including measures of CVD-related causes of mortality. The functionality and capacity of vital registration systems in developing countries varies substantially due in part to the finances and technical capacity required to initiate them (AbouZahr et al., 2007). While there has been increasing momentum recently on the international level to support the development of more comprehensive systems in low and middle income countries (Lee, 2003), there remains a lack of leadership in efforts to capitalize on this momentum because both the implementation costs and benefits are spread across a broad range of sectors at the national government level (AbouZahr et al., 2007). Given the importance of accurate cause of death reporting for understanding the impact of cardiovascular disease on population health, this presents a potential opportunity for the global CVD community to partner with existing actors to increase awareness and support for vital registration. The successful use of vital registration systems is also dependant upon regular funding, maintenance, and validation, because the most widespread data collection systems available will not be successful if the coding and recording methods used are inadequate (Joshi et al., 2009). This was seen recently in the Islamic Republic of Iran where a sampling of cases from the national vital registration system that covers 29 of the 30 Iranian provinces revealed that nearly half of the deaths that had been assigned to ill-defined causes were in fact attributable to cardiovascular disease endpoints (Khosravi et al., 2008).
In addition, more research is needed on the most effective and efficient methods for collecting vital statistics. While the majority of countries that have prioritized vital registration have attempted to establish broad, comprehensive national data collection systems, some countries, including China, India, and Tanzania have had success using sample registration and verbal autopsy to obtain an accurate picture of national trends (Mathers et al., 2005). A growing number of models have become available for this, such as the Sample Vital Registration with Verbal Autopsy (SAVVY) (MEASURE Evaluation, 2007). While implementation of SAVVY in developing countries is not well documented, the use of SAVVY to supplement current data collection methods is planned for the Central Statistics Office of Zambia (Sikanyiti and Nalishebo, 2009).
Systematic Surveillance Sites In order to collect more expansive data that can more effectively guide policy and intervention approaches, routine surveillance is needed not only on cause-specific mortality but also on CVD-related risk factors. More than 30 Health and Demographic Surveillance Sites (HDSSs) in Africa, Asia, and the Americas currently provide a basic network of routine data collection in developing countries, often through formal mechanisms such as the INDEPTH Network (Baiden et al., 2006; Krishnan et al., 2009; Ng et al., 2009). At a minimum, these sites serve to collect longitudinal data on birth rates, death rates, and population migration patterns; however, through a number of research initiatives they have been successfully supplemented with additional indicators including measures of maternal mortality, malaria prevention, and even noncommunicable diseases. This makes them an important entry point into which CVD surveillance efforts could be integrated (Baiden et al., 2006). Thus, where feasible, these systems could be augmented to include a minimum set of CVD indicators, potentially minimizing the burden associated with introducing a entirely new system and the potential future risk for creating a CVD measurement “silo.” Additionally, in many developing countries the now-established HDSSs began as large program evaluation centers and were later converted for routine surveillance use, making them a potential model for ways in which program and policy interventions can contribute to the long-term sustainability of measurement capacity (Baiden et al., 2006; Ng et al., 2009).
Opportunities also exist to draw on experiences from some developed countries, where larger, routine surveillance programs that do not rely solely on sentinel sites have also been established. One example of this in the United States is the Behavioral Risk Factor Surveillance Survey (BRFSS) conducted by the Centers for Disease Control and Prevention for the past 25 years. The BRFSS is a monthly phone survey that also includes technical assistance documents, data analysis tools, and guidance on strategies to secure funding for municipalities that would like to participate. Measures of CVD risk, such as tobacco use, participation in physical activity, and previous CVD diagnoses, are included. In addition, the use of a standard set of core indicators for each survey allows for multi-site comparisons (Centers for Disease Control and Prevention, 2010). The success and relative simplicity of BRFSS suggests that this telephone survey approach may be transferable to some developing country settings, especially given the proliferation of cell phones in many regions (Tryhorn, 2009).
Health Information Systems Finally, health information systems (HISs) offer another opportunity for systematic data collection. Policies and programs that attempt to alter risk behavior among a broad cohort of individuals will often depend on these types of information systems to demonstrate
the effect of their interventions. As with surveillance, efforts are currently underway to provide support to developing countries seeking to strengthen and harmonize their national HIS. Therefore, it is important for the global CVD community to ensure that metrics related to CVD and chronic diseases are integrated within these systems in developing countries, especially given the growing international focus on the integration and coordination of disparate HISs (Stansfield, 2009).
At WHO, for example, the Health Metrics Network (HMN) offers the documents a “Framework and Standards for Country Health Information Systems” and “Assessing the National Health Information System: An Assessment Tool,” which are available for all nations as guidance during self-assessments of coverage and gaps in their existing HISs (WHO Health Metrics Network, 2008a, 2008b). Low and middle income countries are also eligible for additional targeted technical assistance and, in some cases, funding support (WHO Health Metrics Network, 2010). In 2006, for example, Cambodia completed the first stage of its assessment with funding from the HMN, following which they developed an HIS Strategic Plan that was disseminated at a Health Metrics Network regional workshop (Veasnakiry, 2007). Similarly, in 2007 Ethiopia completed an HIS overall assessment with the assistance of the HMN Assessment Tool as part of its Health Sector Development Plan (WHO Health Metrics Network, 2007). To date 60 developing countries have received HMN grants for a variety of HIS assessment and improvement initiatives (WHO Health Metrics Network, 2010). While the Health Metrics Network does not advise each country on the specific indicators to include in their HIS strategic plan, the inclusion of CVD measures will be important in any well-rounded health information system. Effort will be needed by the global cardiovascular disease community to ensure that other disease needs do not crowd out the inclusion and reporting of CVD metrics as efforts supported by HMN are implemented.
In the past three decades, a number of major multinational surveys of patient populations in clinical settings have illuminated the growing burden of CVD worldwide and have been critical in helping to establish priorities for intervention approaches. These include the WHO Multinational Monitoring of Trends and Determinants in Cardiovascular Disease project and the European Action on Secondary Prevention by Intervention to Reduce Events (EUROASPIRE) in Europe, and the WHO PREMISE study, which included countries in the Middle East, Asia, and Latin America (EUROASPIRE Study Group, 1997; Mendis et al., 2005; Tunstall-Pedoe et al., 2003). These data collection efforts have been
beneficial in three ways. First, they provide a cross-sectional view of the epidemic. Second, they provide a richer estimation of what measurement can tell us about the epidemic through the inclusion of more detailed indicators and analysis, which, if particularly useful, would expand and guide future measurement efforts. Third, they set standards for more rigorous collection and validation processes. However, due to the lack of repetition for most of these collection efforts, limited analysis can be done regarding temporal trends. In addition, implementing, coordinating, and funding a multinational survey of such detail is less likely to be feasible in a developing-country setting.
By contrast, routine surveys done at the national or cross-national level, supported by national governments and/or multinational organizations and implemented every 1, 2, or 5 years, can provide ongoing data collection that is critical to measure the scale and scope of CVD as well as trends in disease burden over time. While the cost of these surveys, both to the national government and implementing partners, limits the frequency with which they can be conducted, strong models, such as the Demographic and Health Surveys (DHS), do exist and have been successfully conducted in a wide variety of developing countries. DHS survey templates have been developed for a range of topics; however one is not yet publicly available for CVD or CVD risk-factor prevalence (MEASURE DHS, 2010). Other tools for population-level data collection in developing countries are being developed at the Institute for Health Metrics and Evaluation (IHME), most notably in the area of modeling for disease prevalence, health outcomes, and coverage of health interventions. In addition, IHME has the capacity to develop needs-specific survey instruments for data collection (University of Washington, 2010). Organizations such as MEASURED HS and IHME represent key actors with extensive knowledge of successful developing country survey strategies and, in some cases, established infrastructure. Therefore, they have the potential to be future partners in CVD burden data collection efforts.
One major effort to develop chronic disease-specific survey tools is WHO’s STEPwise Approach to Chronic Disease Risk Factor Surveillance (STEPS). STEPS has been piloted and implemented in a demonstration capacity in 41 different countries spanning all income categories (WHO, 2010a). The goal of the STEPS program is to provide flexible options for countries seeking to assess their chronic disease and risk-factor burden in the form of core, expanded, or optimal sets of indicators depending on the technical and financial capacity of the individual government (WHO Noncommunicable Diseases and Mental Health Cluster Surveillance Team, 2001). In addition to the basic implementation tool with a spectrum of measurements, WHO also provides a variety of blank templates, implementation guidance, and analytical software (WHO, 2010b).
The STEPS program advances the possibility of implementing routine population surveys in multiple countries using comparable indicators and survey methodologies. For example, PAHO has worked with countries to adapt STEPS for local context in Latin American countries. Their version, called PanAm STEPS, contains many of the same core questions, but has alternative indicators in the area of food and cigarette consumption that take into account regional and contextual issues (PAHO, 2008). The next steps are to evaluate the feasibility of implementing STEPS on a larger scale as a population-level surveillance tool in developing-country settings. This will require evaluation of costs of implementation and the sustainability of these methodologies for serial surveillance, for example, by evaluating the potential burden associated with different numbers and types of indicators.
As with the expansions of surveillance and health information systems, future assessments of STEPS and other population surveys should also consider the opportunity costs of committing local capacity to assessing CVD and behavioral risk factors compared to other priorities. In some cases, surveys to collect new information on cardiovascular disease can make use of existing resources and capacity by being integrated into existing data collection efforts. For example, STEPS was recently successfully integrated into nine sites in five Asian nations that are part of the INDEPTH Network in order to establish baseline CVD risk-factor data that could inform policy makers (Ng et al., 2009). It will also be important to consider questions of local data ownership and empowerment to adapt survey design, implementation, and analysis. These questions speak to the need to avoid costs without benefit and data without utility (Stansfield, 2009).
In a more narrowly targeted risk-factor survey, the Global Youth Tobacco Survey (GYTS), and the subsequent Global Adult Tobacco Survey, are additional sources that already provide critical information on smoking and tobacco use prevalence internationally within predefined geographic regions (Warren et al., 2008). The standardized methodology employed by the GYTS sites allows for cross-national comparisons of results and analyses to identify where or if progress occurs. Present in 142 nations and recording indicators that cover tobacco use, knowledge and attitudes, advertising levels, and school-based curricula (Asma, 2009), the GYTS may provide either sufficient information on youth tobacco use that does not need to be replicated, or an opportunity to partner with a broad, well-established systematic data collection effort in order to address other risk factor topics. In addition, the indicators in successful surveys of this kind that are currently validated and in use should also be considered during efforts to establish internationally agreed upon measures, as discussed earlier in this chapter.
Conclusion 4.1: Gaining knowledge about the specific nature of the CVD epidemic in individual countries and about what will work in developing-country settings is a high priority. Improved country-level population data would serve to inform policies and programs.
Recommendation: Improve Local Data
National and subnational governments should create and maintain health surveillance systems to monitor and more effectively control chronic diseases. Ideally, these systems should report on cause-specific mortality and the primary determinants of CVD. To strengthen existing initiatives, multilateral development agencies and WHO (through, for example, the Health Metrics Network and regional chronic disease network, NCDnet) as well as bilateral public health agencies (such as the Centers for Disease Control and Prevention [CDC] in the United States) and bilateral development agencies (such as United States Agency for International Development [USAID]) should support chronic disease surveillance as part of financial and technical assistance for developing and implementing health information systems. Governments should allocate funds and build capacity for long-term sustainability of disease surveillance that includes chronic diseases.
Efforts to scale up, expand, or adapt any of the existing routine data collection methods described above to develop better chronic disease measurement in low and middle income countries need to take into account potential areas of duplication of efforts and resources and to carefully consider how best to incorporate or take advantage of existing international initiatives. It will be a major challenge for developing countries to invest in chronic disease measurement capacity while the burden of infectious and perinatal disease still remains high (Boutayeb, 2006; Ezzati et al., 2005). Policy makers should strive to identify the minimum scope of data that must be collected to support CVD control programs in their country. Program- or funder-specific data collection may help developing countries meet some of these needs but may also perpetuate information “silos” (Stansfield et al., 2006) that can hinder the efficiency of managing chronic disease programs that simultaneously address multiple risk factors. This leads to a compelling rationale for leveraging resources and capacity by building the collection of new CVD information into integrated data collection efforts with existing surveillance mechanisms, which in many countries focus on infectious diseases and maternal and child health. New efforts to expand population measurement to include chronic disease should be conceptualized in the context of building and strengthening national systems for data collection rather than competing for limited resources. Although challenging, op-
portunities exist to promote collaboration rather than rivalry by using new technologies, disseminating findings, working with civil society, and training transferrable skilled analysts.
However, it is also important to note that there are inherent differences between chronic and infectious diseases that may limit the direct overlap of infectious disease surveillance methods with CVD and other chronic diseases (Nsubuga et al., 2006). For example, unlike the long-term, sustained services needed to address chronic diseases, many infectious diseases have a narrow range of treatment or vaccine options (which in some cases confer lifetime protection), and a response time frame for intervention outcomes of weeks, compared to decades for chronic diseases. The contributing factors to disease incidence can also be much less complex when there is a single infectious agent and a defined path or vector for transmission. HIV and tuberculosis (TB), the so-called chronic infectious diseases, are probably most analogous to CVD due to their need for chronic care, the role of lifestyle risk factors, and the need for extensive prevention efforts. Elements from HIV and TB surveillance that can most directly inform CVD surveillance include risk-factor measurement, maintenance of disease registries for managing treatment and secondary prevention, and techniques for monitoring service management effectiveness (Diaz et al., 2009).
Measurement of Health Systems
Over the past decade, there has been growing recognition that achieving many of the current global health priorities will require strengthening of health systems rather than simply concentrating efforts on disease-specific programs. Measurement of health systems performance is a critical component for national governments and donors to use in evaluating the capacity of countries to promote the health of their citizens, the effectiveness of health programs, and the transferability of successful health systems reforms from one country to another.
There is ongoing debate within the policy and global health communities over how to define health systems and their impact (IOM, 2009a; Kruk and Freedman, 2008; Murray and Frenk, 2000; Shengelia et al., 2003). A number of different frameworks for measuring health systems performance have been proposed. These vary in the definition of the scope and primary objectives of health systems and therefore what aspects should be evaluated to assess their performance, ranging from a narrow focus on health services to a broader inclusion of public health efforts (Arah et al., 2003; Kruk and Freedman, 2008). In any of these frameworks, a clear definition of the goals of a health system is the essential first step to performance evaluation (Kruk and Freedman, 2008; Murray and Frenk, 2000).
The variability in existing efforts to assess health systems performance
highlights two of the biggest current challenges: the lack of available data and the lack of an internationally standardized framework or set of indicators. Determining how to measure health systems in a standardized way that is valid, reliable, and locally relevant, with feasible and implementable data collection, is an important item on the global health research agenda (Kruk and Freedman, 2008). A number of major initiatives are currently being used to standardize health systems metrics to be used globally, including the United Nations’ Interagency and Expert Group on Millennium Development Goal Indicators, the WHO-based HMN, and the IHME at the University of Washington (Kruk and Freedman, 2008). Coordination among these initiatives will be essential to avoid duplicate systems of measurement and to maximize the limited resources for evaluation and measurement.
Thus, measurement and evaluation of health systems is another area in which it is important for the global CVD community to engage in these emerging initiatives and ensure that metrics that are relevant to the quality, costs, and financing of chronic care are integrated as health systems indicators are developed. This will help ensure that measurement of health systems going forward will be informative in terms of the capacity of these systems to address CVD and related chronic diseases.
Measurement of Quality of Care
The quality of clinical practice, which relies on measurement and feedback, is critical to translating evidence based-practices into effectiveness. The fundamental importance of evaluations of the quality of clinical care is that they relate risk factors to the successful application of interventions such as treatment of hypertension and hypercholesterolemia and other areas where clinical interventions mitigate risk, including counseling about diet indiscretions, tobacco use, and exercise. Improving the quality and effectiveness of patient care can also reduce waste and inefficiencies, which is especially important for CVD treatment within health systems and clinics in low-resource settings. These measures also have the advantage of being more readily obtained than an outcome event (e.g., an MI) and are by definition aimed at the provider behavior that needs to change to improve health.
A number of methods are available for measuring practitioner performance: provider self-report, patient vignettes, patient self-report, and record reviews. The Institute of Medicine (IOM, 2001) recommends that quality be defined along six dimensions, as timely, effective, safe, equitable, cost-effective, and patient centered. Measures of quality ideally reflect these dimensions in addition to being amenable to improvement, able to account for differences in the type of patients treated by different providers, and economically feasible to use over time and across systems (Epstein, 2006;
Spertus et al., 2003). Measures should also be reliable and valid, and not be able to be “gamed” (Petersen et al., 2006).
Currently, a variety of studies have been published that identify indicators for use in high income countries, which cover a broad range of illnesses and areas. However, the current challenge with many of these indicators is that they may not always be suitable in all contexts, meaning they will need to be developed and adapted to be locally relevant (Engelgau, 2009). However, with feedback mechanisms established, these measurements could be used to more effectively target problem areas of service provision in low-resource settings.
A set of evidence-based measures specific to quality of CVD care have been developed by the American College of Cardiology (ACC) and the American Heart Association (AHA). These include performance measures that can be used for reporting and evaluation (Bonow et al., 2008). As defined by ACC and AHA, quality measures largely reflect the “processes of care for which recommendations in practice guidelines are of adequate strength that the failure to follow the recommendations is likely to result in suboptimal patient outcomes.” Examples of such measures include prescribing warfarin for patients with nonvalvular atrial fibrillation, beta blockers after MI, and angiotensin-converting enzyme inhibitors or angiotensin receptor antagonists for heart failure with left ventricular systolic dysfunction. Quality metrics recommended by ACC and AHA also include structural measures, such as staffing ratios, and patient outcomes, such as post-MI mortality. Performance measures—a subset of quality indicators used for public reporting, incentive-based programs, and comparisons across providers—have also been developed by ACC and AHA for heart failure, acute MI, cardiac rehabilitation, and atrial fibrillation.
Intervention Research and Program Evaluation
Measurement can be used to determine the effectiveness of an intervention in an experimental trial, but it is also the foundation for planning, prioritizing, and operationalizing interventions and programs When interventions that have been demonstrated to be effective are implemented on a larger scale as part of programs at scale, whether or not they are part of a trial, program evaluation can be used to monitor their outcomes. In addition, a cycle of ongoing evaluation can be used formatively by program implementers to assess and improve their programs and make adaptations that encourage sustainability.
An extensive review of methodologies for intervention research and program evaluation is beyond the scope of this report and is information that can be found elsewhere in the literature for evaluations not only of programs for CVD and related chronic disease but also for HIV/AIDS, mental
illness and substance abuse, and other areas of health that warrant a similar range of intervention approaches from health promotion and prevention to treatment and disease management. Therefore, only a brief overview of some key points is presented here. Although intervention research and program evaluation are described separately in this section, in reality they overlap significantly. A program can also be implemented at scale as part of an experimental trial, such that evaluation data also serve as outcome data for the intervention trial. In addition, elements of program design such as formative research and tailoring of intervention components are applicable to both experimental intervention trials and programs. In addition, given limited financial resources, careful advance planning is necessary in prioritizing intervention research and program evaluation design and methods.
Although there are an increasing number of CVD risk-reduction and treatment programs currently underway in developing countries addressing an array of local needs, information regarding their effectiveness and impact on health status is very limited and, even when available, not readily conveyed to other locations struggling with similar disease burdens. For some of these programs, improved program evaluation strategies and more reliable and rigorous measurement are needed to document and assess their progress. In other cases, accurate data collection may be occurring, but the transfer of this information is hampered by limited mechanisms available to facilitate dissemination of the results. The available evidence base for intervention research and program evaluation for CVD and related chronic diseases in low and middle income countries will be discussed in more detail in Chapters 5, 6, and 7.
The goal of intervention research is to examine a program’s effects and whether a program can be implemented with sufficient effectiveness and fidelity in different communities and at scale. Intervention research can be used to assess efficacy (impact under ideal conditions) or effectiveness (impact under conditions that are likely to occur in a real-world implementation). This assessment can be made through comparison to a control condition or as a comparison between two types of intervention. Intervention research can also be used to experimentally evaluate approaches to implement, adapt, scale up, and sustain an intervention over time (Flay et al., 2005; IOM, 2009b; Kellam and Langevin, 2003; Pangea Global AIDS Foundation, 2009).
Randomized trials are often seen as the highest standard for intervention research, and they do allow for rigorous conclusions about intervention effects and causal inferences. However, a randomized trial is often not a feasible approach to evaluate interventions in real-world settings. Other
methodologies for the experimental design, measurement, and analysis of the effects of preventive interventions include waitlist comparisons, interrupted time series, and pre-post comparisons. Interventions can also be assessed when natural experiments occur as well as through modeling approaches (Flay et al., 2005; IOM, 2009b; Kellam and Langevin, 2003).
CVD prevention research often focuses on intermediate outcomes such as behavioral measures, which can be a reasonable endpoint if causal links have been well established, for example, between behaviors and improved outcomes. When feasible, more direct measures of effect can include measure of biological risk factors or disease outcome measures, such as cardiac events. To truly assess the feasibility of interventions, economic measures such as cost-effectiveness or benefit-cost analysis are critical as well as measures of implementation processes.
When intervention programs are implemented, measurement is a critical tool both to assess whether the program is having the intended effect and to provide feedback to program managers and providers to help them assess and improve the program and their own performance. Data collection and reporting for programs are typically left to the program implementers depending on their evaluation needs and on the reporting needs of the organization under which they operate or from whom they receive their funding. Although universal program evaluation guidance may not be warranted, organized national, regional, or international reporting mechanisms can maximize the potential for sharing results or best practices. With the growing international emphasis on impact evaluations at the programmatic, community, and national levels, the importance of preplanned and rigorous program evaluations that include data that can be integrated with other measurement systems has never been more important.
Along with this growing emphasis on evaluation in global health efforts, many tools have been developed that provide models and support for program evaluation in low and middle income countries. The majority of these have been either developed or initiated by the infectious disease community, but the principles and approaches have the potential to be adapted for CVD and related chronic diseases. There are more than 200 program evaluation tools currently in use in the HIV/AIDS community (Family Health International, 2010). These have varying degrees of disease specificity. While some are directed at unique components of the HIV/AIDS epidemic or infectious diseases more broadly, there are also frameworks outlining the principles of measurement as well as guidance documents for developing evaluation and analysis plans ready for adaptation by the CVD community (Family Health International, 2010). For example, MEASURE
Evaluation provides an online basic measurement and evaluation Fundamentals Self-Guided Mini Course, originally developed for USAID, which includes discussions on how to identify indicators, plan and conduct intervention evaluations, and analyze the results (Frankel and Gage, 2007). The Global Fund to Fight HIV/AIDS, Tuberculosis and Malaria has also developed a Monitoring and Evaluation Toolkit that addresses not only HIV and TB but also efforts to strengthen health systems (The Global Fund to Fight AIDS, Tuberculosis and Malaria, 2009). Given the potential role of health systems strengthening programs in addressing the global burden of CVD, adaptation of this toolkit could provide an opportunity to harmonize relevant chronic and infectious disease health systems indicators. The Global Fund Guide for Operational Research offers the addition of process measures for long-term adaptation and sustainability of ongoing programs. This kind of operational research is particularly critical for programs to address CVD, which requires ongoing intervention. Indeed, new research may also be needed to develop program evaluation strategies that can address the long-term needs of measurement follow-up and impact evaluation, as the reality is that many of the benefits of CVD risk-factor interventions will not accrue until years or decades after individual programs have been completed.
Impact measures for programs to prevent and manage CVD would follow principles similar to those for intervention research, including both CVD-related outcomes and economic measures, as well as process measures to monitor program implementation. Although measurement strategies need to be tailored to specific interventions or programs, some standardization would provide an opportunity for a better continuum from intervention trials through implementations of interventions at scale, with a set of streamlined indicators that would be useful to assess whether the original effectiveness is being maintained. The incorporation of some agreed-upon standardized metrics would also allow for comparisons across programs and over time and greater long-term feasibility of program evaluation.
As intervention programs increase in scale, so do their data collection and reporting needs, and their risk of developing duplicate systems that operate alongside national health information systems. A review of how reporting mechanisms for major global HIV/AIDS programs interact with national data collection efforts showed that this can lead to inefficient use of resources on parallel reporting structures, a failure to develop one coherent national picture of impact, and an increased burden on program implementers (Oomman et al., 2008). Avoiding this duplication by identifying CVD indicators that can meet the needs of both the program and the health information system as well as encouraging the integration of reporting with the national systems where appropriate will be important considerations as global CVD programs expand in developing countries.
Measurement of Individual Health Status
Measurement of individual behavioral or biological risk factors can be useful to motivate changes in a person’s behavior if the data are meaningful enough for the individual to be able to act upon the results. This requires that the data be presented in clear terms alongside health counseling or education initiatives to establish clinically relevant behavior-change goals for individuals.
Several decades of behavioral research in developed-country settings indicate that an individual’s knowledge of his or her health status and/or risk for disease is a necessary (albeit insufficient) precursor to behavior change. This theoretical principle is supported by research of several cardiovascular risk behaviors. For example, regular self-weighing has been associated in several studies with weight loss and weight maintenance (Butryn et al., 2007; Linde et al., 2005; VanWormer et al., 2009). Also, a recent review found that the use of pedometers to track the number of steps a person takes, particularly if a goal for steps was set, was consistently associated with increased physical activity (Bravata et al., 2007). In addition, limited data indicate that simply the knowledge of cholesterol levels can influence fat intake (Aubin et al., 1998).
In addition to providing feedback to individuals and providers to motivate and guide individual behavior change, individual measures can also be aggregated to improve provider performance, to inform measurement of health systems, or to reflect populations at a broader level when it is statistically appropriate to do so and appropriate methods are used. However, an important consideration in individual-level measurement is whether standards and norms are replicable in different populations. This is true for single measures, such as body mass index, and especially for methods used to score aggregate risk.
Emerging Technologies to Support Measurement Methods
The emergence of electronic health (e-health) and mobile health (m-health) initiatives in both developed and developing countries have opened the door to an enormous new set of potential efforts to help make both health care delivery and measurement more effective and efficient. These technologies cut across all levels of measurement and interact with each to varying degrees. While there is a need for much more research on the training, infrastructure, and cost barriers to introducing new technology and mobile data collection devices, they present a rapidly growing field of research and investment on which global health initiatives have already begun to capitalize (United Nations Foundation, 2010). Thus, it is in the
interests of the global CVD community to actively pursue involvement in ongoing efforts to improve these nascent systems.
A recent review of e-health initiatives in developing countries showed that technology is already being used in resource-poor settings with some success for a wide variety of projects, ranging from electronic health records to laboratory and pharmacy management systems to data collection and evaluation tools (Blaya et al., 2010). E-health and m-health technology are also emerging to support measurement through new tools to conduct population-based surveys and surveillance, to link data to geographic information, and to present that data to policy makers in more coherent and compelling manners (Gapminder Foundation, 2010; IDRC, 2009; Tegang et al., 2009).
In particular, the potential application of new tools to track patient status over the long term and to integrate information with health systems is uniquely suited to chronic disease management. For example, the use of electronic medical records systems in health care settings is one potential mechanism for improved data collection and analysis. These systems are already in use in a number of developing countries for monitoring patients on antiretroviral therapy (Braitstein et al., 2009; Forster et al., 2008; Kalogriopoulos et al., 2009), and a variety of both proprietary and open source software tools are available. However, it is important that these be adapted to local needs in order to prevent inefficiencies caused by a failure to ensure the collection of all necessary data or by the use of multiple systems to cover duplicate reporting needs (Forster et al., 2008; Kalogriopoulos et al., 2009; OpenMRS, 2010). Some organizations, such as AMPATH in Kenya, have already begun to adapt their antiretroviral therapy focused electronic medical records systems to include measures for diabetes and cardiovascular disease (Braitstein et al., 2009). In addition to assisting with the management of patient-level data, electronic medical records systems, if designed appropriately, also have the potential to incorporate measures that can be aggregated to inform health systems priorities. The use of mobile health approaches to improve patient outcomes is discussed further in Chapter 5.
GLOBAL USES OF MEASUREMENT
The use of measurement data compiled and analyzed at the global level is crucial to the success of current and future initiatives as it can serve to raise awareness and to prioritize and coordinate efforts among global stakeholders. As described in Chapters 1 and 2, analyses of the global burden of CVD have been critical in illuminating the scope and magnitude of the CVD epidemic and advancing the advocacy message of the CVD community.
Burden-of-disease analyses are an important method of data aggregation and modeling, which can lay out mortality and morbidity estimates,
showing changes in the epidemic across countries and linking this information to economic data (Abegunde et al., 2007; Lopez et al., 2006). In addition to these aggregated analyses, individual country efforts need to be tracked in a coordinated manner in order to inform global efforts, learn from emerging best practices, prevent duplication, and identify where additional resources and focus should be directed.
A number of broad measures of progress would benefit from leadership at the global level, including the evaluation and dissemination of the impact and implementation of global efforts, behavioral and biomedical surveillance and its integration into national surveillance systems at the population level, infrastructure, training, health education, and tracking and evaluating the effectiveness of funding and expenditures. In addition, all new policies by major global players should be backed by a financial assessment of the implementation cost and should describe means by which pledges and commitments will be reported.
A variety of stakeholders are currently responsible for either coordination or measurement at the international level. First and foremost an extensive list of measures was proposed in the 2008 WHO Noncommunicable Disease Action Plan to track global progress and characterize the different actions underway in member states. WHO is scheduled to release a preliminary progress report on a select number of these metrics (WHO, 2008). In addition, globally coordinated research efforts, such as the newly created Global Alliance for Chronic Disease (Daar et al., 2009), will need to establish indicators for tracking the distribution of funds, demonstrating the impact of their efforts, and identifying successful coordination strategies. Given the overlapping interest of many of these multilateral organizations, the development of harmonized indicators is an essential next step, as described previously in this chapter. An epidemiology reference group has also been working with WHO staff from headquarters and regional offices to develop guidance for chronic disease surveillance systems and to agree on core indicators that will be used to monitor the major chronic diseases and their risk factors (Ala Alwan, World Health Organization, 2009, personal communication). Finally, the creation of a routine global reporting mechanism that convenes to compare and disseminate results is also needed. Mechanisms for developing this are discussed further in Chapter 8.
Measurement is crucial to the success of efforts at every stage of the process to avert the rise of CVD in developing countries. Stakeholders of all kinds, from national governments to development agencies and other donors, who have committed to taking action to address the burden of chronic diseases will need to carefully assess the needs of the population
they are targeting, the state of current efforts, the available capacity and infrastructure, and the political will to support the available opportunities for action. This assessment will inform priorities and should lead to specific and realistic goals for intervention strategies that are adapted to local baseline capacity and burden of disease and designed to improve that baseline over time. These goals will determine choices about the implementation of both evidence-based policies and programs and also capacity-building efforts. Ongoing evaluation of implemented strategies will allow policy makers and other stakeholders to determine if implemented actions are having the intended effect and meeting the defined goals, and to reassess needs, capacity, and priorities over time.
Over the past two decades great progress has been made toward identifying risk-factor prevalence and CVD incidence, prevalence, severity, and mortality, as described in Chapter 3. At the global level, this has fulfilled the first step in the cycle of measurement for CVD. However, many low and middle income countries still lack sufficient local data to inform their decisions about how to prioritize actions to target CVD. In addition, while basic epidemiologic knowledge has been expanding, other core functions of measurement, such as policy analysis, health services research, intervention research, and program impact evaluation, have not been keeping pace. As a result, although there exists greater awareness about which risk factors require the most attention, less is known about what intervention approaches will be most effective and feasible in the resource-constrained settings of low and middle income countries. This lack of knowledge about program and policy effectiveness within local realities not only constrains program implementers, but also prevents national governments, nongovernmental organizations, and multilateral organizations from effectively making and implementing decisions to address the cardiovascular disease epidemic.
For some CVD measurement needs, there are well-established models for evaluation and data collection in developing countries, such as models for national surveillance, behavioral surveys, electronic medical records, and tools for program evaluation. For other purposes, new tools need to be developed. In either case, it is important, when feasible, to build upon current approaches used in monitoring and evaluation both locally and globally in order to take advantage of existing infrastructure, to build capacity in measurement and monitoring, and to avoid the inefficiencies of duplicate systems. Finally, for comparable use of data across programs and countries, there also needs to be international agreement on what is to be measured and how the information is disseminated.
Abegunde, D. O., C. D. Mathers, T. Adam, M. Ortegon, and K. Strong. 2007. The burden and costs of chronic diseases in low-income and middle-income countries. Lancet 370(9603):1929-1938.
AbouZahr, C., J. Cleland, F. Coullare, S. B. Macfarlane, F. C. Notzon, P. Setel, S. Szreter, R. N. Anderson, A. A. Bawah, A. P. Betran, F. Binka, K. Bundhamcharoen, R. Castro, T. Evans, X. C. Figueroa, C. K. George, L. Gollogly, R. Gonzalez, D. R. Grzebien, K. Hill, Z. Huang, T. H. Hull, M. Inoue, R. Jakob, P. Jha, Y. Jiang, R. Laurenti, X. Li, D. Lievesley, A. D. Lopez, D. M. Fat, M. Merialdi, L. Mikkelsen, J. K. Nien, C. Rao, K. Rao, O. Sankoh, K. Shibuya, N. Soleman, S. Stout, V. Tangcharoensathien, P. J. van der Maas, F. Wu, G. Yang, S. Zhang. 2007. The way forward. Lancet 370(9601):1791-1799.
Anonymous. 2008. The state of health research worldwide. Lancet 372(9649):1519.
Arah, O. A., N. S. Klazinga, D. M. J. Delnoij, A. H. A. Ten Asbroek, and T. Custers. 2003. Conceptual frameworks for health systems performance: A quest for effectiveness, quality, and improvement. International Journal for Quality in Health Care 15(5):377-398.
Asma, S. 2009. Global tobacco surveillance system. Presentation at the Public Information Gathering Session for the Institute of Medicine Committee on Preventing the Global Epidemic of Cardiovascular Disease, Washington, DC.
Aubin, M., G. Godin, L. Vezina, J. Maziade, and R. Desharnais. 1998. Hypercholesterolemia screening. Does knowledge of blood cholesterol level affect dietary fat intake? Canadian Family Physician 44:1289-1297.
Baiden, F., A. Hodgson, and F. N. Binka. 2006. Demographic surveillance sites and emerging challenges in international health. Bulletin of the World Health Organization 84(3): 163.
Bennett, S., J. T. Boerma, and R. Brugha. 2006. Scaling up HIV/AIDS evaluation. Lancet 367(9504):79-82.
Bennett, S., T. Adam, C. Zarowsky, V. Tangcharoensathien, K. Ranson, T. Evans, and A. Mills. 2008. From Mexico to Mali: Progress in health policy and systems research. Lancet 372(9649):1571-1578.
Blaya, J., H. S. Fraser, and S. Holt. 2010. E-health technologies show promise in developing countries. Health Affairs 29(2):245-251.
Bonow, R. O., F. A. Masoudi, J. S. Rumsfeld, E. Delong, N. A. Estes, 3rd, D. C. Goff, Jr., K. Grady, L. A. Green, A. R. Loth, E. D. Peterson, I. L. Pina, M. J. Radford, and D. M. Shahian. 2008. ACC/AHA classification of care metrics: Performance measures and quality metrics: A report of the American College of Cardiology/American Heart Association Task Force on Performance Measures. Journal of the American College of Cardiology 52(24):2113-2117.
Boutayeb, A. 2006. The double burden of communicable and non-communicable diseases in developing countries. Transactions of the Royal Society of Tropical Medicine and Hygiene 100(3):191-199.
Braitstein, P., R. M. Einterz, J. E. Sidle, S. Kimaiyo, W. Tierney, P. Braitstein, R. M. Einterz, J. E. Sidle, S. Kimaiyo, and W. Tierney. 2009. “Talkin’ about a revolution”: How electronic health records can facilitate the scale-up of HIV care and treatment and catalyze primary care in resource-constrained settings. Journal of Acquired Immune Deficiency Syndromes: JAIDS 52(Suppl 1):S54-S57.
Bundale, B. 2009. Company’s software benefits Belize. New Brunswick Business Journal, 8 June.
Butryn, M. L., S. Phelan, J. O. Hill, and R. R. Wing. 2007. Consistent self-monitoring of weight: A key component of successful weight loss maintenance. Obesity (Silver Spring) 15(12):3091-3096.
Centers for Disease Control and Prevention. 2010. CDC’s behavioral risk factor surveillance system. http://www.cdc.gov/brfss/ (accessed October 8, 2009).
Corkum, S., L. G. Cuervo, and A. Porras. 2008. EVIPNet Americas: Informing policies with evidence. Lancet 372(9644):1130-1131.
Daar, A. S., E. G. Nabel, S. K. Pramming, W. Anderson, A. Beaudet, D. Liu, V. M. Katoch, L. K. Borysiewicz, R. I. Glass, J. Bell. 2009. The global alliance for chronic diseases. Science 324(5935):1642.
Diaz, T., J. M. Garcia-Calleja, P. D. Ghys, and K. Sabin. 2009. Advances and future directions in HIV surveillance in low- and middle-income countries. Current Opinion in HIV AIDS 4(4):253-259.
Engelgau, M. M. 2009. Measuring success: Using tools we already have. Presentation at the Public Information Gathering Session of the Committee on Preventing the Global Epidemic of Cardiovascular Disease, Washington, DC.
Epstein, A. J. 2006. Do cardiac surgery report cards reduce mortality? Assessing the evidence. Medical Care Research and Review 63(4):403-426.
EUROASPIRE Study Group. 1997. EUROASPIRE: A European Society of Cardiology survey of secondary prevention of coronary heart disease: Principal results. European Heart Journal 18(10):1569-1582.
EVIPNet. 2008. Evidence informed policy network: For better decision making. Geneva: World Health Organization.
Ezzati, M., S. Vander Hoorn, C. M. Lawes, R. Leach, W. P. James, A. D. Lopez, A. Rodgers, and C. J. Murray. 2005. Rethinking the “Diseases of affluence” paradigm: Global patterns of nutritional risks in relation to economic development. PLoS Med 2(5):e133.
Family Health International. 2010. An inventory of program evaluation tools and guidelines. Arlington, VA: Family Health International.
Flay, B. R., A. Biglan, R. F. Boruch, F. G. Castro, D. Gottfredson, S. Kellam, E. K. Moscicki, S. Schinke, J. C. Valentine, and P. Ji. 2005. Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science 6(3):151-175.
Forster, M., C. Bailey, M. W. Brinkhof, C. Graber, A. Boulle, M. Spohr, E. Balestre, M. May, O. Keiser, A. Jahn, and M. Egger. 2008. Electronic medical record systems, data quality and loss to follow-up: Survey of antiretroviral therapy programmes in resource-limited settings. Bulletin of the World Health Organization 86(12):939-947.
Frankel, N., and A. Gage. 2007. M&E fundamentals: A self-guided mini course. Chapel Hill, NC: MEASURE Evaluation.
Gapminder Foundation. 2010. For a fact-based world view. http://www.gapminder.org/ (accessed March 9, 2010).
Gaziano, T. A., G. Galea, and K. S. Reddy. 2007. Scaling up interventions for chronic disease prevention: The evidence. Lancet 370(9603):1939-1946.
Gilson, L., and N. Raphaely. 2008. The terrain of health policy analysis in low and middle income countries: A review of published literature 1994-2007. Health Policy Plan 23(5):294-307.
The Global Fund to Fight AIDS, Tuberculosis and Malaria. 2009. Monitoring and evaluation toolkit: HIV, tuberculosis, and malaria and health systems strengthening. Geneva: The Global Fund.
Global HIV M&E Information. MERG background. http://www.globalhivmeinfo.org/AgencySites/Pages/MERG%20Background.aspx (accessed February 2, 2010).
IDRC (International Development Research Centre). 2008. Fixing health systems, 2nd edition—executive summary. Ottawa, Canada: International Development Research Centre.
IDRC. 2009. Assessing the use of PDAs for household surveys in Tanzania. http://www.idrc.ca/en/ev-88026-201-1-DO_TOPIC.html (accessed March 9, 2010).
IOM (Institute of Medicine). 2001. Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academy Press.
IOM. 2009a. State of the USA health indicators: Letter report. Washington, DC: The National Academies Press.
IOM. 2009b. Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: The National Academies Press.
Joshi, R., S. Jan, Y. Wu, and S. MacMahon. 2008. Global inequalities in access to cardiovascular health care: Our greatest challenge. Journal of the American College of Cardiology 52(23):1817-1825.
Joshi, R., A. P. Kengne, and B. Neal. 2009. Methodological trends in studies based on verbal autopsies before and after published guidelines. Bulletin of the World Health Organization 87(9):678-682.
Kalogriopoulos, N. A., J. Baran, A. J. Nimunkar, and J. G. Webster. 2009. Electronic medical record systems for developing countries: Review. Conference Proceedings Engineering in Medicine and Biology Society 1:1730-1733.
Kellam, S. G., and D. J. Langevin. 2003. A framework for understanding “evidence” in prevention research and programs. Prevention Science 4(3):137-153.
Khosravi, A., C. Rao, M. Naghavi, R. Taylor, N. Jafari, and A. D. Lopez. 2008. Impact of misclassification on measures of cardiovascular disease mortality in the Islamic Republic of Iran: A cross-sectional study. Bulletin of the World Health Organization 86(9):688-696.
Kivimaki, M., M. J. Shipley, J. E. Ferrie, A. Singh-Manoux, G. D. Batty, T. Chandola, M. G. Marmot, and G. D. Smith. 2008. Best-practice interventions to reduce socioeconomic inequalities of coronary heart disease mortality in UK: A prospective occupational cohort study. Lancet 372(9650):1648-1654.
Knaul, F. M., H. Arreola-Ornelas, O. Mendez-Carniado, C. Bryson-Cahn, J. Barofsky, R. Maguire, M. Miranda, and S. Sesma. 2006. Evidence is good for your health system: Policy reform to remedy catastrophic and impoverishing health spending in Mexico. Lancet 368(9549):1828-1841.
Krishnan, A., B. Nongkynrih, S. K. Kapoor, and C. Pandav. 2009. A role for INDEPTH Asian sites in translating research to action for non-communicable disease prevention and control: A case study from Ballabgarh, India. Global Health Action 2.
Kruk, M. E., and L. P. Freedman. 2008. Assessing health system performance in developing countries: A review of the literature. Health Policy 85(3):263-276.
Lee, J. 2003. Global health improvement and WHO: Shaping the future. Lancet 362(9401): 2083-2088.
Linde, J. A., R. W. Jeffery, S. A. French, N. P. Pronk, and R. G. Boyle. 2005. Self-weighing in weight gain prevention and weight loss trials. Annals of Behavioral Medicine 30(3): 210-216.
Lopez, A. D., C. D. Mathers, M. Eszati, D. T. Jamison, and C. J. L. Murray. 2006. Global burden of disease and risk factors. Washington, DC: World Bank.
Madon, T., K. J. Hofman, L. Kupfer, and R. I. Glass. 2007. Public health. Implementation science. Science 318(5857):1728-1729.
Majumdar, S. R., and S. B. Soumerai. 2009. The unhealthy state of health policy research. Health Affairs (Millwood).
Mathers, C. D., D. Ma Fat, M. Inoue, C. Rao, and A. D. Lopez. 2005. Counting the dead and what they died from: An assessment of the global status of cause of death data. Bulletin of the World Health Organization 83:171-177c.
MEASURE DHS. 2010. Demographic and health surveys: Measure DHS: Surveys and methodology. http://www.who.int/healthmetrics/library/countries/HMN_8Board_4hi_khm.pdf (accessed March 9, 2010).
MEASURE Evaluation. 2007. Sample vital registration with verbal autopsy: An overview. Chapel Hill, NC: Carolina Population Center. http://www.cpc.unc.edu/measure/tools/monitoring-evaluation-systems/savvy (accessed March 8, 2010).
Mendis, S., D. Abegunde, S. Yusuf, S. Ebrahim, G. Shaper, H. Ghannem, and B. Shengelia. 2005. WHO study on prevention of recurrences of myocardial infarction and stroke (WHO-premise). Bulletin of the World Health Organization 83(11):820-828.
Millennium Development Goals Africa Steering Group. 2008. Achieving the millennium development goals in Africa. New York: United Nations.
Murray, C. J., and J. Frenk. 2000. A framework for assessing the performance of health systems. Bulletin of the World Health Organization 78(6):717-731.
Ng, N., H. Van Minh, S. Juvekar, A. Razzaque, T. Huu Bich, U. Kanungsukkasem, A. Ashraf, S. Masud Ahmed, and K. Soonthornthada. 2009. Using the INDEPTH HDSS to build capacity for chronic non-communicable disease risk factor surveillance in low and middle-income countries. Global Health Action 2.
Nsubuga, P., M. E. White, S. B. Thacker, M. A. Anderson, S. B. Blount, C. V. Broome, T. M. Chiller, V. Espitia, R. Imtiaz, D. Sosin, D. F. Stroup, R. V. Tauxe, M. Vijayaraghavan, and M. Trostle. 2006. Public health surveillance: A tool for targeting and monitoring intervention. In Disease control priorities in developing countries. 2nd ed. New York: Oxford University Press. Pp. 997-1018.
Oomman, N., M. Bernstein, and S. Rosenzweig. 2008. Seizing the opportunity on AIDS and health systems. Washington, DC: Center for Global Development.
OpenMRS. 2010. OpenMRS. http://openmrs.org/ (accessed February 1, 2010).
PAHO (Pan American Health Organization). 2007. Basic country health profiles for the Americas: Summaries. http://www.paho.org/English/DD/AIS/cp_index.htm (accessed October 20, 2009).
PAHO. 2008. Pan American version of STEPS. Washington, DC: Pan American Health Organization.
PAHO Health Analysis and Statistics Unit. 2007. Regional core health data initiative: Indicators glossary. Washington, DC: Pan American Health Organization.
Pangea Global AIDS Foundation. 2009. Report from the Expert Consultation on Implementation Science Research: A requirement for effective HIV/AIDS prevention treatment and scale-up. Cape Town, South Africa: Pangea Global AIDS Foundation.
Petersen, L. A., L. D. Woodard, T. Urech, C. Daw, and S. Sookanan. 2006. Does pay-for-performance improve the quality of health care? Annals of Internal Medicine 145(4): 265-272.
Rommelmann, V., P. W. Setel, Y. Hemed, G. Angeles, H. Mponezya, D. Whiting, and T. Boerma. 2005. Cost and results of information systems for health and poverty indicators in the United Republic of Tanzania. Bulletin of the World Health Organization 83(8):569-577.
Shengelia, B., C. Murray, and O. Adams. 2003. Beyond access and utilization: Defining and measuring health system coverage. In Health systems performance assessment: Debates, methods and empiricism. Edited by C. Murray and D. Evans. Geneva: World Health Organization. Pp. 221-234.
Sikanyiti, P., and S. Nalishebo. 2009. Sample vital registration with verbal autopsy: Background, scope, and methodology. Presentation at the 2nd Global HIV/AIDS Surveillance Meeting, Bangkok, Thailand.
Spertus, J. A., M. J. Radford, N. R. Every, E. F. Ellerbeck, E. D. Peterson, and H. M. Krumholz. 2003. Challenges and opportunities in quantifying the quality of care for acute myocardial infarction: Summary from the Acute Myocardial Infarction Working Group of the American Heart Association/American College of Cardiology First Scientific Forum on Quality of Care and Outcomes Research in Cardiovascular Disease and Stroke. Circulation 107(12):1681-1691.
Stansfield, S. K. 2009. Cardiovascular disease in developing countries: Meeting the challenge. Presentation at the Public Information Gathering Session for the Institute of Medicine Committee on Preventing the Global Epidemic of Cardiovascular Disease, Washington, DC.
Stansfield, S. K., J. Walsh, N. Prata, and T. Evans. 2006. Information to improve decision making for health. In Disease control priorities in developing countries. 2nd ed. New York: Oxford University Press. Pp. 1017-1030.
Syed, S. B., A. A. Hyder, G. Bloom, S. Sundaram, A. Bhuiya, Z. Zhenzhong, B. Kanjilal, O. Oladepo, G. Pariyo, and D. H. Peters. 2008. Exploring evidence-policy linkages in health research plans: A case study from six countries. Health Research Policy and Systems 6:4.
Tegang, S., G. Emukule, S. Wambugu, I. Kabore, and P. Mwarogo. 2009. A comparison of paper-based questionnaires with PDA for behavioral surveys in Africa: Findings from a behavioral monitoring survey in Kenya. Health Informatics in Developing Countries 3(1):22-25.
Ten-year evaluation of the regional core health data initiative. 2004. Epidemiological Bulletin 25(3):1-7.
Tryhorn, C. 2009. Developing countries drive explosion in mobile phone use. The Guardian, March 2, 2009.
Tunstall-Pedoe, H., K. Kuulasmaa, H. Tolonen, M. Davidson, S. Mendis, and WHO MONICA Project. 2003. MONICA monograph and multimedia sourcebook: World’s largest study of heart disease, stroke, risk factors, and population trends 1979-2002. Geneva: World Health Organization.
UNAIDS (United Nations Joint Programme on HIV/AIDS). 2009a. New HIV indicator registry improves access to high-quality indicators. http://www.unaids.org/en/KnowledgeCentre/Resources/FeatureStories/archive/2009/20090313_Propertyright_UNDP.asp (accessed February 2, 2010).
UNAIDS. 2009b. UNAIDS: Indicator registry. http://www.indicatorregistry.org (accessed February 2, 2010).
United Nations Foundation. 2010. United Nations foundation: Mhealth alliance. http://www.unfoundation.org/global-issues/technology/mhealth-alliance.html (accessed March 9, 2010).
University of Washington. 2010. Institute for Health Metrics and Evaluation (IHME). http://www.healthmetricsandevaluation.org/ (accessed March 9, 2010).
van Kammen, J., D. de Savigny, and N. Sewankambo. 2006. Using knowledge brokering to promote evidence-based policy-making: The need for support structures. Bulletin of the World Health Organization 84(8):608-612.
VanWormer, J. J., A. M. Martinez, B. C. Martinson, A. L. Crain, G. A. Benson, D. L. Cosentino, and N. P. Pronk. 2009. Self-weighing promotes weight loss for obese adults. American Journal of Preventive Medicine 36(1):70-73.
Veasnakiry, L. 2007. Cambodia briefing note. Geneva: WHO Health Metrics Network, World Health Organization.
Warren, C. W., N. R. Jones, A. Peruga, J. Chauvin, J. P. Baptiste, V. Costa de Silva, F. el Awa, A. Tsouros, K. Rahman, B. Fishburn, D. W. Bettcher, and S. Asma. 2008. Global youth tobacco surveillance, 2000-2007. MMWR Surveillance Summaries 57(1):1-28.
WHO (World Health Organization). 2008. 2008-2013 action plan for the global strategy for the prevention and control of noncommunicable diseases. Geneva: World Health Organization.
WHO. 2010a. STEPS country reports. http://www.who.int/chp/steps/reports/en/index.html (accessed October 8, 2009).
WHO. 2010b. STEPS resources. http://www.who.int/chp/steps/resources/en/index.html (accessed October 8, 2009).
WHO Health Metrics Network. 2007. Assessment of the Ethiopian national health information system: Final report. Geneva: World Health Organization.
WHO Health Metrics Network. 2008a. Assessing the national health information system: An assessment tool. Geneva: World Health Organization.
WHO Health Metrics Network. 2008b. Framework and standards for country health information systems. Geneva: World Health Organization.
WHO Health Metrics Network. 2010. How can countries benefit from HMN? http://www.who.int/healthmetrics/about/howcancountriesbenefitfromhmn/en/index.html (accessed March 9, 2010).
WHO Noncommunicable Diseases and Mental Health Cluster Surveillance Team. 2001. STEPS instruments for NCD risk factors (core and expanded version 1.4): The WHO STEPwise approach to surveillance of noncommunicable diseases (STEPS). Geneva: World Health Organization.