Kinpritma Sangha, M.P.H.
Institute of Medicine
SMART Vaccines is a data-intensive tool and compilation of pertinent, high-quality data can be burdensome. At its core, SMART Vaccines relies on three types of data for producing final scores:
Readily available “known-knowns.” Some examples of this type of data would include population data for most nations of the world (and sometimes for smaller areas within nations such as states and counties in the United States). These data are available from several sources such as the World Health Organization and provide the lowest level of challenge for future users of SMART Vaccines. In high-income countries, economic and epidemiologic data resources are also available. These types of data include disease burden, typical patterns and cost of treating these diseases, productivity loss arising from these diseases, and similar data needed as inputs into SMART Vaccines. However, these types of data may be available only sporadically or not at all in low-income countries with less investment in data gathering infrastructures.
Theoretically knowable “unknown knowns.” Some data that are available readily in high-income countries would be in theory be available in low- income countries as well, but do not exist. In these cases, potential users of SMART Vaccines will have to decide whether to make new investments in data gathering infrastructures or to rely on low-cost approximations for initial priority setting exercises using SMART Vaccines.
In many settings, the most useful way to get sufficient data to begin
the priority setting process is to obtain best expert estimates, often relying on different individuals for different types of estimates. This could be done in a more formalized structure using group consultation techniques such as the Delphi methods or its variations for estimating unknown data.
The sensitivity analysis capabilities of SMART Vaccines provide another way to help users understand the importance of improving the quality of various types of data. If the user finds that within their context—and using their own weights—and across a wide range of possible data levels that the priority rankings do not change, then it is an indication that further investments to improve those data elements is not necessary. Alternatively, if the priority rankings are sensitive to the levels of certain data elements, then it signals the importance of investing resources to improve those data.
Currently unknowable “known unknowns.” These are data whose nature is well known but they do not exist, including the vaccines that do not yet exist, and the nature and hence the dangers of diseases that do not yet exist.
Specific data needed for SMART Vaccines can be categorized into four broad categories: population, epidemiology, economics, and vaccine-related characteristics. As mentioned in this report, the data are merely estimates derived from available information in order to offer guidance for further data collection.
The primary sources for SMART Vaccines data were publications reporting primary epidemiologic and economic data or else reporting sufficient information to derive estimates of the primary data. Studies reporting data on a national scale were given precedence over those that analyzed populations on a state, county, or provincial basis. In some instances when national estimates were unavailable, estimates from a smaller subset of a population were extrapolated to the entire population. Indirect estimates from mathematical modeling studies were consulted when firsthand data were unavailable. Specific source references are embedded in the data spreadsheets.
As part of the Phase II data collection efforts, the disease burden and vaccines data for Phase I candidates were revised to closely reflect the population measures of the 2009 U.S. and South African populations as used in the software. Data will naturally differ from year to year as updated sources become available. The examples provided here represent only a subset of all data sources and estimation approaches; they are not representative of
the data from all the surveillance measures that are available to the user. All datasets and specific source references are embedded in the data spreadsheets that are available online with the report and the software.
To ensure best estimates for each disease and associated vaccine, the data inputs were verified and standardized to the 2009 U.S. and South African populations. Disease burden data were verified internally within the model and also against existing estimates. Number of deaths and cases were standardized to the 2009 U.S. and South African populations using estimates for each age group (available from life tables) to obtain the data needed for the model in the following four age groups: less than 1 year old, 1 to less than 20 years old, 20 to less than 65 years old, and 65 years or older.
Next, given the number of cases and deaths within a population, the incidence and case fatality rates were calculated using the following formula: Incidence = (number of cases/total population)*100,000, and Case fatality rate = (number of deaths/number of cases). The calculated incidence and case fatality rates are checked with original number of deaths and cases, followed by verification using existing publications. To the extent possible, costs were also standardized to 2009 U.S. dollars; however, in many instances cost estimates from another year were used.
Standard Data Sources
Centers for Disease Control and Prevention (CDC):CDC is a reliable source of disease- and vaccine-specific information within the United States. Its surveillance systems follow national guidelines within its scope of work to maintain standardized data collection. For instance, two surveillance systems collect information about rotavirus disease and the rotavirus vaccine: the National Respiratory and Enteric Virus Surveillance System and the New Vaccine Surveillance Network. Similarly, there are multiple surveillance systems for influenza, including FLU VIEW (a weekly influenza surveillance report) and International Influenza Surveillance. Information from surveillance systems on mortality and morbidity are available through the Morbidity and Mortality Weekly Report (MMWR). The MMWR series, prepared by the CDC, contains disease burden data from 1990 to present.
World Health Organization (WHO):The WHO Global Health Observatory (GHO) data repository provides epidemiologic and health indicator data for WHO’s 194 member states. The GHO data repository contains more than 50 datasets on priority health topics, including the mortality and burden of diseases, immunization, and health systems. Annual summaries of health-related data are also available for member states. WHO Choosing
Interventions that are Cost Effective (WHO-CHOICE) assembles regional databases on the costs, population health impact, and cost-effectiveness of key health interventions. Using WHO-defined regions, WHO-CHOICE developed standard tools and methods to generate regional databases for collecting costs and health data. The costs and effectiveness of a wide range of health interventions are determined with probabilistic uncertainty analysis. Currently there is also a contextualization tool that makes it possible to adapt regional results to the country level. Information about health interventions, demography, epidemiology, and cost effectiveness analyses for certain diseases are also available by WHO regions.
Global Burden of Disease (GBD):GBD (most recent version released in 2012) is an international collaboration that describes the global distribution and causes of a wide array of major diseases, injuries, and health risk factors. GBD provides data on, among other things, age and sex-specific mortality, global and regional mortality from 235 causes of death, disability-adjusted life years for 291 diseases, and healthy life expectancy for 187 countries.
Healthcare Cost and Utilization Project (HCUPnet):HCUPnet is a free online data system that provides access to health statistics and information on U.S. hospital inpatient and emergency department utilization, both at the national and the state level. The Nationwide Inpatient Sample (NIS) is particularly important because it contains information found in a typical hospital discharge or billing record. Using this information, HCUPnet provides data for specific conditions and their associated durations of stay, hospital costs, national costs, percent of patients who died in the hospital, and discharge status. However, the national averages are not suited for regional analyses because of geographic differences among and within states both in health care utilization and in costs.
Custom Data Sources
When high-quality data were unavailable or when data sources varied in quality, accuracy, and comprehensiveness, proxy measures were used to produce estimates. Estimates of vaccine manufacturing costs were largely informed by vaccine industry experts on the committee. When the age-specific disease burden for certain diseases was unavailable in South Africa, epidemiologists based in South Africa were contacted via e-mail or telephone for their assistance in developing estimates. Furthermore, disease burden data from South Africa provinces were used in place of national assessments, and vaccine costs and analyses from other low- and middle-
income countries were used to substitute for costs of vaccines in South Africa.
Data Collection Methodology
Population Characteristics: SMART Vaccines was designed to accommodate demographic differences within a country, a region, or a consortium of countries. The U.S. and South African population data were obtained from the WHO country statistics database, which houses actuarial data for various countries.
Population data for each country are given by sex and are divided into 5-year age intervals, with the exception of children under 5, who are further divided into children up to 1 year in age and children from 1 to 5 years old. The data provided for each age interval are the total population (N), the number of people left alive at age x (lx), person-years lived between ages x and x+n (nLx), and life expectancy at age x (ex). The standard life expectancy (sx) is the life expectancy for Japanese women—the group that is known to have the world’s longest life expectancy and so is used in calculations for disability-adjusted life-years. The health utilities index 2 (HUI2) is used to estimate the quality of life for people in the various age intervals in order to calculate quality-adjusted life-years. Because HUI2 data are unavailable for South Africa, U.S. estimates were used.
The hourly wage rate is used in estimating the value of time lost to illness; the hourly wage rate of parents is used for children less than 15 years old. The wage rate for South Africa was approximated in U.S. dollars by using the prevailing currency exchange rate.
Special populations can include groups manifesting specific characteristics that ought to be considered in developing or delivering a vaccine—for instance, immunocompromised individuals with multiple co-morbidities, or people in a particular state or province within a country. A population table divided into males and females will need to be filled in for this group as well.
To illustrate this further, let us consider a special population that includes HIV-positive individuals. If nearly 2 percent of a country’s total population is living with HIV, then the absolute number of people living with HIV can be obtained by multiplying the total population by 0.02. If we assume, for instance, that the life expectancy of an HIV-positive individual is 10 years less than that of a healthy individual, then it is a straightforward matter to fill in the life table for this special population using the standard life table. Similarly, an HIV-positive individual will likely have a lower quality of life and hence a lower HUI2. However, these assumptions will change
depending on the country in question. In a low-income country, HIV is a debilitating condition physically, economically, and socially, whereas it can be managed with anti-retroviral drugs in the United States.
Similarly, a health official may be interested in focusing on a particular province within a country as a special population. In addition to specifying the total population, there would be another step to define the special population—the total number of people (N) in the province, the number of people left alive at age x (lx), the person-years lived between ages x and x+n, (nLx), and the life expectancy at age x (ex). A major concern in creating subsets of a special population from the total population is double counting.1 To avoid this error, it is important to subtract the special (province) subset from the total (country) population to represent the different population, disease and vaccine characteristics between the two groups.
When using SMART Vaccines to prioritize several vaccines in a population, the demographic data defining that population remain constant, while parameters specifying disease burden and vaccine-related information vary.
Disease Characteristics: To illustrate the methods used to collect data on disease characteristics, we will use the case of influenza in the United States as an example. The first step in estimating disease burden is to estimate age- and sex-specific incidence and case-fatality rates for the selected population. In this case, contrasting claims for the magnitude of influenza incidence and case-fatality rate must be reconciled in order to capture the appropriate and complete burden of disease. This is done by calculating the number of deaths by age and sex using incidence and case fatality rate, and the figures must sum to the total number of deaths for that age–sex group estimated via the standard data sources described above.
Because vaccine-preventable diseases affect sex and age groups differently, it is important to make the distinction between, for example, the disease burden caused in infants and the disease burden in the elderly. This is an important consideration for decision makers thinking about investment in pediatric vaccines versus adult vaccines. Thus, disease burden is specified by sex and in the following age intervals: infants (less than 1 year), children (1 to <20 years), adults (20 to <65 years), and elderly (65 years or older).These age intervals are selected to reflect the availability of data because most disease burden is measured in aggregated age groups of infants, children, adults and elderly. These categories also relieve user
1 A discussion on potential double counting can be found in Phase I report, pp. 63â€“65 (IOM, 2012).
burden because the software imports the previously specified age-specific population data to create the aggregated age intervals for the disease burden profile.
Despite the difference in disease burden due to age, influenza affects both sexes equally, thus the incidence and case fatality rate are assumed to be the same for both males and females. However, conditions affecting certain ages or sexes disproportionately will generally have higher mortality for certain groups; for instance, human papillomavirus has higher mortality rates in women because of cervical cancer, while rotavirus causes high mortality in children in low-income countries.
For influenza, the epidemiologic information was obtained from a CDC-based publication examining disease burden and costs associated with seasonal influenza in the United States (Molinari et al., 2007). To estimate age and sex-specific deaths for influenza, consider influenza in female infants (less than 1 year old) in 2009 in the United States: The incidence is 20,300 cases per 100,000 people, there are 2,183,518 female infants (less than 1 year old) and the case fatality rate (the proportion of deaths within those affected with the disease cases) is 0.000040. This information is then used by the software to calculate the number of deaths due to influenza in female infants in the United States by using the following formula: incidence*the population (N) in the age group*case fatality rate = [(20,300/100,000)*2,183,518*0.000040] = 17.73 female infant deaths.
Because disease burden includes both mortality and morbidity, SMART Vaccines allows the user to specify morbidities associated with the disease in question. Morbidities are any conditions causing health and economic burden due to the disease and can be specified simply by stating the consequent condition with its severity, such as mild (influenza without outpatient visit), moderate (influenza without outpatient visit), and severe (influenza with inpatient visit).
Morbidities can also include other complications that are a consequence of the principal condition, such as meningitis, sinusitis, and otitis media due to pneumococcal infection or fever and abdominal pain due to rotavirus. Morbidities are obtained from the disease burden publications relevant for each condition and vaccine (Molinari et al., 2007; O’Brien et al., 2009; Payne et al., 2008).
Health Utility Index 2 and Disability Weights: A reduction in the health-related quality of life for the amount of time an individual is sick is represented using the HUI2. For instance, if a healthy individual with an average HUI2 score of 0.99 is home sick with the influenza for 3 days, his or her quality of life may drop down to, say, 0.90 or perhaps even 0.80 in severe
cases. To account for this reduction in quality of life due to morbidity, disutility tolls are calculated. Disutility tolls represent the difference between the HUI2 of the healthy state prior to illness (0.99) and the state during sickness (0.90), or an overall toll of 0.09.
HUI2 values were used because these are available for the general U.S. population and for the disease and vaccine disutility tolls. The disutility tolls were taken from articles in the refereed literature that directly measured disutilities related to the target conditions. In the absence of such data either reported directly or else indirectly supported in the literature (e.g., used in a cost-effectiveness analysis with substantiated disutility estimates), estimates were obtained using nearest analogy health states to segment the data (Fryback, 2009; Fryback et al., 2007). Individuals’ answers to the quality of well-being scale were used to identify relevant health states. The HUI2 values were regressed on an indicator for the health condition and age, the regression coefficient for the indicator being a rough estimate of the toll. The disutility tolls are also rough estimates.
There are several other health utility measures that are available, such as HUI2, HUI3, and EQ-5D, and information from them can be used as well. The only caveat is that users should be consistent throughout the model because the health utility and disutility information is used to calculate QALYs, which will not be appropriately computed if different indexes are used. For example, if HUI3 data are used for the population tables, it is best to also use HUI3-related disutility tolls for the disease and vaccine morbidities, as listed in the disease morbidity and vaccine complications in the Phase I report, Ranking Vaccines: A Prioritization Framework (IOM, 2012).
To estimate HUI2 tolls for influenza, individuals’ answers on the quality of well-being scale were used to identify relevant health states—e.g., a day with influenza was equated to a positive answer to Question 2(w), which asks respondents which of the past 3 days they have had fever, chills, or sweats. The HUI2 was regressed on an indicator for the health condition and age, and the regression coefficient for the indicator served as a rough estimate of the toll.
To estimate disability weights, datasets from the GBD were used (Salomon et al., 2012). The GBD provides disability weights by categories, such as communicable diseases, cancers, and chronic diseases. For this report, proxies for DALY weights were identified from the GBD 2012 list of conditions that were sufficiently similar to the morbidities in question. Specifically, the proxies were estimated based on the severity of the disease. For instance, GBD 2012 lists the following for infectious diseases: (1) acute episode, mild = 0.005, (2) acute episode, moderate = 0.053, and (3)
acute episode, severe = 0.210. Using these estimates as well as the judgment of clinicians, the DALY weight for influenza was derived based on the severity of the condition—mild, moderate, and acute.
Economic Characteristics: The economic burden of each disease was estimated at the population level. Both payer and societal perspectives were used to calculate direct medical costs, vaccine delivery costs, and workforce productivity costs. Costs were estimated for four outcomes: (1) death due to the disease, (2) outpatient visit due to the disease, (3) hospitalization (with the disease as primary diagnosis), and (4) medication costs (including over-the-counter drugs).
For deaths that occurred in the hospital, the cost for an inpatient admission that resulted in a fatality was obtained using the 2010 HCUPnet. Cost per case was averaged using the disease-associated diagnostic categories, such as the International Classification of Diseases, Ninth Revision (ICD-9) codes for primary diagnosis in the United States (AHRQ, 2012). Hospital costs for a death that occurred due to influenza were estimated to be $6,000.
For hospitalization, the cost for an inpatient admission that resulted in a discharge was also obtained using information from the 2010 HCUPnet. Again, cost per case was averaged using the disease-associated diagnostic categories, such as the ICD-9 codes for primary diagnosis in the United States as listed in HCUPnet (AHRQ, 2012) In the case of influenza, hospitalization costs were not included in the analysis.
Outpatient visits include costs for visits to the physician that did not include hospital admission. Direct medical expenses for outpatient visits included physician costs and outpatient and pharmaceutical needs, such as lab tests, imaging tests, and consults. The outpatient visit costs for influenza were estimated at $250. For cases in which the patient did not seek medical attention, the direct costs were assumed for over-the-counter medications. For example, the average over-the-counter influenza medications cost $3 in the United States (Molinari et al., 2007).
Vaccine Characteristics: If the vaccine under consideration currently exists, data for coverage costs were obtained from CDC and WHO for the United States and South Africa. Vaccine effectiveness and length-of-immunity information were derived from published literature in vaccine randomized control trials. Because effectiveness and length of immunity depend on the location of the population as well as age, sex, and environment, these estimates are specific to regions and demographics. SMART Vaccines also allows the option to “turn on” herd immunity; if herd immunity is applied,
indirect protection is conferred when vaccine coverage is greater than or equal to 80 percent of the target population. Once vaccine coverage reaches the threshold of 80 percent, there are no additional benefits from increasing vaccination coverage because those who are unvaccinated receive protection (according to vaccine effectiveness) from those who have been vaccinated.
Data for cost per dose, doses required per person, and cost to administer per dose were purely hypothetical and based upon proxy vaccines. Cost per dose is generally the price paid by the government to purchase wholesale vaccines from manufacturers. Cost to administer a dose are the costs involved in provision of the vaccine. Because SMART Vaccines considers new preventive vaccines for which these data do not exist, those vaccines that are judged to be sufficiently similar to the one under consideration are used to derive this information. Estimates from industry experts on the committee provided estimates for research and development costs, time to adoption, licensure costs, and one-time start-up costs.
Vaccine safety is not quantified in the current version of SMART Vaccines, it is only included in the qualitative attributes because vaccines considered within the software are hypothetical and may not have this information readily available.
AHRQ (Agency for Healthcare Research and Quality). 2012. HCUPnet, healthcare cost and utilization project. http://hcupnet.ahrq.gov.
Fryback, D. G. 2009. United States national health measurement study, 2005–2006. http://www.icpsr.umich.edu/cocoon/NACDA/STUDY/23263 xml: Inter-university Consortium for Political and Social Research (ICPSR) [distributor].
Fryback, D. G., N. C. Dunham, M. Palta, J. Hanmer, J. Buechner, D. Cherepanov, S. Herrington, R. D. Hays, R. M. Kaplan, and T. G. Ganiats. 2007. U.S. norms for six generic health-related quality-of-life indexes from the national health measurement study. Medical Care 45(12):1162–1170.
IOM (Institute of Medicine). 2012. Ranking Vaccines: A prioritization framework: Phase I: Demonstration of concept and a software blueprint. Washington, DC: The National Academies Press.
Molinari, N.-A. M., I. R. Ortega-Sanchez, M. L. Messonnier, W. W. Thompson, P. M. Wortley, E. Weintraub, and C. B. Bridges. 2007. The annual impact of seasonal influenza in the U.S.: Measuring disease burden and costs. Vaccine 25(27):5086–5096.
O’Brien, K. L., L. J. Wolfson, J. P. Watt, E. Henkle, M. Deloria-Knoll, N. McCall, E. Lee, K. Mulholland, O. S. Levine, and T. Cherian. 2009. Burden of disease caused by streptococcus pneumoniae in children younger than 5 years: Global estimates. Lancet 374(9693):893–902.
Payne, D. C., M. A. Staat, K. M. Edwards, P. G. Szilagyi, J. R. Gentsch, L. J. Stockman, A. T. Curns, M. Griffin, G. A. Weinberg, C. B. Hall, G. Fairbrother, J. Alexander, and U. D. Parashar. 2008. Active, population-based surveillance for severe rotavirus gastroenteritis in children in the United States. Pediatrics 122(6):1235–1243.
Salomon, J. A., T. Vos, D. R. Hogan, et al. 2012. Common values in assessing health outcomes from disease and injury: Disability weights measurement study for the global burden of disease study 2010. Lancet 380(9859):2129–2143.