2
Data Synthesis and Framework
SMART Vaccines requires information about the population affected by the vaccine-preventable disease, the disease burden—including the costs of treating diseases of different levels of severity—and the characteristics of potential vaccines. Upon entering these data, the user chooses a set of attributes that, when combined with the user’s weights, determine the vaccine priorities. A detailed explanation of the data needs is presented in the Phase II report Ranking Vaccines: A Prioritization Software Tool (IOM, 2013; see Appendix C). The following sections summarize the data requirements and present additional considerations especially for those variables requiring estimations.
The data for SMART Vaccines typically come from different sources. SMART Vaccines has basic population and wage data already loaded for the 34 Organisation for Economic Co-operation and Development (OECD) member countries in addition to India and New York State, data for which were provided by the user groups (see Table 2-1). Analyses for subpopulations such as provinces or states require inputting additional data.
Disease burden data are typically more difficult to obtain, especially for populations with a limited public health infrastructure. SMART Vaccines requires data both on mortality and on disease incidence. Data on the costs of treating various diseases will typically be the most difficult to obtain. In the United States, many data sources provide insight into these treatment cost patterns, but the data from these sources typically pertain to specialized populations (e.g., Medicare databases deal mainly with those
TABLE 2-1
National Population Data in SMART Vaccines 1.1
|
Australia Austria Belgium Canada Chile Czech Republic Denmark Estonia Finland Greece Hungary Iceland |
India Ireland Israel Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland |
Portugal Slovak Republic Slovenia South Africa Spain Sweden Switzerland Turkey United Kingdom United States (and, individually, New York State) |
over age 65, Medicaid databases pertain to low-income populations and commercial insurance databases). In other nations with more centralized health care systems, treatment cost data may be more readily available.
SMART Vaccines 1.1 takes a summary measure of the costs of treatment as a single variable. In the previous version 1.0, users had to fill out a detailed data input table to complete the treatment costs data section. The revised format allows users to calculate their treatment costs completely offline (e.g., in a spreadsheet analysis) and enter the resulting computed total cost in SMART Vaccines. This approach provides more flexibility, including allowing the use of approximations when more precise data are not available.
Details concerning the population, disease burden, and vaccine data requirements are provided in Tables 2-2, 2-3, and 2-4.
Once the data entry is complete, SMART Vaccines prompts the user to choose from a list of attributes. SMART Vaccines includes 28 pre-defined attributes and allows users to add up to 7 more. From these, users can select up to 10 attributes to use in the analyses. A user-defined attribute needs to be a binary question with a yes or a no answer. The chosen attributes are then ranked and weighted by the user.
The limit of 10 attributes represents a software design decision from Phase II. Both the Phase I and Phase II reports (IOM, 2012, 2013) emphasized that the selection of a large number of attributes usually does not lead to meaningful amounts of utility-weight entering the calculation for the attributes as the bottom end of the priority list. Furthermore, aesthetically speaking, the “real estate” in the software screen presenting results becomes overly cluttered if more than 10 attributes are allowed.
TABLE 2-2
Demographic Data Needs for SMART Vaccines 1.1
| Demographic Informationa | Description | Notes and Specific Considerations |
|
1. Total population (N) |
Total number of people in a population by sex and 5-year age groups for a selected country. |
Data available in spreadsheet format from the United Nations World Population Prospects (2012 version). |
|
2. Number of people alive at age x (lx) |
These three variables are standard population life-table attributes, and each country’s life table includes lx, nLx, and ex for both sexes, by age groups. |
Data extracted from the life tables by country from the country statistics division of the World Health Organization’s Global Health Observatory. |
|
3. Person-years lived between ages x and x + n (nLx) |
||
|
4. Life expectancy (ex) |
||
|
5. Standard life expectancy (sx) |
Life expectancy for the Japanese population is used as the standard. |
Used for calculations related to DALYs. Japan’s life table contains life expectancy for both sexes and age groups. |
|
6. Hourly wage rate (USD) |
Hourly wage rate for a population is calculated by dividing average income by total hours worked per year. |
As applicable, this value can be converted to U.S. dollars using the prevailing exchange rate to arrive at approximate data for the selected country’s subpopulation and their age groups. |
a SMART Vaccines 1.0 required user input on values from Health Utilities Index 2 (HUI2)—in particular, data providing an estimate of the quality of life that were used to calculate QALYs—but SMART Vaccines 1.1 has this information built in and does not require it as a separate entry, thus reducing the user burden.
NOTE: DALYs = disability-adjusted life years; QALYs = quality-adjusted life years.
Attributes and Boundary Setting
In the multi-attribute utility theory model1 that underpins the SMART Score, users choose attributes as part of the ranking method used to prioritize vaccines. The attributes are based on either quantitative or qualitative measures. The qualitative attributes are simple binary—yes or no—measures or else are based on a Likert scale with 1 being the lowest grade and 5 being the highest grade.
The quantitative attributes are calculated numbers and must be given some boundary values that make appropriate contextual sense. Set-
____________
1 The details of the multi-attribute utility model and its constituent computational submodel—with its mathematical functions and testing results—are explained in the appendices of the Phase I (IOM, 2012) and Phase II (IOM, 2013) reports.
TABLE 2-3
Disease Burden Data Needs for SMART Vaccines 1.1
| Disease Burden Information | Description | Notes and Specific Considerations |
|
1. Incidence |
New cases of a specified disease during a given time period divided by the number of persons in a stated population in which the cases occurred. |
Disease burden includes incidence, which can be available from national disease databases and peer-reviewed literature. Incidence of the disease for both sexes and age groups: infants (<1 year old), children (1 to <20 years), adults (20 to <65), and elderly (65 years or older). |
|
2. Case fatality rate |
Probability of death, conditional on the disease being present. Thus, the number of expected deaths equals the annual incidence rate times the case fatality rate. |
Disease burden includes case fatality rate, which can be available from national disease databases, the World Health Organization (WHO), and peer-reviewed literature. Case fatality rate for a disease for both sexes and age groups: infants (<1 year old), children (1 to <20 years), adults (20 to <65), and elderly (65 years or older). |
|
3. Death (i) Costs (USD)a |
Costs per case diagnosed with a disease resulting in death; includes medication and outpatient and inpatient costs. |
Data available from the Healthcare Cost and Utilization Project (HCUP net) in the United States. WHO-CHOICE (CHOosing Interventions that are Cost-Effective) publishes health service delivery costs for inpatient and outpatient visits by country. |
TABLE 2-3
Continued
|
4. Permanent impairment (i) Percentage of cases (ii) Disutility (iii) Disability weight (iv) Duration (days) (v) Costs |
(i) Out of all disease cases, what percentage result in permanent impairment? (ii) Disutility tolls represent the difference between HUI2 of the healthy state prior to illness (0.99) and the state during sickness. (iii) Disability weights quantify health losses for nonfatal consequences of diseases. (iv) Duration of permanent impairment. (v) Costs per case diagnosed with a disease resulting in permanent impairment; includes medication and outpatient and inpatient costs. |
Select and specify a permanent impairment caused by the disease—for instance, permanent loss of hearing due to an infectious disease. (i) Percentage of cases are obtained from disease burden estimates. (ii) and (iii) Disutility tolls and disability weights are used to calculate QALYs and DALYs, respectively. Select one of the two to enter the information. Disability weights are available from the Global Burden of Disease Study (2010). (iv) Duration depends on the intensity of disease. (v) Costs are estimated from national hospital and health services delivery databases. |
|
5. Morbidity (i) Percentage of cases (ii) Disutility (iii) Disability weight (iv) Duration (v) Costs |
(i) Out of all disease cases, what percentage result in morbidity? (ii) and (iii) Disutility tolls and disability weights quantify health losses for nonfatal consequences of diseases. (iv) Duration of morbidity. (v) Costs per case diagnosed with a disease resulting in morbidity; includes medication and outpatient and inpatient costs. |
Select and specify morbidity caused by the disease—for instance, morbidity due to an infectious disease. (i) Percentage of cases is obtained from disease burden estimates. (ii) and (iii) Disutility tolls and disability weights are used to calculate QALYs and DALYs, respectively. Select one of the two to enter the information. Disability weights are available from the Global Burden of Disease Study (2010). (iv) Duration depends on the intensity of disease within the chosen population. (v) Costs are estimated from national hospital and health services delivery databases. |
a SMART Vaccines 1.1 takes a summary measure of the costs of treatment as a single variable and does not require more refined data as SMART Vaccines 1.0 did. The committee decided on this approach to reduce user burden.
TABLE 2-4
Vaccine Product Profile Information for SMART Vaccines 1.1
| Vaccine Product Profile Information | Description | Notes and Specific Considerations |
|
1. Coverage |
Anticipated coverage rate for the new the vaccine. |
Because new vaccines do not yet exist, coverage rates can only be conjectured. Changing the inputs allows a sensitivity analysis for “what if” scenarios in which parameters are varied to observe what changes result in other aspects of the vaccine. |
|
2. Effectiveness |
Anticipated effectiveness for the new vaccine. |
For a vaccine that does not yet exist, these inputs are derived from clinical trials conducted for the potential candidates or estimated in advance of such data using the history of similar vaccines. |
|
3. Length of immunity |
Anticipated length of immunity from the new vaccine. |
Effects of a vaccine vary widely depending on the population characteristics—age, sex, environment, etc. For a vaccine that does not yet exist, these inputs are derived from clinical trials conducted for the potential candidates or estimated in advance using data from similar vaccines. |
|
4. Doses required per person |
Anticipated doses required per person. |
Changing this value allows a sensitivity analysis for “what if” scenarios. How does changing the number of doses affect cost or coverage? |
|
5. Cost per dose |
Expected costs per vaccine dose. |
These costs represent a dose of vaccine. |
|
6. Cost to administer per dose |
Expected costs to administer a dose. |
These costs can include health care workforce costs, costs to maintain the vaccine potency, etc. |
|
7. R&D and licensure costs |
Anticipated costs for a vaccine manufacturer to develop and license a vaccine. |
Select from one of the four provided options. |
ting the boundaries for quantitative attributes that hinge on characteristics such as population size, disease burden, or hourly income can be a complicated issue. Preliminary boundaries have been suggested in SMART Vaccines along with strategies for improvement and advice on how to change them, as necessary. As the committee notes in Chapter 4, it would be useful if future software enhancements could be made to augment the program’s ability to modify boundaries, particularly when new population data (e.g., data for subpopulations) are used.
As explained in Edwards and Barron (1994), defining attribute boundaries makes it possible to score all attributes on a scale from 0 to 100, even if the attributes have innately different ways of being measured. As a simple example, one might use three attributes with which to compare automobiles: miles per gallon fuel usage (mpg), stopping distance from 60 miles per hour (mph), and the maximum number of passengers. In the miles-per-gallon category, one might set a lower bound (worst case) of 0 and an upper bound (best case) of 50, but almost every car on the road actually achieves at least 15 mpg, so a tighter range would be 10 to 50. Similarly, the stopping distance for real cars will usually fall somewhere between 100 feet (for sports cars) and 150 feet (for heavy sport-utility vehicles, for example). The number of passengers will generally vary between two (sports car) and eight (minivan).
A typical multi-attribute utility weighting would convert the values for each of these factors to fall on a range from 0 to 100—called the “swing distance.” In the automobile example this would result in the following metrics: The 40-mpg range would converted to a linear 100-point scale, with 50 mpg being the best (score of 100) and 10 mpg being the worst (score of 0). Similarly, for the stopping distance, the 50-foot range is converted to a 100-point scale, with the shortest distance (100 feet) being the best and the longest distance (150 feet) the worst. Likewise, for the number of passengers the six-passenger range would be converted to a 100-point scale, with two passengers receiving the lowest score of 0 and eight passengers receiving a score of 100.
Having converted all attributes to a common 100-point range using weights provided by the user, the multi-attribute utility model then provides a measure of how well a car performs based on the user’s definition of what is desirable. An overall score of 100 is achieved for a car that carries eight passengers, stops in 100 feet, and gets 50 mpg. An overall score of 0 is achieved for a car that carries two passengers, stops in 150 feet, and gets only 10 mpg. Scores in between depend on the weights established by the user on each attribute (mpg, stopping distance, and passenger count) and where each car falls on the three 0-to-100 scales.
To see how this works, suppose a car gets 35 mpg, carries five passengers, and stops from 60 mph in 120 feet. Its attribute scores would be mpg = 62.5 (62.5 percent of the way from worst to best), passengers = 50 (halfway between worst of 2 and best of 8), and stopping distance = 60 (60 percent of the way from 150 feet to 100 feet). If this user had put weights on the attributes of mpg = 0.6, passengers = 0.3, and stopping distance = 0.1, this car would get a score of 58.5 (0.6 × 62.5 + 0.3 × 50 + 0.1 × 60) out of a possible 100.
It is also straightforward to deal with situations in which one or more of the attribute values lie outside the boundaries. Suppose, for example, that a car got 55 mpg on the miles-per-gallon attribute (which is 5 mpg outside the range of 10 to 50 mpg used to determine the 100-point scale) and that it, like the previous car, carries five passengers and stops from 60 mph in 120 feet. Now the miles-per-gallon attribute measure is 112.5, because the additional 5 mpg is 12.5 percent of the 40-mpg range, and adding 12.5 to the upper boundary score of 100 gives 112.5. The total utility score is now 88.5 (0.6 × 112.5 + 0.3 × 50 + 0.1 × 60), which is perfectly legitimate. The score of 88.5 can be compared directly to the previous score of 58.5—it is 30 points better.
In extreme cases the attribute score of an outlier may rise sharply above the upper boundary or fall below the lower boundary. This can create a visualization problem if the display for the utility score only runs from 0 to 100, but the meaning of the score can still be interpreted without a problem. In particular, the new score is still interpreted in comparison to other scores. A car able to achieve 75 mpg instead of 55 mpg would have an attribute score of 162.5, and its overall utility score would be 120.5. This interpretation of this is that the new 75-mpg car is 60 utility points better than the original 35-mpg car, and 30 utility points better than the 55-mpg car.
Such situations where scores fall outside the 0-to-100 range can be avoided by setting the boundaries low and high enough that no conceivable candidate can have attribute scores beyond the boundaries, but setting boundaries too wide can make it difficult or impossible for any scenario to attain the highest score (i.e., 100). For example, if the best stopping distance boundary was set at 0—a convenient but unrealistic scenario—no car would get a decent score on the new scale, which now runs from 150 feet to zero. A car that can stop from 60 mph within 100 feet—and thus achieved a score of 100 on the previous scale—would now get a score of only 33.33, while a car with a 150-foot stopping distance would still get a score of 0. The result is that the possible attribute scores for realistic cars get compressed from the 0-to-100 range into a 0-to-33.33 range, making it impossible for cars to demonstrate their full potential. In short, a too-wide boundary does not allow a scenario to attain the best case because it is not realistic.
Conversely, if the boundaries are set far too narrowly on an attribute, so that, for example, some vaccine candidates are able to achieve a value of 10 times the boundary value, then the model’s calculations can mislead the user. In this case, a single attribute would dominate the calculated SMART Scores, effectively making SMART Vaccines something near to a single-attribute weighting system.
Current Algorithms for Setting Boundaries
SMART Vaccines calculates nine variables that need to have boundary values set: deaths averted, incident cases averted, workforce productivity saved, net costs, one-time costs, quality-adjusted life years (QALYs) disability-adjusted life years (DALYs), cost ($)/QALYs, and cost ($)/DALYs. The following section discusses how SMART Vaccines 1.1 sets boundaries for the preloaded national and state population data.
- Deaths averted. This depends on the size of the population, the incidence, and the case fatality rate for the relevant infectious diseases. In SMART Vaccines, the best possible score is taken as 50 percent of the annual deaths in each population that come from the worst death-causing disease. The lower bound is set at 0. Thus, to score 100 on the deaths-averted attribute, a vaccine would have to eliminate half of the deaths caused by the disease that causes the most deaths annually in the population.
- Incident cases averted. Like deaths, incident cases depend on the population size and the disease burden in each country. The current version of SMART Vaccines includes estimated values for the upper boundary with the best case being a 50 percent reduction in the number of incident cases caused by the highest-incidence disease in each country, the incidence data being estimated as a multiple of mortality boundary data. The choice of 50 percent is an arbitrary value, designed so that complete elimination of the worst disease would not become the upper boundary for incident cases averted.
- Workforce productivity improvements. Workforce productivity losses come from a combination of disease incidence, value of time, and duration of illness. The upper boundary for workforce productivity is set by using 50 percent of the highest-incidence disease rates multiplied by the average disease duration multiplied by the average daily wage rate (hourly wage × 16, allowing for 16 hours of productive uses of time under different conditions—
-
whether working in the market, at home, or enjoying leisure activities—and 8 hours of sleeping). While this approach may overstate the pure financial consequences of disease prevention modestly, it provides a simple and straightforward way of approximating these data without attempting to adjust for specific labor market conditions or taxation.
- Net costs saved. The lower boundary for net costs in all cases is 0—no medical costs saved, which is the case, for example, if the disease causes mild fatigue that requires no medical intervention. The upper boundary is set using a ratio that begins with country-level data where there are extensive data on disease-specific treatment costs, and those costs are then rescaled to other nations. To set the initial SMART Score boundaries for medical costs saved, the committee relied on U.S. data containing precise estimates of costs on many diseases from multiple sources.
From there the transformation to other population settings was carried out. This transformation is carried out as follows: Let Cj be the U.S. treatment costs for disease j and TUS be the total per capita medical spending in the United States. Then using World Health Organization (WHO) data, determine the U.S. dollar–equivalent total spending in country n, Tn. The upper bound on medical costs saved is set as CUS × (Tn/TUS). Thus, if population n spends $400 per capita (in U.S. dollars) in medical care per year, and on average the United States spends $8,000 per capita, then the ratio Tn/TUS = 0.05, and the upper bound for treatment costs saved would be 0.05 times the U.S. costs for treating that disease. - One-time costs. The boundaries for one-time costs—relating to research, development, and licensure—are set by the user by selecting the options available on the page where vaccine characteristics are specified. The boundaries range from 0 dollars (lower bound; if the user scenario, e.g., was the distribution of already existing vaccines) to greater than $1 billion (upper bound).
- QALYs, DALYs, $/QALY, and $/DALY. The upper boundary or best-case scenario for cost per DALYs or cost per QALYs is 0—which occurs in a situation in which the vaccine saves as much in medical costs as the vaccine program itself costs. While a few early vaccines actually reduce the total cost of care, most modern vaccines may have significantly higher $/QALY or $/DALY. The lower boundary is taken from the WHO guidelines for “acceptable” cost-effectiveness ratios, which is set at a value of 15 times
the country’s per capita gross domestic product (GDP). Thus, for example, with a per capita GDP in the United States of $51,755 in 2012, the worst-case boundary for $/QALY or $/DALY would be $155,265. In South Africa, the per capita GDP in 2012 was $7,314, so the worst-case boundary would be $21,942.
Considerations for Modifying Boundaries
As noted previously, multi-attribute utility models (and hence SMART Vaccines) rely on the boundary values set by the user. The boundaries suggested in SMART Vaccines are informed by current data, but over time these may become outdated or inapplicable. Furthermore, because no gold standard method exists for setting boundaries, the boundaries suggested by SMART Vaccines are simply suggested ways to create the best and worst scenarios for quantitative attributes.
Changing boundary values mid-course may render all previous calculations useless in terms of comparison with new values created after boundaries shift. Thus, the boundary setting should be done once, thoughtfully, at the beginning of the analysis for a given population. If the boundaries change, then all previous calculations should be redone. SMART Vaccines 1.1 currently does not permit users to alter the boundaries, but future versions of the sotware may offer this function.
Data for SMART Vaccines will accrue through time from various sources. In Chapter 4 the committee discusses the importance of having a host organization and active user community—one function of which would be to manage a central data warehouse, providing widespread user access to demographic data, disease burden data, and illness-treatment cost data for various populations and subpopulations that have been assembled for use in SMART Vaccines. The following discussions of the data framework presume the existence of a host organization (or its equivalent) that will manage the data infrastructure and accept data inputs from outside entities including individual users, contracted providers, and crowdsourced data creators.
No matter how these new data arrive at the data warehouse, the best mechanisms for receiving, validating, storing, and reporting data to users will likely involve a database structure that allows for flexible approaches to the data from many perspectives without a need to reorganize the data-
base tables. In short, the data warehouse will require a relational database management system.
User contributions to the data warehouse will best serve the user community if individual users or data creators can directly enter data into organized spreadsheet formats (e.g., Excel and the many proprietary and open-source equivalents), which then can be imported into the relational database warehouse. Future program modifications for SMART Vaccines could allow data importation either through the central data warehouse—built and maintained by the host organization and supervised by user-group committees—or directly by individual users who wish to use data without going through the central warehouse facility.
A blank spreadsheet template for data assembly was prepared in Phase II and is available for download at www.nap.edu/smartvaccines.
