2
Priority Setting for Health-Related Investments: A Review of Methods
One of the first tasks in ranking and choosing among health-related investments is selection of an appropriate method. Several methods have been applied successfully to problems conceptually similar to that of setting priorities for accelerated development of vaccines. Examples of these problems include setting priorities for resource allocation to medical technologies; setting priorities in medical research; selecting chemicals for toxicity testing; and selecting hazardous waste sites for cleanup. The methods themselves draw from techniques in systems analysis, decision analysis, and cost-benefit analysis.
METHODS FOR PROJECT RANKING AND SELECTION
The five methods considered for use in ranking vaccine candidates were (1) multiattribute accounting, (2) multiattribute scoring, (3) decision analysis with multiple objectives, (4) cost-effectiveness and cost-utility analysis, and (5) benefit-cost analysis. They differ from one another in several ways, most notably the extent of quantification demanded and the extent to which the ranking procedure is fashioned to reflect particular normative rules.
The system developed by the committee, presented in Chapter 3, was designed primarily to assist in planning efforts of the U.S. National Institute of Allergy and Infectious Diseases (NIAID). It combines essential features of cost-effectiveness analysis and decision analysis. Other organizations may find that modifications of the system discussed below may be better suited to their needs; however, the adoption of any method or combination of methods for setting vaccine development priorities must include recognition of two basic issues.
Major portions of this chapter were prepared originally for Volume I of the report of the Committee on Issues and Priorities for New Vaccine Development (National Academy Press, 1985). They are reprinted here to assist those who wish to use Volume II as an independent document.
First, the availability of data on morbidity and mortality, pathogenicity, host responses to infection, the resources and time required for vaccine development, and potential vaccine utilization varies tremendously for various diseases and national settings. Because this information is incomplete and because the features of vaccines not yet available, their development, and the behavior of health care providers and vaccine recipients cannot be predicted with certainty, any effort to set priorities must incorporate estimates or judgments.
Second, the selection of a structural framework in which to combine these expert opinions with available data does not in itself improve the quality of the data. Although equations may be employed to define how various elements should be organized, their use does not imply that the factors or the results have the accuracy sometimes associated with formal mathematical calculations.
The formal analytical methods described below and the system proposed by the committee in Chapter 3 can improve the quality of the decision-making process. They require identification of each factor contributing to a decision, which makes later reconstruction of the priority-setting process and examination of the effects of changing assumptions easier and more accurate.
The last part of this chapter considers some general issues in implementing any method of ranking, including sources of estimates, appropriate use of sequential or “lexicographic” methods, problems of interdependence among projects, and the “portfolio” question.
MULTIATTRIBUTE ACCOUNTING
The ranking method requiring the least quantification and demanding the fewest normative assumptions is multiattribute accounting. This approach arrays the performance of each alternative on each valued objective, without attempting to produce an explicit overall score for each alternative. In deferring the final ranking to decision makers or consensus panels, multiattribute accounting differs from the other methods considered. In other respects, however, many of the steps in this process are identical to those required for the other techniques.
As in all the methods, the first step in multiattribute accounting is to specify the alternatives from which the projects will be selected. The second step is to define a set of valued objectives or criteria for the program (i.e., costs and benefits of various kinds). The result of the first two steps is to define the rows and columns of a matrix; a simplified example is shown in Table 2.1.
The third step is to fill in the cells in the matrix. Since multiattribute accounting requires no quantitative aggregation of scores across criteria (objectives), the entries in the matrix may be either quantitative or qualitative (e.g., high/medium/low). Table 2.1 contains both quantitative and qualitative information.
The fourth step is to determine if some candidates clearly dominate others, that is, perform equally well or better on all objectives. In Table 2.1, vaccine C is dominated by vaccine B and, therefore, should be ranked below vaccine B in the final rankings.
TABLE 2.1 A Hypothetical Example of Multiattribute Accounting
|
Criteria |
||||||||
Vaccine Candidates |
Potential Lives Saved per Year |
Potential Direct Economic Cost Saved per Year (dollars) |
Potential Morbidity Averted |
Vaccine Efficacya (percent) |
Cost of Development ($ millions) |
Time to Development (years) |
Likelihood of Successful Development |
Ease of Implementation |
Cost of Production and Implementation |
A |
10,000 |
500,000 |
High |
75 |
10 |
2–3 |
Good |
Excellent |
Moderate |
B |
15,000 |
1,500,000 |
Moderate |
60 |
20 |
1–2 |
Fair to Good |
Good |
Moderate |
C |
3,000 |
1,000,000 |
Low |
50 |
30 |
2–3 |
Fair |
Good |
High |
D |
0 |
500,000 |
Very High |
40 |
5 |
3–4 |
Excellent |
Poor |
Low |
aReduction in expected frequency of disease among vaccinees. |
The fifth step—the ranking itself—is left to the judgment of the decision makers or panels and is not an inherent part of the methodology. The only constraint imposed by the methodology is that dominance between pairs be preserved in the final rankings. The rest is left to intuitive judgment, which may be viewed either as an advantage or a limitation of the method.
MULTIATTRIBUTE SCORING
The method of multiattribute scoring goes beyond multiattribute accounting by generating a composite score for each candidate project. This requires three additional steps: (1) entry of a quantitative score (xij) in each cell in the matrix corresponding to the jth criterion (objective) and the ith project (vaccine candidate); (2) specifying a set of weights, wj, by which the individual factor scores will be combined; and (3) computing the weighted scores (si),
Projects are ranked according to these scores. As an intermediate step, scores for groups of criteria are often combined into subscores (e.g., a “disease impact” subscore composed of the first three criteria in Table 2.1), and then the subscores are combined. Also, the individual scores are often “normalized” to a 0–100 scale before weighting for computational convenience. Sometimes, multiplicative rather than additive aggregation rules are used.
A hypothetical example of the process of multiattribute scoring is shown in Table 2.2. The end result is that vaccine candidate A is ranked highest, followed by vaccines B, D, and C. If desired, a sensitivity analysis can be performed in which the weights are varied to see whether the rankings change. If only one of the four vaccine candidates in Table 2.2 could be developed, a sensitivity analysis would be desirable because the scores of A and B are so close. However, if two vaccines could be developed, vaccines A and B probably would come out on top for most plausible sets of weights.
Multiattribute scoring and decision analysis with multiple objectives (see below) may incorporate implicit (subjective) judgments about expected outcomes. The committee believes that every effort should be made to use available data in an explicit fashion and to clearly identify and define areas in which personal values may influence the choices.
DECISION ANALYSIS WITH MULTIPLE OBJECTIVES
One obvious limitation of the multiattribute scoring method just described is that the weights are arbitrary. This is especially disconcerting, considering that one is adding such disparate items as likelihood of success and disease mortality.
TABLE 2.2 A Hypothetical Example of Multiattribute Scoringa
|
Criterion (Weighting Factor) |
||||||||||||
|
Disease Impact (0.4) |
Vaccine Costs and Efficacy (0.4) |
Characteristics of Development (0.2) |
|
|||||||||
Vaccine Candidate |
Potential Lives Saved per Year (0.6) |
Potential Direct Economic Cost Saved per Year (0.1) |
Potential Morbidity Averted (0.3) |
Subscore |
Vaccine Efficacy (0.7)b |
Ease of Implementation (0.2) |
Cost of Production and Implementation (0.1) |
Subscore |
Cost of Development (0.2) |
Time to Development (0.2) |
Likelihood of Success (0.6) |
Subscore |
Total Score |
A |
67 |
33 |
60 |
61 |
75 |
100 |
50 |
77 |
80 |
60 |
80 |
76 |
70 |
B |
100 |
100 |
30 |
79 |
60 |
60 |
50 |
59 |
40 |
100 |
65 |
67 |
68 |
C |
20 |
67 |
10 |
22 |
50 |
60 |
0 |
47 |
0 |
60 |
50 |
42 |
36 |
D |
0 |
33 |
100 |
33 |
40 |
30 |
100 |
44 |
100 |
20 |
90 |
78 |
46 |
aScores and subscores are normalized to 100. bReduction in expected frequency of disease among vaccinees. |
Decision analysis avoids this problem by distinguishing between probabilities and consequences of project alternatives. These are then combined in logical fashion to obtain, for each candidate project, an expected effectiveness or expected utility score. For the examples presented in Table 2.1, for instance, one may estimate the expected number of lives saved per year with the following equation:
Expected number of lives saved |
= |
potential number of lives saved |
× |
vaccine efficacy |
× |
vaccine coverage |
× |
probability of vaccine development. |
For vaccine A (Table 2.1), for example, the efficacy score is 75. Vaccine coverage combines information on ease of implementation and cost, which for vaccine A are “excellent” and “moderate,” respectively. Expert judgment might translate this into a score of 0.80 for vaccine coverage. The 0.70 estimate for the probability of successful development makes more explicit the entry “good” in Table 2.1. Thus,
Expected number of lives saved |
= |
(10,000) |
× |
(0.75) |
× |
(0.80) |
× |
(0.70) |
= |
4,200 per year. |
A similar calculation of expected values could be made for other valued consequences, such as days of morbidity averted, medical costs saved, and costs of development. These expected values for each consequence could be combined into a composite score using the methods illustrated in Table 2.2, with expected values of valued consequences as the weighted items rather than a mixture of consequences and probabilities.
More rigorous application of decision analysis would entail combining the valued consequences into a utility score for each possible scenario, prior to averaging out by the probabilities, rather than averaging out each valued consequence separately and then combining the averaged-out values. The two ways of performing these steps will give the same result as long as the rule for combining consequence scores (utilities) is linear and additive. If the combination rule were multiplicative, for example, the answers generally would differ.
COST-EFFECTIVENESS AND COST-UTILITY ANALYSIS
Cost-effectiveness analysis is a formal method for selecting projects under a resource constraint. It requires that the constrained resource be identified (e.g., NIAID budget for new vaccine development or national expenditures on vaccinations) and that the resource burden of each candidate project be estimated. It also requires that a
TABLE 2.3 A Hypothetical Application of Cost-Effectiveness Analysis
Vaccine Candidate |
Cost of Development (Ci) ($ million) |
Total Expected Resource Costsa (Ci+PCi−MCi) ($ million) |
Net Expected Utility (Ui) |
Net Effectivenessb (Ei) |
Ci/Ui (dollars) |
(Ci+PCi−MCi)/Ei (dollars) |
A |
10 |
25 |
50 |
5,000 |
200,000 |
5,000 |
B |
20 |
35 |
40 |
4,000 |
500,000 |
8,750 |
C |
30 |
50 |
20 |
2,500 |
1,500,000 |
20,000 |
D |
5 |
12 |
25 |
2,000 |
200,000 |
6,000 |
NOTE: See text for definitions. aPresent value. bExclusive of resource costs, expressed in quality-adjusted years of life. |
measure of effectiveness or expected value be defined for each candidate project. If the resource cost for the ith candidate is Ci and the expected effectiveness is Ei, then the resources will be optimized if the candidate projects are ranked in increasing order by the cost-effectiveness ratio, Ci/Ei, and selected in that rank order as far down the list as resources permit.
There are at least two ways of applying cost-effectiveness analysis to the vaccine development problem. The first would treat the NIAID budget as the constrained resource, so the cost Ci would be the burden of developing the ith vaccine on that budget. The effectiveness, then, would be the net effectiveness, considering all other benefits and costs (excluding the cost of development). One possible effectiveness measure would be the score, or expected utility, resulting from the multiattribute scoring method or the decision-analytic method described above. For example, Ui might be the expected utility for the ith vaccine, as estimated by the procedure described in the preceding section. Then the cost-utility ratio, Ci/Ui, would become the basis for ranking. This is illustrated in Table 2.3. In the example, the ranking based on C/U would be as follows: candidates A and D (tied at $200,000), B ($500,000), and C ($1,500,000). Note that the lower the ratio, the higher the priority. Note also that vaccine D is given a high priority because of its low cost of development, even though its expected utility score is not as high as A or B.
Alternatively, the constrained resource might be all health-related expenditures. In that case, the numerator of the cost-effectiveness ratio would include three terms: the cost of development (Ci), plus the present value of future expected costs of production and administration (PCi), less the present value of expected savings in morbidity costs from the disease (MCi). The method of present value (the inverse of compound interest) is required to ensure that all costs and benefits are expressed in terms consistent with the same point in time.
The denominator of the cost-effectiveness ratio would be a measure of the expected health (noneconomic) benefits from the vaccine. It is also calculated as a present value. One measure used by several
researchers is the expected number of quality-adjusted years of life saved (Weinstein and Stason, 1977). This quantity, Ei, may be derived using the decision-analytic approach described above. Finally, the ratio (Ci+PCi−MCi)/Ei is calculated for each candidate project, and the ranking is based on the ratios. For example, in Table 2.3, vaccine A ($5,000 per quality-adjusted year of life) would be given the highest priority, followed by D, B, and C.
BENEFIT-COST ANALYSIS
In benefit-cost analysis, all consequences are reduced to a single, monetary quantity: the net expected economic benefit of a project. This requires that a monetary value be placed on health outcomes, such as lives saved, as well as on nonhealth outcomes. (As noted above, multiattribute scoring and decision analysis with multiple objectives often use this kind of judgment implicitly.) Measures of economic productivity, such as earnings, often are used to monetize health improvements, but any such method has serious problems. After all valued consequences have been monetized, the calculation of expected values proceeds as in multiattribute decision analysis: probabilities of various scenarios are multiplied by the corresponding utility values (or, in benefit-cost analysis, dollar values) and then summed.
Benefit-cost analysis is deeply rooted in the economic theory of social welfare. A society that wishes to maximize its welfare, according to theory, is supposed to adopt programs whose aggregate benefits exceed aggregate costs, to whomever those benefits and costs accrue. In recent years, the normative rationale for benefit-cost analysis has been challenged, although its value as a prescriptive tool is recognized even by some critics of its ethical standing (Office of Technology Assessment, 1980; Swartzman et al., 1982).
SELECTION OF AN APPROACH
The committee found that initial efforts to define its own goals and to identify the kinds of information necessary to choose among vaccine candidates simplified the task of selecting an appropriate methodology.
Neither the multiattribute accounting method nor the multiattribute scoring method satisfied the committee’s intention to make full use of available data. In addition, the methods did not permit identification of all subjective elements included in the analysis.
From the committee’s perspective, the benefit-cost approach also had two major drawbacks. First, it required that a monetary value be assigned to health benefits, such as avoidance of death, pain, and suffering. This is a very difficult and controversial task. The second problem was that the benefit-cost approach seemed to go beyond the committee’s goal of comparing ways to reduce morbidity and mortality.
After lengthy consideration, a decision analysis approach that focuses on the potential health benefits of vaccine candidates but also identifies cost considerations was selected as the most appropriate for the committee’s purpose. It provides insights on both the expected health benefits from a vaccine (i.e., the morbidity and mortality it could avert) and the costs of achieving those benefits.
Every method has limitations and drawbacks, and the proposed approach is no exception. It is important to note that some factors cannot be quantified and incorporated into such an analysis. Chapter 8 deals with issues of this kind that should be considered in the ultimate selection of vaccines for accelerated development.
ISSUES IN PROJECT RANKING METHODOLOGIES
Sources of Estimates
Data from case reports, published studies, government statistics, and other sources, as well as the subjective judgments of experts, are required for all of the methods described above. Expert judgments may be elicited either informally or by such formal procedures as the Delphi method (Dalkey, 1969). These are described further in Volume 1 of the committee’s report (Institute of Medicine, 1985).
Sequential or “Lexicographic” Methods
Sometimes ranking schemes are based on sequential rather than simultaneous consideration of objectives. One variation of this approach is often called “lexicographic” because, like the ordering of words in a dictionary, it first groups the candidates according to their performances on a selected attribute (e.g., number of deaths due to the disease), then according to a second attribute, and so forth, until all ties are broken. Obviously, the order in which the attributes are considered is important. The assumption inherent in such methods is that one does not need to look at any but the first attribute, except in the case of ties. Most decision analysts discredit the use of lexicographic methods (Keeney and Raiffa, 1976) , although sequential screening methods are sometimes necessary if the number of candidates is very large.
Interdependence Among Projects
The methods described above assume, in general, that the consequences of implementing one project are independent of which other projects also are selected. This may not be a valid assumption if, for example, costs of administration can be shared (e.g., the combination diphtheria, tetanus, pertussis vaccine), or if immunologic responses are related, either synergistically or antagonistically. If the assumption of independence does not hold, then the affected vaccines
must be assessed separately under each possible list of alternatives before the optimal group is selected.
The “Portfolio” Question
Aside from determining interdependence among vaccines with respect to costs and effectiveness, NIAID may wish to consider certain goals with respect to specific target populations or diseases. For example, the individual rankings might reveal that vaccines K, L, and M are all of higher priority than vaccine N. However, if K, L, and M all benefit populations in one region (e.g., Africa), while N benefits the population of another region (e.g., South America), then a portfolio of three vaccines might reasonably include vaccine N along with K and L. In other words, a pairwise comparison might reveal that M is preferred to N, yet the portfolio (K, L, N) is preferred to the portfolio (K, L, M). This kind of effect may be examined as a second-order iteration after the initial rankings are in hand, or it may be built into the process by examining each possible combination of vaccines separately. However, if there were 20 candidates, and the objective was to pick the top 5, for example, then the latter approach would involve examining
combinations of 5, rather than just 20 individual candidates.
Several different criteria could be used to compile portfolios in establishing vaccine priorities for important diseases in developing countries. As the example illustrates, one criterion might be the geographic boundaries of the target population. Another might be the level of development. The needs of the world’s poorer nations may be quite different from those of developing countries that have progressed further. A third criterion might be the age range of the principal target population; diseases that affect young children may be considered separately from those that attack adults in certain occupations (e.g., in some parts of the world, young men who work in forested areas have a high risk of yellow fever).
SUMMARY
The selection of an appropriate method is the first step in setting priorities for accelerated vaccine development. This chapter reviews five methods that have been used successfully in other efforts to choose among health-related investments: (1) multiattribute accounting, (2) multiattribute scoring, (3) decision analysis with multiple objectives, (4) cost-effectiveness and cost-utility analysis, and (5) benefit-cost analysis.
All of the methods require a mixture of factual information, carefully defined estimates, and subjective judgments. They differ in the extent to which they specify how the various elements should be combined. Multiattribute accounting, which requires the fewest
normative assumptions, depends heavily on the intuitive judgment of decision makers. In contrast, benefit-cost analysis reduces all consequences to a single, monetary quantity: the net expected economic benefit of a project.
The committee decided that an approach that combines essential features of cost-effectiveness analysis and decision analysis would be the most appropriate for ranking vaccines for accelerated development. Such an approach generates substantial information on both the expected health benefits from a vaccine and the costs of achieving those benefits. Unlike the benefit-cost approach, it does not require that a monetary value be placed on health benefits. The proposed method is described in Chapter 3.
REFERENCES
Dalkey, N.C. 1969. The Delphi Method: An Experimental Study of Group Opinion. Research Memorandum RM-58888-PR. Santa Monica, Calif.: The Rand Corporation.
Institute of Medicine. 1985. New Vaccine Development: Establishing Priorities, Volume I. Diseases of Importance in the United States. Washington, D.C.: National Academy Press.
Keeney, R.L., and H.Raiffa. 1976. Decisions with Multiple Objectives: Preferences and Value Tradeoffs. New York: John Wiley and Sons.
Office of Technology Assessment. 1980. The Implications of Cost-Effectiveness Analysis of Medical Technology. U.S. Congress. Washington, D.C.: U.S. Government Printing Office.
Swartzman, D., R.A.Liroff, and K.G.Croke, eds. 1982. Cost-Benefit Analysis and Environmental Regulations: Politics, Ethics and Methods. Washington, D.C.: The Conservation Foundation.
Weinstein, M.C., and W.B. Stason. 1977. Foundations of cost-effectiveness analysis for health and medical practices. N. Engl. J.Med. 296(13):716–721.