Click for next page ( 79


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 78
7 The Role of Decision Analysis in the Translation of Research Findings into Clinical Practice ALBERT G. MULLEY, JR. The purpose of this paper is to consider the potential of decision analysis for improving the transfer of the fruits of clinical research into clinical practice, where health benefits can be realized. A narrow view of that potential might focus on the use of decision analysis to synthesize research findings in the con- text of existing evidence. Indeed, decision analysis has been used extensively to help define the clinical role of new drugs, devices, and procedures (1,2~. Examples include drugs and procedures to treat coronary disease, devices to crush kidney stones or gallstones, immunoassays to detect disease or protect the blood supply, and devices that provide images of normal and diseased human anatomy. But if we consider these examples, or other new drugs, devices, or procedures on the horizon, it seems clear that the "demand-pull" of clinical practice is at least as powerful a force as the "innovation-push" of science and technology. Human problems translate into clinical problems and those clinical problems stimulate investigation. Intelligence must move between clinical investigation and clinical practice in both directions. Decision analysis can emphasize the interactive nature of this process and thereby improve both the efficiency of clinical investigation and the timely clinical application of new technologies. DECISION ANALYSIS AT THE INTERFACE BETWEEN CLINICAL INVESTIGATION AND CLINICAL PRACTICE Decision analysis can facilitate the interactive transfer of information between clinical investigation and clinical practice by assisting in three func- tions: (a) setting priorities and identifying clinically important parameters for 78

OCR for page 78
ROLE OF DECISION ANALYSIS IN CLINICAL PRACTICE ~ 1 Clinical Investigation Decision Analysis _ Ha. Clinical Practice A Role for Decision Analysis Setting priorities and parameters for clinical investigation Synthesizing, interpreting, disseminating results of clinical investigation Distinguishing between matters of fact and value judgments . . . Implications for Agency 79 FIGURE 7.1 The role of decision analysis at the interface between clinical investigation and clot cat practice. clinical investigation; (b) synthesizing, interpreting, and disseminating the results of clinical investigation; and (c) making important distinctions between matters of factthe evidence produced by clinical investigation- and the value judgments inherent in decisions about use of new drugs, devices, and technolo- gies. This last function draws attention to agency and perspective: the clini- cian's role as rational agent for patients, the investigator or developer's respon- sibility to the public and prospective patients, and government officials' respon- sibility to protect the public welfare (see Figure 7.1~. To understand how deci- sion analysis can be helpful with these functions, its strengths and limitations must be understood. WHAT DECISION ANALYSIS IS Decision analysis is a systematic approach to decisions that have to be made in the face of uncertainty (3,4,5~. It is systematic for three reasons. It requires an explicit formulation of the problem, including alternative choices that are available to the decision maker and important specific outcomes. This formula- tion is often represented by a figure called a decision tree. Second, it requires the explicit quantitative representation of uncertainty in the form of probabili- ties. Third, it requires the explicit quantitative representation of preferences in the form of utilities. Decision analysis is potentially prescriptive. If one is willing to assign prob- abilities to all uncertain events and utilities to all outcomes, and accept assump- tions inherent in the expected utility model, decision analysis can prescribe the

OCR for page 78
80 ALBERT G. MULLEY, JR. course of action that should be followed. For some, this prescriptive intent of decision analysis is cause for skepticism, often because it is misunderstood. The very simple decision tree represented by Figure 7.2 illustrates the explic- it nature of decision analysis and the expected utility model. In this case, just two options are available to the decision maker. If the choice is alternative 1, the outcome is uncertain. There is a chance that the outcome will be "healthy," but there is also a chance that the outcome will be state j. If alternative 2 is cho- sen, the decision maker can be certain that the outcome will be state i. This simple model captures the essence of the clinical decision involving a patient whose condition, state i, could either be cured or made worse (state j) by a particular intervention (alternative 1~. The decision analyst would insist on an explicit, precise estimate of the probability of cure and the complementary probability of harm. Preferences for states i and j, relative to being healthy, would be expressed quantitatively as utilities. The sum of the utilities of "healthy" and state j, weighted by their respective probabilities, would be the expected utility or benefit of alternative 1. The expected benefit of alternative 1 minus that of alternative 2 (in this case, simply 1 multiplied by the utility of state i), would be the net expected benefit. If it were positive, alternative 1 would be advised; if negative, alternative 2 would be advised. As noted, will- ingness to make a decision based on the expected value of an alternative course of action depends on acceptance of the expected utility model. The results of decision analyses often seem overly precise. After all, proba- bility estimates may be highly uncertain and preferences may vary greatly among different raters and across time. But if explicit formulation of problems and representation of uncertainty and preferences is the first virtue of the ~~'\'~ 0< _ HEALTHY \ I\< at., ..~/ ~ l p STATE j \ - STATE i FIGURE 7.2 A simple decision tree. Square nodes represent decision points; round nodes repre- sent chance events.

OCR for page 78
ROLE OF DECISION ANALYSIS IN CLINICAL PRACTICE An OI .92 On LL I .88 81 Choose: "follow" Choose: "operate" .5x 1 x 1 .5x 2x 2.5x 3x OPERATIVE MORTALITY (times baseline rate) FIGURE 7.3 A two-way sensitivity analysis that displays results of a decision analysis for men with symptoms of benign prostatic hyperplasia who are considering TURP. The decision depends on operative mortality and utility associated win the baseline symptom state. SOURCE: Barry MJ, Mulley AG, Fowler FJ, Wennberg JE. Watchful waiting versus immediate transurethral resection for symptomatic prostatism: The importance of patients' preferences. Journal of Me American Medical Association 1988;259:3010-3017. method, flexibility is the second. Problem formulation can be altered, and prob- abilities or utilities can be varied across a plausible range, to estimate the sensi- tivity of the result, which is the net expected benefit of the preferred option. Specific "threshold" values can be identified for probabilities (e.g., of state j) or utilities (e.g., of state i) at which the net expected benefit changes from positive to negative, and the preferred option thereby changes. Figure 7.3 is an example of a two-way sensitivity analysis drawn from a detailed model of the decision to perform transurethral resection of the prostate (ATOP). This procedure is used to improve quality of life that has been dimin- ished by the symptoms of prostatism; note that life with these symptoms is anal- ogous to state i in our simple model (6~. The figure depicts the threshold utility for the symptom state that would make the expected utility of surgery just equal to the expected utility of "watchful waiting" for a range of operative mortality rates. It is this flexibility of decision analysis that gives it the potential to help set priorities for clinical investigation and effectively transfer research findings to clinical practice. THE LIMITATIONS OF DECISION ANALYSIS: WHAT IT IS NOT Decision analysis is a powerful method when used appropriately, but appro- priate use requires recognition of some important limitations. Decision analysis is not a substitute for knowledge. The method does nothing to reduce uncer-

OCR for page 78
82 ALBERT G. MU~EY, JR. tainty faced by the decision maker. Rather, it forces an untangling of multiple uncertainties and helps identify those that most affect the choice. In this context it is worth considering the kinds of uncertainty that can be represented by probabilities in a decision analytic model (7~. First, there is the personal uncertainty that exists when a decision maker is unaware of informa- tion that others have. Using a probability to represent one's strength of belief under these circumstances may be less appropriate than education, depending on the availability of the information and the urgency of the decision. Second, there is the collective uncertainty of the professional community. This may reflect the unfinished business of clinical research that has not been performed, or more difficult questions that are less amenable to investigation. In either case, if decisions must be made now, there is no good alternative to the most informed opinion. The decision analyst would express that opinion in the form of a subjective probability estimate. Finally, there is the stochastic uncertainty that always exists when dealing with biologic systems and human behavior. This element of chance will persist no matter how precisely we can estimate a probability based on past experience under similar circumstances. Another problem is that decision analysis is not descriptive. To describe the analytic approach, which depends heavily on the expected utility model, is not to describe the way most people behave. There is, however, a very rich body of descriptive decision theory that provides an important, often unappreciated, complement to decision analysis (8,9~. By identifying patterns in actual deci- sion-making behavior, this theory can sensitize us to differences between the way we behave and the way we ought to behave if we subscribe to the axioms of rational choice that form the basis for decision analysis. The choice remains with the decision maker. Decision analysis is not necessarily prescriptive. SETTING PRIORITIES AND PARAMETERS FOR CLINICAL INVESTIGATION The explicit formulation of a decision problem and the use of probabilities to represent uncertainty can prevent errors of intuition in anticipating the impact of a new drug, device, or procedure. However, the principal role of decision anal- ysis in setting priorities and parameters for clinical investigation is in forcing an orientation to health outcomes and the values associated with them. The formu- lation of a decision analytic model makes us consider which health outcomes are important, and how important they are relative to one another. Decision analysis also facilitates consideration of the potential marginal ben- efit of a new intervention by forcing comparisons with other alternatives or "fallback positions." In the example already presented, the effect of ~JRP on health outcome must be compared with the effect of the alternative, watchful waiting. It is the comparison that gives us the estimate of net expected benefit. As obvious as this may seem, there are countless examples where insufficient

OCR for page 78
ROLE OF DECISION ANALYSIS IN CLINICAL PRACTICE 83 attention is paid to the fallback position, both in establishing criteria for appro- priate use of existing technologies and in embarking on clinical investigation to estimate the potential contribution of new technologies. The requirement for explicit formulations and the orientation toward out- come and value provide a real advantage for those who would establish priori- ties and set parameters for clinical investigation. Clinical research tends to focus on the efficacy of a particular drug or procedure, or on the sensitivity and specificity of a diagnostic test. Often it does not consider the string of uncer- tainties and choices that precede or follow an action and that provide the links between the initial choice among alternatives and the valued outcomes. We well know that mistakes with profound implications are made when such a suc- cession of choices, contingent events, and conditional probabilities are consid- ered intuitively rather than systematically (10~. The systematic approach can be used to set parameters for clinical research. For example, the design of a clinical trial can establish the clinically important difference in efficacy, used with alpha and beta errors to calculate sample size, by using a sensitivity analysis that varies relative efficacy of the new interven- tion as opposed to the available alternatives. Such an approach may be even more valuable when establishing parameters for performance of diagnostic tests rather than therapeutic interventions. The following example illustrates this point. Currently, there is substantial excitement and some controversy about the development of a new serologic assay that could protect the blood supply against the transmission of non-A non-B hepatitis. Yet tests that would offer some protection have been available for many years. Alanine aminotransferase (ALT) testing has not been implemented because, with a sensitivity of only 30 percent and a specificity of 92 percent (i.e., 8 percent of donated blood would have to be discarded because of false-positive results), it did not seem worth the cost. However, when one considers the health and cost implications of non-A non-B hepatitis, it becomes apparent that there is a broad range of sensitivity- specificity pairs for which ALT testing would not only prevent morbidity and mortality but also save health care dollars (11~. The estimated sensitivity and specificity of ALT testing fall well within this range (see Figure 7.4~. The term "fallback position," used earlier, is adapted from Phelps and Muslin (12), who have used a similar approach to establish priorities for the evaluation of magnetic resonance imaging (MRI). They perform decision anal- yses to put the information that MRI might provide in a particular clinical situa- tion in an outcome-oriented context. They first ask whether MRI would make a positive contribution to health outcome if it were perfectly sensitive and specif- ic. In other words, is knowing the diagnosis for the condition under considera- tion at a particular point in its course going to make a difference? If that hurdle is passed, how much beuer does MRI have to be than a less costly fallback test (or, in the case of tests other than MRI that involve risk of morbidity, a less

OCR for page 78
84 ALBERT G. MU~EY, JR. En 3,000 o ~ 2,0()0 llJ At > 1,000 G 111 co 8 -1 loon of o -2,000 Sensitivity (percent) l 50 45 40 35 30 25 60 70 80 SPECIFICITY (percent) 90 100 FIGURE 7.4 The net cost per case of non-A non-B hepatitis prevented by use of a screening test as it vanes with Me sensitivity and specificity of Me test. SOURCE: Silverste~n MD, Mulley AG, Dienstag JL. Should donor blood be screened for elevated alan~ne am~notransferase levels? A cost- effectiveness analysis. Journal of the American Medical Association 1984;252:2839-2845. risky test) to justify its use? Phelps and Muslin answer the question with a receiver operating characteristic curve that displays a "challenge region" for MRI (see Figure 7.5~. Again, the result is more targeted clinical investigation with clear and explicit definition of the "clinically important difference." SYNTHESIZING, INTERPRETING, AND DISSEMINATING RESULTS OF CLINICAL RESEARCH The characteristics of decision analysis that make it valuable for setting pri- orities and parameters are useful in bringing the results of research to clinical practice. Sensitivity and threshold analyses can be used to consider the limits to external validity of a clinical study more explicitly and systematically. The eff~- cacy or complication rates seen in the highly selected populations that partici- pate in clinical trials can be varied across a wide range to determine the impact of either decreased effectiveness or greater risk on overall outcome. The previ- ously mentioned TARP analysis is an example (see Figure 7.3~. The same approach has been used to define levels of risk that warrant pre- ventive interventions. Figure 7.6 summarizes the results of an analysis per- fo~med to determine indications for vaccination against hepatitis B. based pri- marily on cost considerations (13~. The figure displays estimates of the cost per prevented case of hepatitis when the vaccine is used in populations facing dif-

OCR for page 78
ROLE OF DECISION ANALYSIS IN CLINICAL PRACTICE 111 > E En o IL I' 111,,111' I Challengel I I I I Region I1 11~ / ~ / - Original ROC Curve l _ FALSE-POSITIVE RATE 85 FIGURE 7.5 Use of receiver operating characteristic (ROC) curves to define die challenge region for a new diagnostic test. The original ROC curve displays the performance characteristics of an available test. The challenge region displays the range of improved performance characteristics that would justify a more costly or risky alternative. SOURCE: Phelps CE, Muslin AI. Focusing tech- nology assessment using medical decision theory. Medical Decision Making 1988;8:279-289. ferent attack rates. Costs below zero indicate that the vaccine would actually save money. This analysis is a good example of the iterative, bidirectional pro- cess that decision analysis can facilitate; it puts the results of the randomized trial in context, while focusing additional research efforts aimed at identifying hepatitis risk for different populations. DISTINCTIONS BETWEEN MATTERS OF FACT, OR EVIDENCE, AND VALUE JUDGMENTS Clinical research generally addresses questions that are represented by prob- abilities in a decision analysis. These are matters of evidence or fact. When translating these results to clinical practice, it is important to recognize the vari- ability of different patients' utilities (14~. For example, the vertical axis in Figure 7.3 may be related indirectly to many objective measures of symptom severity for men with prostate disease. That analysis further demonstrated that

OCR for page 78
86 1 0,000 city cot 8,000 an ~ Oh O ~ O Z 6,000 LU > IL cr MEL LL ' ~ 4,000 LU CO LLI ~ 2,000 at cL LL I o (1 ,000) ALBERT G. MU~FY, JR. / . . a, - \: . ..~. / 1/2 Base Case Hepatitis Costs Base Case Hepatitis Costs //, 2x Base Case Hecatitis Costs . ,/ 4x Base Case Hepatitis Costs I -.1 1 1 1 1 0 1 2 3 4 ~ v ~ ~ ANNUAL ATTACK RATE (percent) 6 7 FIGURE 7.6 Net medical care costs per case of hepatitis prevented by vaccination of susceptible populations with different annual attack rates. SOURCE: Mulley AG, Silverstein MD, Dienstag JL. Indications for use of hepatitis B vaccine, based on cost-effectiveness analysis. New England Journal of Medicine 1982;307:60652. the range of values justifying surgery on the basis of baseline symptoms and operative mortality is increasingly constrained as patients' dislike of the poten- tial loss of sexual function increases (61. PRIORITIES FOR DECISION ANALYSTS Decision analysis is underused at the interface between clinical research and clinical practice. Those who would like to see its use increase should address a number of priorities. First, there must be more and better examples of iterative work involving clinical investigation, decision analysis, and clinical practice. Too often decision analysts do their work and leave it at that. Too often clini- cal investigators pay no attention. There needs to be more collaborative work. Second, the field needs better integration of prescriptive decision theory (i.e., decision analysis) and descriptive decision theory. Prospect theory, regret theo- ~y, and other formulations of usual deviations from the expected utility model

OCR for page 78
ROLE OF DECISION ANALYSIS IN CLINICAL PRACTICE 87 promote understanding of the analytic sponges of He prescriptive method and its problems. Third, we need better measurements of patients' preferences (141. Methods borrowed from economists and psychome~icians may suffice for the scaling task. But we need a clinical theory based on the variability of prefer- ences among persons and across time. Finally, decision theorists should pay more attention to the single-actor perspective of both prescriptive and descr~p- tive decision theory. Most clinical decisions are made by parties who share unequally in information, experience, ability to make Be relevant value judg- ments, and decision-making responsibility. Ideally, as decision analysis pro- gresses in these directions it will become even more valuable at the interface between clinical investigation and clinical practice. REFERENCES 1. Kassirer JP, Moskowitz AJ, Lau K, Pauker SG. Decision analysis: A progress report. Annals of Internal Medicine 1987;106:275-291. 2. Pauker SG, Kassirer JP. Decision analysis. New England Journal of Medicine 1987;316:250-257. 3. Raiffa H. Decision Analysis: Introductory Lectures on Choice Under Uncertainty. Reading, Mass.: Addison-Wesley, 1968. 4. Weinstein MC, Fineberg HV. Clinical Decision Analysis. Philadelphia: W. B. Saunders, 1980. 5. Sox HC Jr, Blatt MA, Higgins MC, Marton KI. Medical Decision Making. Boston: Butterworth, 1988. 6. Barry MJ, Mulley AG, Fowler FJ, Wennberg JE. Watchful waiting versus immedi- ate transurethral resection for symptomatic prostatism: The importance of patients' preferences. Journal of the American Medical Association 1988;259:3010-3017. 7. Mulley AG. Medical decision making and practice variation. In Andersen TF, Mooney G (eds). The Challenges of Medical Practice Variation. London, U.K.: Macmillan, in press. 8. Kahneman D, Slovic P. Tversky A feds). Judgement Under Uncertainty: Heuristics and Biases. Cambridge, U.K.: Cambridge University Press, 1982. 9. Elstein AS. Clinical judgment: Psychological research and medical practice. Science 1976;194:696-700. 10. Eddy DM, Billings J. The quality of medical evidence: Implications for quality of health care. Health Affairs 1988;7:19-32. 11. Silverstein MD, Mulley AG, Dienstag JL. Should donor blood be screened for ele- vated alanine aminotransferase levels? A cost-effectiveness analysis. Journal of the American Medical Association 1984;252:2839-2845. 12. Phelps CE, Muslin AI. Focusing technology assessment using medical decision theory. Medical Decision Making 1988;8:279-289. 13. Mulley AG, Silverstein MD, Dienstag JL. Indications for use of hepatitis B vac- cine, based on cost-effectiveness analysis. New England Journal of Medicine 1982;307:644 652. 14. Mulley, AG. Assessing patients' utilities: Can the ends justify the means? Medical Care 1989;27:S269~281.