Integration of Data-Based Evidence and Expert Opinion in Decision Making
In certain cases, it may be necessary to update research data with new findings and with stakeholder opinions (where stakeholders are defined to be mission specialists, National Aeronautics and Space Administration [NASA] directors, managers, and flight surgeons). Bayesian updating may be one strategy for integrating stakeholder opinions with data from research studies. Such a strategy can accommodate contrasting points of view from stakeholders expressed in a subjective manner. It is necessary that the data from different research studies measure the same underlying factor (e.g., diastolic blood pressure, depression) on the same scale (e.g., millimeters of mercury for blood pressure, Hamilton scale for depression).
This updating strategy consists of the following steps: (1) selection of a sample of stakeholders; (2) elicitation of probability information from these stakeholders; (3) translation of this information to statistical distributions, called “prior” distributions, for each contrasting view of the stakeholders; (4) assignment of an “importance” weight to each of these prior distributions for each of the contrasting views; (5) with these importance weights, derivation of a “summary prior” distribution by taking a weighted combination of the contrasting prior distributions; (6) derivation of a “summary likelihood” pooling all study datasets while accounting for the varying variability and sample sizes across the study datasets; (7) derivation of a “sum-
mary posterior” distribution from the summary prior distribution and summary likelihood; (8) choosing a utility function to incorporate costs and stakeholders’ sensitivity to such costs; and (9) decisions based on regrets or opportunity costs in cost–benefit or risk–benefit models by weighing outcome information from summary posterior distribution (e.g., mean differences, risk differences, risk ratios, odds ratios, and interactions involving these effects) against utility functions. Each of these steps is described in more detail below.
Steps 1–2. Selection of a sample of stakeholders and elicitation of probability information from these stakeholders: The selection process should at least be comprehensive, maximizing the number of contrasting points of view, if the process is not random. Stakeholders’ prior opinions will be elicited with questionnaires. In these questionnaires, stakeholders will be asked to provide ranges of probabilities of confidence in positive and negative results. The design of the questionnaire will be selected from several different designs available in the research literature. Chaloner and Rhame (2001) presented an interactive approach based on iterative elicitation from physicians enhanced by real-time iterative and graphical feedback to the physicians of their quantified opinions. Parmar et al. (1994, 2001) and Spiegelhalter et al. (1994) presented a questionnaire for eliciting prior distributions in a pair of large randomized trials conducted as part of the British Medical Research Council Cancer Trials. In a hepatocellular carcinoma clinical trial, Tan et al. (2003) also used such a questionnaire to elicit prior information on the equivalence between surgery with adjuvant therapy versus surgery alone on recurrence-free survival. Alternatively, a series of individual description formats have been developed by Vennix et al. (1994). These individual questionnaire formats focus on the different phases proposed by for elicitation: (1) the positioning phase, which defines the context of the information; (2) the description phase, which guides stakeholders through four aspects of description (visual, verbal, textual, and graphic); and (3) the discussion phase, in which the individual descriptions from phase 2 are examined and compared.
Step 3. Translation of this information to prior distributions: Individual histograms representing the prior beliefs of each investigator can be constructed from the relative probability values that stakeholders may be asked to provide in Step 2. Following Spiegelhalter et al. (1994), these probability values may be summarized across stakeholders with similar opinions to then construct “overall histograms”and “skeptical histograms.”
These histograms will represent the overall (or clinical) and skeptical (or cautious) prior distributions associated with the stakeholders’ views of the problem. A skeptical prior distribution corresponds to the beliefs of individuals who are reluctant to accept alternative hypotheses of interest to the investigators. The resulting histograms can be transformed to the scale on which research data have been collected (Tan et al., 2003).
Steps 4–7. Determination of multivariate prior distributions from multiple stakeholders, estimation of summary likelihoods from multiple datasets, resulting in a derivation of a summary posterior distribution: Such a procedure is based on Bayes’ rule and entails intractable integration resulting in simulation-based integration (e.g., Markov Chain Monte Carlo), which many commercially available software packages now offer (Spiegelhalter et al., 1994).
Steps 8. Choosing a utility function to incorporate costs and stakeholders’ sensitivity to such costs: Such functions involve determining costs from implementing mitigation strategies and reduced costs from preventing problem outcomes (Pliskin et al., 1980; Berger, 1985; Lindley, 1985; Gold et al., 1996). Sensitivity analysis of the overall procedure outlined here includes varying such cost estimates (Matchar and Samsa, 1999; Matchar et al., 1997).
Step 9. Decisions based on regrets or opportunity costs from weighing information on outcomes under mitigation strategies against outcomes under the absence of mitigation strategies: Regrets are based on loss functions as contrasts between decisions that lead to optimal utility benefits and the utility benefits based on observed or predicted data. The expected loss functions allow the incorporation of research data and previous opinions of stakeholders by integrating utility functions for the optimal and observed decisions with respect to the summary posterior distributions (Berger, 1985: Lindley, 1985). For complex sequences of branching decisions based on outcomes of previous decisions, backward induction algorithms may be used (Bellman, 1957).
Overall, such decision processes present a complex web of different statistical procedures, research datasets, and opinions by stakeholders. This complex web is sensitive to selected procedures and corresponding assumptions; thus, this sensitivity is assessed by varying assumptions and operational procedures (Matchar and Samsa, 1999). Varying assumptions, procedures, and information used for forming utility functions and prior distribution may be done formally with quantified ranges of possible values
and model-averaging techniques or informally by choosing plausible values of model and prior distribution parameters (Spiegelhalter et al., 1994).
Bellman RE. 1957. Dynamic Programming. Princeton, NJ: Princeton University Press.
Berger JO. 1985. Statistical Decision Theory and Bayesian Analysis, 2nd Ed. New York: Springer-Verlag.
Chaloner K, Rhame FS. 2001. Quantifying and documenting prior beliefs in clinical trials. Statistics in Medicine 20: 581–600.
Gold MR, Siegel JE, Russell LB, Weinstein MC (eds). 1996. Cost-Effectiveness in Health and Medicine. Oxford, UK: Oxford University Press.
Lindley DV. 1985. Making Decisions, 2nd Ed. New York: Wiley.
Matchar DB, Samsa GP. 1999. Using outcome data to identify best medical practice: the role of policy models. Hepatology 29 (6 Suppl): 36S–39S.
Matchar DB, Samsa GP, Mathews JR, Aneukiewicz M, Parmigiani G, Hasselblad V, Wolf PA, D’Agostino RB, Lipscomb J. 1997. The Stroke Prevention Policy Model (SPPM): linking evidence and clinical decisions. Ann. Int. Med. 127: 704–711.
Parmar MKB, Griffiths GO, Spiegelhalter DJ, Souhami RL, Altman DG, van der Scheuren E, CHART steering committee. 2001. Monitoring of large randomised clinical trials: a new approach with Bayesian methods. Lancet 358(9279): 375–381.
Parmar MKB, Spiegelhalter DJ, Freedman LS. 1994. The CHART trials: Bayesian design and monitoring in practice. Statistics in Medicine 13: 1297–1312.
Pliskin JS, Shepard D, Weinstein MC. 1980. Utility functions for life years and health status: theory, assessment, and application. Operations Research 28: 206–224.
Spiegelhalter DJ, Freedman LS, Parmar MKB. 1994. Bayesian approaches to randomised trials. J. R. Stat. Soc. A 157: 357–387.
Tan SB, Chung YF, Tai BC, Cheung YB, Machin D. 2003. Elicitation of prior distributions for a phase III randomized controlled trial of adjuvant therapy with surgery for hepatocellular carcinoma. Control. Clin.Trials 24(2): 110–121.
Vennix JAM, Anderson DF, Richardson GP, Rohrbaugh J. 1994. Model building for group decision support: issues and alternatives in knowledge elicitation. In Morecroft J, Sterman J. (eds.) Modeling for Learning Organizations. Portland, OR: Productivity Press.