CHAPTER 2: IDENTIFYING AND CHARACTERIZING UNCERTAINTY
Workshop presentation by Tarek A. Hammad: Key Sources of Uncertainty in the Assessment of Benefits and Risks of Pharmaceuticals and Associated Challenges
Berlin, J. A., B. J. Crowe, E. Whalen, H. A. Xia, C. E. Koro, and J. Kuebler. 2012. Meta-analysis of clinical trial safety data in a drug development program: Answers to frequently asked questions. Clinical Trials 10:20-31.
Eichler, H. G., B. Bloechl-Daum, D. Brasseur, A. Breckenridge, H. Leufkens, J. Raine, T. Salmonson, C. K. Schneider, and G. Rasi. 2013. The risks of risk aversion in drug regulation. Nature Reviews Drug Discovery 12(12):907-916.
Hammad, T. A., G. A. Neyarapally, S. Iyasu, J. A. Staffa, and G. Dal Pan. 2013. The future of population-based postmarket drug risk assessment: A regulator’s perspective. Clinical Pharmacology & Therapeutics 94(3):349-358.
Maro, J. C., and J. S. Brown. 2011. Impact of exposure accrual on sequential postmarket evaluations: A simulation study. Pharmacoepidemiology and Drug Safety (20):1184-1191.
__________________
1 This bibliography contains resources provided during presentations by workshop speakers but not necessarily cited in the workshop summary report. These resources are included here as additional direction for readers interested in further exploration of the topics discussed at the workshop.
Workshop presentation by Deborah A. Zarin: ClinicalTrials.gov and Addressing Challenges in Finding Evidence
Earley, A., J. Lau, and K. Uhlig. 2012. Haphazard reporting of deaths in clinical trials: A review of cases of ClinicalTrials.gov records and matched publications: A cross-sectional study. http://bmjopen.bmj.com/content/3/1/e001963.full.pdf+html (accessed September 13, 2014).
Jüni, P., A. W. S. Rutjes, and P. A. Dieppe. 2002. Are selective COX 2 inhibitors superior to traditional non-steroidal anti-inflammatory drugs? British Medical Journal 324:1287-1288. http://www.bmj.com/content/324/7349/1287.full.pdf+html (accessed September 13, 2014).
Riveros, C., A. Deschartres, E. Perrodeau, R. Haneef, I. Boutron, and P. Ravaud. 2013. Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals. PLoS Medicine 10(12):e1001566.
Ross, J. S., T. Tse, D. A. Zarin, H. Xu, L. Zhou, and H. M. Krumholz. 2012. Publication of NIH funded trials registered in ClinicalTrials.gov: Cross-sectional analysis. British Medical Journal 344:d7292.
Zarin, D. A., and T. Tse. 2013. Unambiguous identification of obesity trials. New England Journal of Medicine 368:580-581.
Zarin, D. A., T. Tse, R. J. Williams, R. M. Califf, and N. C. Ide. 2011. The ClinicalTrials.gov results database: Update and key issues. New England Journal of Medicine 364:852-860.
Workshop presentation by Michaela Kiernan:
Identifying and Retaining Subgroups in Clinical Trials in the Context
of Uncertainty About the External Validity of Clinical Trials
Goldberg, J. H., and M. Kiernan. 2005. Innovative techniques to address retention in a behavioral weight-loss trial. Health Education Research 20:439-447.
Kiernan, M., S. D. Brown, D. E. Schoffman, K. Lee, Katherine, A. C. King, C. B. Taylor, N. C. Schleicher, and M. G. Perri. 2013. Promoting healthy weight with “Stability Skills First”: A randomized trial. Journal of Consulting and Clinical Psychology 81(2):336-346.
Moher, D., S. Hopewell, K. F. Schulz, V. Montori, P. C. Gøtzsche, P. J. Devereaux, D. Elbourne, M. Egger, and D. G. Altman. 2010. CONSORT 2010 Explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. British Medical Journal 340:c869.
Workshop presentation by Sebastian Schneeweiss: Addressing Uncertainties in the Evaluation of Pharmaceutical Benefits and Harms During the Market Access Phase
Gagne, J. J., S. V. Wang, J. A. Rassen, and S. Schneeweiss. 2014. A modular, prospective, semi-automated drug safety monitoring system for use in a distributed data environment. Pharmacoepidemiology and Drug Safety 23:619-627.
Patrick, A. R., J. M. Franklin, M. C. Weinstein, R. J. Glynn, and S. Schneeweiss. 2013. Sequential value-of-information assessment for prospective drug safety monitoring using claims databases: The comparative safety of prasugrel v. clopidogrel. Medical Decision Making 33(7):949-960.
Schneeweiss, S., and J. Avorn. 2009. Antipsychotic agents and sudden cardiac death: How should we manage the risk? New England Journal of Medicine 360:294-296.
Workshop presentation by John P. A. Ioannidis: Systematic Approaches to Assessing the Internal and External Validity of Randomized Controlled Trials
Kappagoda, S., and J. P. A. Ioannidis. 2012. Neglected tropical diseases: Survey and geometry of randomised evidence. British Medical Journal 345:e6512.
Kim, D., J. Y. Tang, and J. P. A. Ioannidis. 2014. Network geometry shows evidence sequestration for medical vs. surgical practices: Treatments for basal cell carcinoma. Journal of Clinical Epidemiology 67(4):391-400.
Lathyris, D. N., N. A. Patsopoulos, G. Salanti, and J. P. A. Ioannidis. 2010. Industry sponsorship and selection of comparators in randomized clinical trials. European Journal of Clinical Investigation 40(2):172-182.
Mauri, D., N. P. Polyzos, G. Salanti, N. Pavlidis, and J. P. A. Ioannidis. 2008. Multiple-treatments meta-analysis of chemotherapy and targeted therapies in advanced breast cancer. Journal of the National Cancer Institute 100:1780-1791.
Mills, E. J., J. P. A. Ioannidis, and T. K. Schünemann. 2013. Demystifying trial networks and network meta-analysis. British Medical Journal 346:f2914.
Pereira, T. V., R. I. Horwitz, and J. P. A. Ioannidis. 2012. Empirical evaluation of very large treatment effects of medical interventions. Journal of the American Medical Association 308(16):1676-1684.
Salanti, G., S. Dias, N. J. Welton, A. E. Ades, V. Golfinopoulos, M. Kyrgiou, D. Mauri, and J. P. A. Ioannidis. 2010. Evaluating novel agent effects in multiple-treatments meta regression. Statistics in Medicine 29(23):2369-2383.
Saquib, N., J. Saquib, and J. P. A. Ioannidis. 2013. Practices and impact of primary outcome adjustment in randomized controlled trials: Meta-epidemiologic study. British Medical Journal 347:f4313.
Savovic, J., H. E. Jones, D. G. Altman, R. J. Harris, P. Jüni, J. Pildal, B. Als-Nielsen, E. M. Balk, C. Gluud, L. L. Gluud, J. P. A. Ioannidis, K. F. Schulz, R. Beynon, N. J. Welton, L. Wood, D. Moher, J. J. Deeks, and J. A. Sterne. 2012. Influence of reported study design characteristics on intervention effect estimates from randomized controlled trials. Annals of Internal Medicine 157(6):429-438.
Schulz, K. F., I. Chalmers, R. J. Hayes, and D. G. Altman. 1995. Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association 273(5):408-413.
Siontis, K. C. M., E. Evangelou, and J. P. A. Ioannidis. 2011. Magnitude of effects in clinical trials published in high-impact general medical journals. International Journal of Epidemiology 40:1280-1291.
CHAPTER 3: THE REGULATORS’ CHALLENGE
Workshop presentation by Mary H. Parks:
Challenges to the Regulator in Communicating
Uncertainties in Risks of Approved Pharmaceuticals
FDA (U.S. Food and Drug Administration). 2013. Structured approach to benefit–risk assessment in drug regulatory decision making, Draft PDUFA V implementation plan, February 2013, Fiscal Years 2013-2017. Washington, DC: FDA.
Workshop presentation by Robyn Lim:
A Regulatory Approach to Uncertainty for the “Real World”
Hammond, J. S., R. L. Keeney, and H. Raiffa. 1999. Smart choices. A practical guide to making better decisions. Boston, MA: Harvard Business School Press.
Spiegelhalter, D. J. 2010. Quantifying and visualising uncertainty, presentation to Kings College, London, United Kingdom. http://www.mth.kcl.ac.uk/finmath/semfiles/DS.pdf (accessed September 13, 2014).
Workshop presentation by Theresa Mullin:
FDA Patient-Focused Drug Development Initiative
FDA (U.S. Food and Drug Administration). 2013. Structured approach to benefit–risk assessment in drug regulatory decision making, Draft PDUFA V implementation plan, February 2013, Fiscal Years 2013-2017. Washington, DC: FDA.
CHAPTER 4: BASIC METHODOLOGIES AND APPLICATIONS FOR UNDERSTANDING AND EVALUATING UNCERTAINTY
Workshop presentation by M. Granger Morgan:
Some Thoughts on Expert Elicitation
Evans, J. S. 1994. A distributional approach to characterizing low-dose cancer risk. Risk Analysis 14(1):25-34.
Evans, J. S., G. M. Gray, R. L. Sielken, A. E. Smith, C. Valdez-Flores, and J. D. Graham. 1994. Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency. Regulatory Toxicology and Pharmacology 20(1 Pt 1):15-36.
Henrion, M., and B. Fischhoff. 1986. Assessing uncertainty in physical constants. American Journal of Physics 54(9):791-798.
Morgan, M. G. 1998. Uncertainty analysis in risk assessment. Human and Ecological Risk Assessment 4(1):25-39.
Morgan, M. G. 2014. The use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings of the National Academy of Sciences of the United States of America 111(20):7176-7184.
Morgan, M. G., M. Henrion, and M. Small. 1990. Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge, United Kingdom, and New York: Cambridge University Press.
Tversky, A., and D. Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science 185(4157):1124-1131. Also: Kahneman, D., P. Slovic, and A. Tversky (Eds.). 1982. Judgment under uncertainty: Heuristics and biases. Cambridge, United Kingdom, and New York: Cambridge University Press.
Wallsten, T. S., D. V. Budescu, A. Rapoport, R. Zwick, and B. Forsyth. 1986. Measuring the vague meanings of probability terms. Journal of Experimental Psychology: General 155(4):348-365.
Workshop presentation by Joel Greenhouse:
Bayesian Approaches to Evaluating Clinical Trial Data
Greenhouse, J. B., and H. Seltman. 2005. Using prior distributions to synthesize historical evidence: Comments on the Goodman-Sladky case study of IVIg in GBS. Clinical Trials 2(4):311-318.
Kass, R., and J. B. Greenhouse. 1989. Comment on “Investigating therapies of potentially great benefit: ECMO” by J. Ware. Statistical Science 4:31-37.
Spiegelhalter, D. J., K. R. Abrams, and J. P. Myles. 2004. Bayesian approaches to clinical trials and health care evaluation. Chichester, England: John Wiley & Sons.
Workshop presentation by David R. Mandel:
Experiences in Implementing Uncertainty Assessments
in the Defense/Intelligence Communities
Kent, S. 1964. Words of estimative probability. Reprinted in Steury (Ed.), Sherman Kent, and the Board of National Estimates. Washington, DC: Central Intelligence Agency, Center for the Study of Intelligence. https://www.cia.gov/library/center-for-the-study-ofintelligence/csi-publications/books-and-monographs/sherman-kent-and-the-boardof-national-estimates-collected-essays/6words.html (accessed August 14, 2014).
Kimmelman, J., J. Brehaut, D. Fergusson, A. J. London, D. Mandel, I. Shrier, and R. Steele. 2013-2016. Can researchers accurately predict trial outcomes? Work in progress, funded by Canadian Institutes of Health Research.
Mandel, D. R. 2007. Toward a concept of risk for effective military decision making. Defence Research and Development Canada, Toronto Technical Report 2007-124. Toronto, Canada: Defence Research and Development Canada. http://cradpdf.drdc-rddc.gc.ca/PDFS/unc68/p529014.pdf (accessed August 23, 2014).
Workshop presentation by Joseph Arvai:
A Structured Approach to Risk–Benefit Assessment
Arvai, J., and L. Rivers III (Eds.). 2013. Effective risk communication. Brighton, United Kingdom: Routledge (Earthscan Risk in Society).
Fischhoff, B. 1982. Judgment and decision making. London, United Kingdom, and New York: Routledge (Earthscan Risk in Society).
Gregory, R., L. Failing, M. Harstone, G. Long, T. McDaniels, and D. Ohlson. 2012. Structured decision making: A practical guide to environmental management choices. Indianapolis, IN: Wiley-Blackwell.
Keeney, R. L. 1996. Value-focused thinking: A path to creative decisionmaking. Cambridge, MA: Harvard University Press.
NRC (National Research Council). 2003. Understanding risk: Informing decisions in a democratic society. Washington, DC: The National Academies Press.
Slovic, P. 2010. The feeling of risk: New perspectives on risk perception. Abingdon, Oxon, United Kingdom: Routledge (Earthscan Risk in Society).
Workshop presentation by Timothy McDaniels:
Eliciting Values for Risk Management Choices:
Principles and Approaches
FDA (U.S. Food and Drug Administration). 2009. Guidance for industry: Format and content of proposed Risk Evaluation and Mitigation Strategies (REMS), REMS assessments, and proposed REMS modifications—Draft guidance. http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM184128.pdf (accessed August 14, 2014).
Workshop presentation by Charles F. Manski:
Adaptive Partial Drug Approval
Eichler, H. G., K. Oye, L. G. Baird, E. Abadie, J. Brown, C. L. Drum, J. Ferguson, S. Garner, P. Honig, M. Hukkelhoven, J. C. W. Lim, R. Lim, M. M. Lumpkin, G. Neil, B. O’Rourke, E. Pezalla, D. Shoda, V. Seyfert-Margolis, E. V. Sigal, J. Sobotka, D. Tan, T. F. Unger, and G. Hirsch. 2012. Adaptive licensing: Taking the next step in the evolution of drug approval. Clinical Pharmacology & Therapeutics 91(3):426-437.
Manski, C. 2009. Diversified treatment under ambiguity. International Economic Review 50:1013-1041.
Manski, C. 2013. Public policy in an uncertain world. Boston, MA: Harvard University Press.
Tamura, R. N., D. E. Faries, J. S. Andersen, and J. H. Heiligenstein. 1994. A case study of an adaptive clinical trial in the treatment of out-patients with depressive disorder. Journal of the American Statistical Association 89:768-776.
CHAPTER 5: COMMUNICATING UNCERTAINTY
Workshop presentation by Baruch Fischhoff:
Communicating Uncertainty
Bruine de Bruin, W., A. Parker, and B. Fischhoff. 2007. Individual differences in adult decision-making competence (A-DMC). Journal of Personality and Social Psychology 92:938-956.
Campbell, P. 2011. Understanding the receivers and the receptions of science’s uncertain messages. Philosophical Transactions of the Royal Society 369:4891-4912.
Downs, J. S., W. Bruine de Bruin, and B. Fischhoff. 2008. Patients’ vaccination comprehension and decisions. Vaccine 26:1595-1607.
Fischhoff, B. 1992. Giving advice: Decision theory perspectives on sexual assault. American Psychologist 47:577-588.
Fischhoff, B. 2011. Communicating the risks of terrorism (and anything else). American Psychologist 66:520-531.
Fischhoff, B. 2012. Communicating uncertainty: Fulfilling the duty to inform. Issues in Science and Technology 29:63-70.
Fischhoff, B., and J. Kadvany. 2011. Risk: A very short introduction. Oxford, United Kingdom: Oxford University Press.
Fischhoff, B., and D. Scheufele (Eds.). 2012. Arthur M. Sackler Colloquia: The science of science communication. Proceedings of the National Academy of Sciences of the United States of America online supplement. http://onlinedigeditions.com/publication/?i=174803 (accessed August 14, 2104). [Overviews and videos from both 2012 Science of Science Communication colloquia are available at http://www.nasonline.org/programs/sackler-colloquia/completed_colloquia/science-communication.html and http://www.nasonline.org/programs/sackler-colloquia/completed_colloquia/agendascience-communication-II.html (accessed August 14, 2014).]
Fischhoff, B., S. Lichtenstein, P. Slovic, S. L. Derby, and R. L. Keeney. 1981. Acceptable risk. New York: Cambridge University Press.
Fischhoff, B., W. Bruine de Bruin, U. Güvenç, D. Caruso, and L. Brilliant. 2006. Analyzing disaster risks and plans: An avian flu example. Journal of Risk and Uncertainty 33:133-151.
Fischhoff, B., N. T. Brewer, and J. S. Downs, Eds. 2011. Communicating risks and benefits: An evidence-based user’s guide. Washington, DC: U.S. Food and Drug Administration. http://www.fda.gov/AboutFDA/ReportsManualsForms/Reports/ucm268078.htm (accessed August 14, 2014).
Florig, H. K., M. G. Morgan, K. M. Morgan, K. E. Jenni, B. Fischhoff, P. S. Fischbeck, and M. DeKay. 2001. A deliberative method for ranking risks (2): Evaluation of validity and agreement among risk managers. Risk Analysis 21:923-938.
Funtowicz, S. O., and J. Ravetz. 1990. Uncertainty and quality in science for policy. Dordrecht, The Netherlands: Kluwer Academic Publishers.
Kahneman, D. 2011. Thinking, fast and slow. New York: Farrar Strauss & Giroux.
Krishnamurti, T., S. L. Eggers, and B. Fischhoff. 2008. The impact of over-the-counter availability of “Plan B” on teens’ contraceptive decision-making. Social Science & Medicine 67:618-662.
Lichtenstein, S., and P. Slovic (Eds.). 2006. The construction of preference. New York: Cambridge University Press.
Morgan, M. G., and M. Henrion. 1990. Uncertainty. New York: Cambridge University Press.
NRC (National Research Council). 2011. Fischhoff, B., and C. Chauvin (Eds.). Intelligence analysis: Behavioral and social science foundations. Washington, DC: The National Academies Press. http://www.nap.edu/catalog.php?record_id=13062 (accessed August 14, 2014).
Schwartz, L. M., and S. Woloshin. 2013. The drug facts box: Improving the communication of prescription drug information. Proceedings of the National Academy of Sciences of the United States of America 110:14069-14074.
Slovic, P. (Ed.). 2000. Perception of risk. London, United Kingdom: Earthscan.
Workshop presentations by Lisa M. Schwartz and Steven Woloshin:
Communicating Uncertainty About Benefits
and Harms of Pharmaceutical Products
Downing, N. S., J. A. Aminawung, N. D. Shah, H. M. Krumholz, and J. S. Ross. 2014. Clinical trial evidence supporting FDA approval of novel therapeutic agents, 2005-2012. Journal of the American Medical Association 311(4):368-377.
Schwartz, L. M., and S. Woloshin. 2011. Communicating uncertainties about prescription drugs to the public: A national randomized trial. JAMA Archives of Internal Medicine 171(16):1463-1468.
Schwartz, L. M., S. Woloshin, A. Andrews, and T. A. Stukel. 2012. Influence of medical journal press releases on the quality of associated newspaper coverage: Retrospective cohort study. British Medical Journal 344:d8164.
This page intentionally left blank.
