Characterizing and Communicating Uncertainty in the Assessment of Benefits and Risks of Pharmaceutical Products: Workshop Summary (2014)

1

Introduction¹

Despite the extensive body of evidence that informs regulatory decisions on pharmaceutical products, significant uncertainties persist, including the underlying variability in human biology, factors associated with the chemistry of a drug, and limitations in the research and clinical trial process itself that might limit the generalizability of results. As a result, regulatory reviewers are consistently required to draw conclusions about a drug’s safety and efficacy from imperfect data. Efforts are under way within the drug development community to enhance the evaluation and communication of the benefits and risks associated with pharmaceutical products, aimed at increasing the predictability, transparency, and efficiency of pharmaceutical regulatory decision making. The U.S. Food and Drug Administration (FDA) is developing an enhanced structured approach to benefit–risk assessments² in drug regulatory decision making

__________________

¹ The planning committee’s role was limited to planning the workshop, and the workshop summary has been prepared by the workshop rapporteurs as a factual summary of what occurred at the workshop. Statements, recommendations, and opinions expressed are those of individual presenters and participants, and are not necessarily endorsed or verified by the Forum or the Institute of Medicine, and they should not be construed as reflecting any group consensus.

² Terminology in defining benefit–risk assessment varies. Some have indicated the most precise way to reference the separate concepts of benefit and risk, and the different tradeoffs, is to leave the terms unhyphenated because “benefit–risk” assessment could imply reducing benefit and risk to one common metric (as in cost–benefit analysis). Many (e.g., Lim, 2014) suggest the term “benefit–risk” is inconsistent and inherently confusing because risk has a wide range of meanings, noting that a more appropriate term is “benefit–harm”

to better communicate this aspect of the human drug review process.³ As FDA has indicated in its draft Prescription Drug User Fee Act (PDUFA) V Implementation Plan (FDA, 2013) (the FDA PDUFA V Plan), identifying and evaluating sources of uncertainty in a regulatory application is an important part of an FDA new drug application reviewers’ work; however, drawing conclusions in the face of uncertainty can be a complex and challenging task. Effectively communicating regulatory decisions necessarily includes explanation of the impact of uncertainty on decision making. The FDA PDUFA V Plan suggests that FDA’s enhanced structured approach is intended to serve as a template for product reviews and a vehicle to explain the basis of regulatory decisions.⁴Box 1-1 provides additional information on the two areas of uncertainty suggested in the FDA PDUFA V Plan as deserving additional attention.

__________________

or “benefit–harm–uncertainty.” For this report we have adopted the most widely used “benefit–risk” terminology for ease of reading and because it was the terminology used in defining the workshop charge.

³ For more information, see http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm326192.htm (accessed August 18, 2014).

⁴ This material is based on Characterizing Uncertainty in the Assessment of Benefits and Risks of Pharmaceutical Products: Workshop in Brief (IOM, 2014), also prepared for this project.

On February 12 and May 12, 2014, the Institute of Medicine’s (IOM’s) Forum on Drug Discovery, Development, and Translation (the Forum) held public workshops at FDA Headquarters in White Oak, Maryland, to advance the development of more systematic and structured approaches to characterize and communicate the sources of uncertainty in the assessment of benefits and risks, and to consider their implications for pharmaceutical regulatory decisions (see Box 1-2 for the Statement of Task). Workshop presentations and discussions on February 12 were convened to explore the science of identifying and characterizing uncertainty in scientific evidence and approaches to translate uncertainties into decisions that reflect the values of stakeholders. The May 12 workshop presentations and discussions explored tools and approaches to communicating about scientific uncertainties to a range of stakeholders in the drug development process. Baruch Fischhoff, Howard Heinz University Professor, Department of Social and Decision Sciences, Department of Engineering

and Public Policy, Carnegie Mellon University, and Robert Ratner, Chief Scientific and Medical Officer, American Diabetes Association, were co-chairs of the workshop planning committee. See Box 1-4 at the end of this chapter for themes identified by the workshop co-chairs.

This report is a summary of the February 12 and May 12, 2014, workshops. Statements, recommendations, and opinions expressed are those of individual presenters and participants and are not necessarily endorsed or verified by the Forum or the IOM, and they should not be construed as reflecting any group consensus. The workshops were webcast live, and online participants were able to contribute to discussion through the hashtag #UncertaintyWorkshopIOM. The presentations, videos, and tweets are archived on the Forum websites.⁵

This summary of the workshop is meant to inform FDA; the scientific research community in academia, government, and regulated industry; policy makers; patient groups; the public; and other stakeholders because they all have an interest in the approaches to characterizing and communicating uncertainty in assessments of benefit and risk of pharmaceutical products. The remainder of this first chapter of the summary provides an overview of the role of uncertainty in FDA’s benefit–risk framework and compiles a brief overview of themes from the workshop as identified by workshop co-chairs Fischhoff and Ratner. Chapter 2 examines the sources of uncertainty in benefit–risk assessments and opportunities to reduce uncertainty through the clinical research process. Chapter 3 considers the challenges of the pharmaceutical regulator in identifying, acknowledging, addressing, and communicating uncertainties in evidence in the context of FDA’s statutory requirements and current efforts to better understand what matters most to patients. Chapter 4 draws on approaches from decision science and statistical techniques to manage uncertainty in the generation of evidence and the regulatory decision-making process. Chapter 5 highlights principles of effective risk communication and potential opportunities to improve the utility of current communication tools and approaches for conveying benefits, risks, and uncertainties to a broad audience, in part through a case study of Tysabri. Chapter 6 concludes this summary of the workshop with views expressed by individual participants during the final workshop session about potential opportunities to move forward in developing approaches for characterizing and communicating uncertainty in the assessments of benefits and risks of pharmaceutical products.

__________________

⁵ For more information, see http://www.iom.edu/BenefitRisk1 (accessed August 20, 2014) and http://www.iom.edu/BenefitRisk2 (accessed August 20, 2014).

THE IMPACT OF UNCERTAINTY ON REGULATORY DECISION MAKING⁶

When a regulatory decision is made, uncertainty can remain about many aspects of a new drug’s performance, said Janet Woodcock, Director, Center for Drug Evaluation and Research (CDER), FDA. As a result, she noted, uncertainty is “central to the evaluation of data,” and can affect our understanding of both benefits and risks. Uncertainty in the drug review process has many sources, all of which, she noted, must be analyzed, quantified to the extent possible, judged, and communicated responsibly (see Box 1-3). FDA’s goal is to bring the best possible science to bear on these tasks, in order to ensure that stakeholders and the public have a clear understanding of both the available evidence and the pending uncertainties, and that stakeholders understand that both evidence and uncertainty are important factors in any given regulatory decision.

Patrick J. Frey, Director, Office of Program and Strategic Analysis, CDER, FDA, introduced FDA’s benefit–risk framework (see Figure 1-1) developed by the agency over several years to delineate the evidence, and accompanying uncertainties, that inform conclusions and decisions about benefits, risks, and the management of risks (e.g., product labeling, Risk Evaluation and Mitigation Strategies, or REMS). Once fully implemented, the framework will serve as both a record of FDA decision making and a tool for communicating the rationale behind regulatory decisions to the public.

FDA currently lacks a systematic approach for dealing with uncertainty, noted Frey. The agency frequently uses advisory committees composed of experts, and in some cases patient representatives, external to government to obtain input on particularly challenging questions about the review of a drug or other issues in drug development and review. The discussions at this workshop are intended by FDA to be the beginning of what will be a multiyear effort by the agency to develop an approach to working through uncertainty that is practical and can be implemented in FDA’s unique regulatory setting. Frey noted that FDA is particularly interested in developing systematic approaches to evaluating uncertainty and perhaps exploring the piloting of such systematic approaches in much the same manner as FDA’s benefit–risk framework has been piloted.

Several academic disciplines already employ effective approaches to characterizing uncertainty and for supporting decisions made under conditions of uncertainty. Frey noted that adapting existing scientific methods for characterizing and assessing uncertainties can lend additional

__________________

⁶ This section is based on presentations by Janet Woodcock, Director, CDER, FDA, and Patrick J. Frey, Director, Office of Program and Strategic Analysis, CDER, FDA.

intellectual credibility to an activity that is unfamiliar to FDA reviewers who are subject-matter experts. Several practitioners of these methods presented them at the workshop, exploring how they might best support decision makers in areas where evidence is limited.

FIGURE 1-1 FDA benefit–risk framework. Color coded areas of the table under the category of “Evidence and Uncertainties” are informed by FDA’s Patient-Focused Drug Development initiative and these IOM workshops, respectively.
NOTE: For more information on FDA’s benefit–risk framework, see the FDA PDUFA V Plan at: http://www.fda.gov/downloads/forindustry/userfees/prescriptiondruguserfee/ucm329758.pdf (accessed September 12, 2014).
SOURCE: Frey, 2014. Presentation at the IOM workshop series on Characterizing and Communicating Uncertainty in the Assessment of Benefits and Risks of Pharmaceutical Products.