Characterizing and Communicating Uncertainty in the Assessment of Benefits and Risks of Pharmaceutical Products: Workshop Summary (2014)

3

The Regulators’ Challenge

Patrick Frey, CDER, FDA, referred to FDA’s multidimensional approach to benefit–risk assessment to frame the challenges that uncer-

tainty poses for regulators. Frey said these assessments, which are the foundation for drug approval decisions, are supported by an extensive body of evidence, informed by the underlying disease and available treatment options, and analyzed through the lens of science, medicine, policy, values, and judgment. Yet uncertainties persist, relating both to the benefits and the risks of products under review.

Although the decision points themselves might be unique to the drug review process, the types of uncertainties that underlie them could be better characterized and addressed by scientific methods that support decision making under uncertainty. For example, Frey noted, establishing predictive expertise in scientifically analyzing uncertainty can be difficult, especially for a process like a drug review, which is driven by scientific evidence. Even though uncertainties exist, reviewers with clinical and regulatory expertise do not typically make explicit the assumptions that might be inherent in dealing with uncertainty in decision making. Furthermore, hypothetical questioning, which could be used to aid in thinking through a decision under uncertainty, is not typical practice in an evidence-driven organization like FDA, noted Frey. A method from the field of decision science, expert elicitation, could be used to guide experts through a process by which their informed opinions on a given area of uncertainty can be synthesized and perhaps even quantified in a way that reveals that uncertainty. Frey indicated that expert elicitation methods might be applicable in the drug regulatory setting, allowing FDA drug reviewers to participate in a process that has been effective for quantifying uncertainty in other areas. Frey noted that such decision science methods have potential in the drug regulatory setting for developing a systematic approach to uncertainty that is both scientifically rigorous and can be practically implemented in FDA’s existing benefit–risk framework.

Francesco Pignatti, Head of Oncology, Hematology, Diagnostics Section, Scientific and Regulatory Management Department, Human Medicines Evaluation Division, European Medicines Agency (EMA), cited parallel efforts in Europe to develop a structured approach to benefit–risk assessments and particularly to further develop the methods and tools that characterize and mitigate uncertainty. Pignatti and Kimby Barton, Director, Bureau of Cardiology, Allergy and Neurological Sciences, Health Canada, indicated that a goal of European and Canadian regulators is also to increase transparency of benefit–risk assessments and explicitly link evaluations of uncertainty with a particular remediating action (i.e., designing a new study, developing a new label to reflect uncertainty, or requiring the development of a risk management system).

UNIQUE CHALLENGES OF THE PHARMACEUTICAL REGULATORY SETTING¹

Baruch Fischhoff, Carnegie Mellon University, reflected on FDA’s efforts to build a benefit–risk framework that quantifies the risks and benefits of a product to the greatest extent possible, but without losing the uncertainties and the context that allow for nuanced interpretation of what is known and what else we would like to know. Fischhoff observed that FDA has separated scientific judgment from regulatory interpretation in the benefit–risk framework, but allowed the acknowledgment of uncertainty throughout the process of assessing benefits and risks. The fact that FDA has not sought a common metric to assess benefit or risk, or both together, but has chosen to treat them as independent variables not to be combined so that each can be characterized in its own right, is, according to Fischhoff, the appropriate approach to inform the regulatory decision maker.

Challenges Facing the Regulator in Communicating Uncertainties in Risks of Approved Products²

Mary H. Parks, Deputy Director, Office of Drug Evaluation II, Office of New Drugs, CDER, FDA, spoke about the uncertainties inherent in the interpretation of scientific evidence in the drug review process, and how those uncertainties are communicated both within the agency and outside.

The FDA PDUFA V Plan states: “To be approved for marketing, a drug must be safe and effective for its intended use.” Parks emphasized that the statement does not mean that an approved drug is risk free. Instead, the statement indicates that FDA, based on its review of scientific evidence, has determined that the benefits of a drug outweigh its risks when used as directed.

As “scientific evidence” is the determining factor for drug approvals, Parks posed three questions to highlight the nuances inherent in the term:

1. How is scientific evidence defined?
2. Is all scientific evidence created equal?
3. When different people look at the same scientific evidence, will they come to the same conclusion?

__________________

¹ This section is based on the presentation by Baruch Fischhoff, Howard Heinz University Professor, Department of Social and Decision Sciences, Department of Engineering and Public Policy, Carnegie Mellon University.

² This subsection is based on the presentation by Mary H. Parks, Deputy Director, Office of Drug Evaluation II, Office of New Drugs, CDER, FDA.

Defining Scientific Evidence

Parks noted that what FDA can treat as scientific evidence is defined by federal statute. Section 505(d) of the federal Food, Drug, and Cosmetic Act defines substantial evidence as:

evidence consisting of adequate and well-controlled investigations, including clinical investigations, by experts qualified by scientific training and experience to evaluate the effectiveness of the drug involved, on the basis of which it could fairly and responsibly be concluded by such experts that the drug will have the effect it purports or is represented to have under the conditions of use prescribed, recommended, or suggested in the labeling or proposed labeling thereof.

Parks emphasized that this definition applies only to evidence that is used to determine a drug’s effectiveness. No legal standard currently exists to define “substantial evidence” for safety determinations.

Not All Scientific Evidence Is Created Equal

A ranking system for scientific evidence has been in use for many years, based on the rigor of the methods used to produce it (see Table 3-1).

TABLE 3-1 Hierarchy of Scientific Evidence

NOTES: Category Ib, scientific evidence from at least one randomized clinical trial (RCT), is used in drug trials. Workshop participants also discussed opportunities to gather important benefit and harm information from prospectively planned RCTs for regulatory approval as well as observational studies conducted after a product is used widely by patients in the real world.
SOURCE: Parks, 2014. Presentation at the IOM workshop series on Characterizing and Communicating Uncertainty in the Assessment of Benefits and Risks of Pharmaceutical Products.

A drug trial collects evidence to support efficacy, which when done through an RCT, yields the highest level of evidence. Sponsors prospectively design RCTs with a specific efficacy endpoint in mind, often with input from FDA scientists.

In contrast, safety data rarely come from prospectively designed, controlled trials. Indeed, obtaining the “signal of concern” about an adverse outcome is one of the most difficult issues that drug regulators face. The RCT does not serve as the same gold standard in safety as in efficacy. Most RCTs are not designed to prospectively look for safety concerns and, as a result, any safety data collected are likely to be insufficient.

More often, as noted by Parks and other workshop participants, potential harms or adverse events are generally found in postmarket, observational study data, or via other methods that are considered less rigorous than RCTs, but that can reveal important safety information that is more relevant to patients outside the restricted RCT population.

Same Evidence, Different Reviewers, Different Conclusions

Drug reviewers must analyze many lines of data that might vary in quality, noted Parks. As a result, they often adopt different interpretations of the reviewed evidence.

Parks indicated that difficult drug applications, where there are substantive disagreements among reviewers about safety and risk, are not simply passed along for a signatory authority to review and decide. Before a final decision is made, internal and external risk communication procedures, including the agency’s safety committees, ensure that offices throughout the agency are involved in helping to resolve the issues. Box 3-1 lists FDA’s procedures that can be deployed in the drug review context to inform risk communication.

COMMUNICATING WITH THE PUBLIC ABOUT BENEFIT, HARM, AND UNCERTAINTY³

Robyn Lim, Senior Science Advisor, Office of Legislative and Regulatory Modernization, Health Products and Food Branch, Health Canada, presented Health Canada’s Benefit–Harm–Uncertainty Initiative (BHU), a new regulatory approach focused on communication and patients. BHU is directing increasing attention to patients in order to help them understand the role of uncertainty in the real-world decisions they need to make about

__________________

³ This section is based on the presentation by Robyn Lim, Senior Science Advisor, Office of Legislative and Regulatory Modernization, Health Products and Food Branch, Health Canada.

treatment options. Health Canada is modernizing its regulatory approach to drug approval by “hard-wiring” communication about uncertainty into federal legislation. The new system will include mandated transparency about the benefit, harm, and uncertainty considerations in regulatory decisions with the goal of aligning regulatory decisions and accompanying communication strategies to better serve the needs of health partners, particularly patients.

The guiding principle behind BHU, according to Lim, is that pairing the terms “benefit” and “risk” is inherently confusing because there is an asymmetrical acknowledgment of uncertainty in the terms, and that confusion undermines patients’ ability to think about choices and trade-offs in a disciplined way. The definition of benefit does not include the concept of uncertainty, Lim said, while risk has a wide ranging set of definitions. Definitions of risk include the possibility that something bad or unpleasant can happen, or a “hazard,” with the possibility of both chance and uncertainty.

As a result, Lim said, tolerance for risk is usually interpreted as tolerance for a harm or hazardous outcome. However, she noted, tolerance for risk is not the same as tolerance for uncertainty. Tolerance for risk is accepting the possibility of harm, while tolerance for uncertainty includes the acceptance of a much broader scope of possible outcomes—that, for better or worse, uncertainty can exist within stated benefits as well as potential hazards, and other outcomes as well.

During fall 2014, Health Canada is finishing the final draft of its Patient’s Decision Guide About Treatments, a structured, step-wise process

that provides patients with a simple, methodical way to think about positive effects, negative effects, and uncertainties of treatments, so they can make better informed decisions.

FDA Patient-Focused Drug Development Initiative⁴

Patients are uniquely positioned to inform FDA’s understanding of the clinical context for a disease or condition. Having an understanding of what matters most to patients, including what they want and need to know, helps inform the agency’s regulatory decisions and outreach that meaningfully communicate uncertain issues in a manner most relevant to patients, noted Theresa Mullin, Director, Office of Strategic Programs, CDER, FDA. While FDA has a number of mechanisms to obtain patient input,⁵ it is typically limited to discussions related to issues associated with a particular product or class of products. The Patient-Focused Drug Development (PFDD) initiative⁶ is different because it is more broadly focused and systematic, and does not focus on a single product.

PFDD operates by providing patients with an in-person forum for discussion about the impact of a specific condition or disease on their daily lives, and the treatment options that are available to them. According to Mullin, the PFDD initiative was the result of agency thinking about limitations in the information and the sources of information available for consideration in a drug review.

For a drug to be approved for marketing, FDA must determine that the drug is effective and that its benefits outweigh its risks. The agency wanted to undertake a more systematic approach to gathering patient input to inform this benefit–risk assessment, particularly the therapeutic context concerning the severity of the disease condition and the degree to which current therapies are meeting patients’ needs (see Figure 1-1 and remarks by Frey). PFDD gives the agency the opportunity to hear from patients directly and gather important input on what it is like to live with the disease, their experience with current treatment options, and what they look for in an ideal treatment. Although uncertainty is not the main focus for PFDD meetings, Mullin noted that patients often must

__________________

⁴ This subsection is based on the presentation by Theresa Mullin, Director, Office of Strategic Programs, CDER, FDA.

⁵ FDA patient engagement tools include FDA Patient Representative Program; Patient Consultant Program; Patient Liaison and Patient Network programs; and CDER’s Professional Affairs and Stakeholder Engagement.

⁶ For more information about Patient-Focused Drug Development (PFDD) and the at least 20 disease-focused meetings FDA has committed to conducting in 2013–2017, as a result of PDUFA V, see http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm347317.htm (accessed August 8, 2014).

consider scientific uncertainty, for example, when considering whether to take a treatment or participate in a clinical trial. PFDD meetings present an opportunity for agency reviewers to engage patients in discussion about the factors that weigh into these decisions and what information they want to know about benefits, risks, and uncertainties (see Box 3-2).

According to Mullin, input from patients to date has been very useful for FDA reviewers, who will conduct future benefit–risk assessments for drugs to treat specific diseases. Patient discussions and conversations strengthen the agency’s understanding of the relative importance of clinical outcomes, and what types of risks might be considered acceptable to patients. Patient input could also support drug development more broadly, for instance, by identifying specific symptoms, such as fatigue, that are not being remedied by current treatment options. The formation of an interested and informed patient community could also support the

identification and development of a process to collect patient-reported outcome⁷ (PRO) measures.

Mullin also elaborated on ways that patient communities not represented in the original 20 disease-focused meetings scheduled through 2017 might conduct their own similar meetings to inform the agency of what matters most to patients with a particular disease or condition. The PFDD meetings currently organized by FDA follow a consistent format and all meeting materials, as well as the final summary reports, are publicly available. Mullin noted that hosting meetings in the Washington area, and generally in close proximity to Silver Spring or Bethesda, would facilitate attendance by FDA staff.

To provide context for workshop discussions surrounding regulatory decision making under uncertainty, FDA developed two case studies to illustrate the types of uncertainties that FDA reviewers face in weighing the evidence for a particular product to receive market approval (see Box 3-3 and Appendix B for complete case studies).

__________________

⁷ A patient-reported outcome is any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else. Additional information is available here: http://www.fda.gov/downloads/Drugs/Guidances/UCM193282.pdf (accessed September 8, 2014).

This page intentionally left blank.