6
Final Reflections on
Ways to Characterize and
Communicate Uncertainty
Key Messages Identified by Individual Speakers
- A research agenda could address questions and focus energy and resources on how best to clinically, methodologically, and statistically reduce uncertainty. Such an agenda could include identifying a scientifically acceptable method to bridge the gap between randomized trials and observational studies.
- A taxonomy of uncertainties could bring structure to thinking about the sources, timing, and impact of uncertainties.
- Social science and qualitative methods to better understand the experience of patients and their medicines could provide a unique opportunity to enrich the context for the regulatory decision maker.
- Patients and providers could benefit from receiving quantitative information on benefits and risks of pharmaceutical products, as well as clear statements about inherent uncertainties.
As part of the final session of the May 12, 2014, workshop, several speakers and workshop participants reflected on what they had heard over the course of the workshops. Their ideas are gathered in this final chapter as a way to highlight and elaborate on potential next steps for improving the characterization and communication of uncertainty in benefit–risk assessments of pharmaceutical products.
IDENTIFYING AND MITIGATING UNCERTAINTY THROUGH MAXIMIZING THE VALUE OF EVIDENCE
Many FDA workshop participants noted that federal laws and regulations provide the boundaries for FDA’s decisions about drug approvals and that, by statute, FDA can tolerate less uncertainty about efficacy than about safety or harms. Drawing conclusions in the regulatory setting, in the midst of uncertainty, is a challenging task that might be made more efficient and transparent with the implementation of systematic approaches to dealing with uncertainty.
There is a need for a research agenda to address questions and focus energy and resources on how best to clinically, methodologically, and statistically reduce uncertainty, noted Paul J. Seligman, Executive Director, U.S. Regulatory Policy, Amgen Inc. Such a research agenda could include finding a scientifically acceptable method to bridge the gap between randomized trials, which are focused on proving drug efficacy in a study population, and observational studies, which focus on risk and adverse events in the real world. Seligman referenced the presentation by Sebastian Schneeweiss of Harvard Medical School (see Figure 2-1) and the opportunity to structure a clinical trial portfolio to use multiple studies with different designs and data sources to reduce uncertainty. Multiple studies could be optimally arranged to reduce chance, better characterize representativeness, and reduce bias with the hope that the resulting information would be timely, valid, and comprehensive for decision makers. Taking a structured approach to designing multiple clinical trials could also be a more efficient use of resources. Statistical techniques such as sensitivity analyses can also help to make better use of the data collected and further characterize uncertainty. Several workshop participants also referenced the work of the Innovative Medicines Initiative (IMI) Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium (PROTECT) to study and address limitations of current methods in the fields of pharmacoepidemiology and pharmacovigilance.1
Several workshop participants also highlighted “low-hanging fruit” opportunities to improve the value of evidence generated from clinical trials to reduce uncertainty. In summarizing the presentations and discussions from the February 12 workshop, Seligman indicated that maximizing the utility of clinical trial data could include ensuring all clinical trials and studies are registered on ClinicalTrials.gov, consistently naming the products being studied in government databases and the scientific literature so
__________________
1 For more information on Innovative Medicines Initiative (IMI) Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium (PROTECT), see http://www.imi-protect.eu (accessed September 18, 2014).
that all studies for a particular product can be easily identified and used, and working to keep patients in trials and following up with those who leave a trial to increase retention and reduce the uncertainty of a trial’s results.
Ralph I. Horwitz, Senior Vice President, Clinical Sciences Evaluation, GlaxoSmithKline, suggested expanding the scope of information used in regulatory decisions to include not just the traditional placebo-controlled efficacy trials, but pragmatic trials that provide richer information about how a drug might actually be used in clinical settings. John Jenkins, Director, Office of New Drugs, CDER, FDA, indicated that FDA supports the elimination of unnecessary exclusion criteria in clinical trials to bring the information closer to what patients would experience in the real world.
CHARACTERIZING AND UNDERSTANDING UNCERTAINTIES
A clear taxonomy of uncertainties that describes the nature of information gaps, their effects, and the steps that need to be taken to address those information gaps could improve the characterization and understanding of uncertainties, noted Seligman. Such a taxonomy could also bring structure to thinking about the sources, timing, and impact of uncertainties.
Bayesian approaches to evaluating clinical trial data have the potential for more robust characterization of inferences drawn from studies, said Seligman. A disciplined Bayesian approach offers opportunities to characterize and accommodate uncertainty in clinical trials. Bennett Levitan, Director of Quantitative Safety Research, Department of Epidemiology, Johnson & Johnson, noted that additional research is needed on the potential of Bayesian approaches to combine data from multiple clinical trials (e.g., an RCT and observational study if the necessary variables are contained in each). Levitan further explained that Bayesian methods might allow different stakeholders to embed their preferences in the analysis of trials by virtue of having a prior distribution from one group versus another. To move from idea to practice, Levitan suggested that FDA could help spur research from industry and academia on Bayesian methods to better characterize uncertainty in evidence from clinical trials.
ELICITING VALUES FROM STAKEHOLDERS, PARTICULARLY PATIENTS
Horwitz suggested there is a need to formalize the personal experience of patients with medicines in the pre- and postapproval periods. He suggested including social science and qualitative methods to better understand the experience of patients and their medicines to provide context for the regulator.
With the support of FDA, PRO instruments are increasingly being included in clinical trials to measure the effect of a medical intervention on one or more concepts (e.g., a symptom or group of symptoms). Some workshop participants suggested that there could be an important role for patient groups to develop PRO instruments outside of the regulatory process for a particular product and ideally before a new drug is contemplated for development. PFDD meetings could serve as one platform for patient groups to gather the information to support the development of a PRO instrument. In addition, one workshop participant suggested that if multiple pharmaceutical companies had an interest in discovery and development activities for a particular disease or condition, they could pool funding to have an independent patient group develop the PRO instrument to be shared by all.
Some workshop participants also explained a “risk-risk” concept for patients when considering whether or not to take a drug. Benefit is traditionally understood as an additional advantage or bonus, but the “benefit” of taking a drug could actually be understood as avoiding the harms and adverse experiences of a disease. Thus, patients weigh the risks, and inherent uncertainties, of experiencing adverse effects from their disease versus the risks, and inherent uncertainties of experiencing adverse effects of the treatment.
Schneeweiss suggested developing a metric to compare evidence on benefits and harms that incorporates patient preferences. Such a metric would ideally be informed by greater reporting of actual risk differences and relative risk. Schneeweiss explained that this risk information is retrievable from studies submitted to FDA for regulatory approval, but should be presented in a more accessible way. This information, coupled with patient preferences, could inform the development of a metric to help weigh benefits and harms.
FDA might also consider novel ways to elicit expert advice. Seligman noted that he knows of no other federal agency that seeks as much expert advice as FDA through the public advisory committee function. The science of expert elicitation could offer new opportunities for FDA to develop systematic ways of gathering expert input (i.e., methods that differ from going around the table to solicit advice during a public advisory committee meeting).
COMMUNICATING UNCERTAINTY ABOUT BENEFIT AND RISK ASSESSMENTS OF PHARMACEUTICAL PRODUCTS
In addition to the importance of identifying and communicating sources of uncertainty in clinical data, Robert Temple of FDA noted that FDA could do better in presenting the important data already on hand.
For instance, product labels currently do not, but could, include information about the effect of a treatment on subgroups (e.g., treatment effect by age, race, and sex) so that patients can gain a better understanding of how the treatment might work for them based on certain characteristics. FDA has this subgroup information, but usually does not include it in the product label unless it contains a striking or unusual finding. Temple also suggested that treatment effect data by subgroup should not be limited to studies of treatments for conditions that lend themselves more easily to hard outcomes (e.g., CV disease and mortality), but should also be provided for conditions such as depression.
A number of workshop participants noted the limitations of language in its ability to communicate accurate information. David R. Mandel of DRDC presented a corrective measure in the form of a prohibition on “weasel words” and phrases, such as “reportedly,” “evidence suggests (or indicates),” “distinctly possible,” and “apparently.” Such uncertain words are not well suited to the complex task of communicating uncertainty about benefit–risk assessments in pharmaceutical products.
Some workshop participants discussed the idea that conveying information numerically can provide greater clarity, but also presents its own challenges. For instance, Temple stated that providing the mean result for a depression score is not very informative because individuals experience a range of treatment effects varying widely from the average result. According to Temple, FDA is increasing its reporting of the cumulative distribution of results, such as the number of people who experience a 10, 20, 30, or 40 percent improvement in their condition as a result of the treatment. Reporting this variability enriches the communication about a drug and helps patients better understand the likelihood that they will benefit from a particular drug.
Lisa M. Schwartz, Dartmouth Medical School and Center for Medicine and the Media at The Dartmouth Institute for Health Policy and Clinical Practice, and Steven Woloshin, Dartmouth Medical School and Center for Medicine and the Media at The Dartmouth Institute for Health Policy and Clinical Practice, suggested FDA adopt a quantitative format for conveying benefit and risk information that includes the base rate. For example, compared to placebo, the drug reduced the relapse rate from 7.4 relapses per 100 people per year to 2.5 relapses per 100 people per year. This quantification gives readers a better understanding of the drug’s benefits than does simply providing a relative risk reduction (e.g., the drug reduced the frequency of relapses by 66 percent relative to placebo), which can exaggerate the perceived benefits. Levitan suggested FDA consider including an effects table that contains benefit and harm information in the same unit of measurement. Including this comparison in a prominent section of the product label could improve interpretability
of benefit–harm information and patient and physician decision making, noted Levitan. The EMA is piloting the use of an effects table as a tool to summarize key benefits and risks in the review process.2
FDA has a number of communication tools currently in its arsenal for conveying information to a broad range of stakeholders. Patrick J. Frey of FDA indicated that as part of FDA’s implementation of the benefit–risk framework, the agency can consider opportunities to improve how it currently communicates information about benefit, risk, and uncertainty when posting review documents on the FDA website. As the new framework is implemented, the agency will have the chance to further optimize communications about the rationale behind regulatory decisions.
__________________
2 For more information on the EMA’s Benefit–Risk Methodology Project, see http://www.ema.europa.eu/docs/en_GB/document_library/Report/2011/07/WC500109477.pdf (accessed September 18, 2014).
