Greg Simon, an investigator with Group Health Research Institute and workshop co-chair, laid out the basic premise for the workshop: that our current system of generating evidence is not meeting our needs. When new treatments are released into the real world, he said, we often lack the information we need to make real-world decisions. Randomized controlled trials (RCTs) have been the gold standard for answering questions of safety and efficacy, but they have not adequately addressed the fundamental questions of how well a new treatment works, particularly in comparison to other options, or when and for whom it should be recommended. Although this realization has not yet led to significant changes in the pipeline for the production of new therapeutics, a number of advances that are underway offer opportunities to reshape the current evidence generation paradigm.
THE CURRENT LANDSCAPE FOR EVIDENCE GENERATION PROCESSES
FDA Commissioner (at the time of the workshop) Robert Califf, who was also the workshop keynote speaker, described the landscape in which medical product development is occurring as one undergoing rapid and profound changes. A technology revolution is happening that, while exciting in terms of its potential, will also increase demands on the current system. Novel techniques such as gene editing raise questions regarding safety and effectiveness that will need to be addressed, he said. Additionally, devices have proliferated that enable consumers to continuously monitor and col-
lect health-related data. Califf emphasized that, if the full potential of these technology advances is to be realized, data sharing is important.
Califf also acknowledged the opportunities arising as a result of health care reform efforts, which are driving a shift from traditional fee-for-service to a value-based reimbursement model. A desire to achieve better value in health care has incentivized movement toward what the National Academies has called a learning health system,1 in which large amounts of electronic health data can be shared rapidly across systems and rapid cycle improvement can be achieved by understanding what works, then measuring impact following implementation. The digital capture, aggregation, and analysis of health care data with the goal of improving quality of care and cost-effectiveness represents a fundamental change in evidence generation processes, with significant implications for medical product development. Califf stressed that with this new model for knowledge generation, there would be less need for a completely separate, and consequently inefficient, clinical research infrastructure. The integration of research with clinical care and use of existing data has the potential, therefore, to drastically reduce the cost of evidence generation.
Overall, Califf concluded, leveraging available data sources is more widespread and commonplace than ever. Yet, he said, “the current system is not delivering adequate evidence in the face of an explosion of new medical products and increased understanding of how to evaluate products already in clinical use,” and many clinical treatment decisions are not supported by evidence (Tricoci et al., 2009; Han et al., 2015). Now is the time, he said, to build on the foundation of recent advances and take them to the next level (see Box 2-1).2
STAKEHOLDER PERSPECTIVES ON PRIORITIES FOR IMPROVING REAL-WORLD DECISION MAKING
During the first workshop session, Califf’s keynote was accompanied by a diverse panel of stakeholders who provided remarks about the most pressing challenges and priorities for improving evidence generation to support real-world decision making. These individual speakers noted that the traditional processes for evaluating new therapeutics focus too narrowly on efficacy and safety outcomes and do not adequately address key questions regarding effectiveness, tolerability, and value—questions that matter to clinicians and patients. As a result, these speakers noted, the traditional pathway for medical product development does not produce the evi-
___________________
1 For more information, see IOM, 2013a.
2 For more information, see Sherman et al., 2016.
dence needed to inform real-world clinical, regulatory, and reimbursement decisions.
Industry
Patrick Vallance, president, Pharmaceuticals Research and Development, GlaxoSmithKline, observed that the medical product development industry is actually becoming less efficient. The cost of clinical trials is increasing sharply (Berndt and Cockburn, 2014) and the failure rate of the clinical research enterprise is profound. As Califf noted, more than 90 percent of drugs that enter Phase I trials do not make it to market because of issues related to effectiveness, toxicity, or reliable production. Vallance remarked that the incorporation of real-world data into the evaluation of therapeutics has the potential not just to improve the efficiency of clinical trials, but to actually answer different questions that, in some cases, can only be answered with real-world evidence. As one example, Vallance cited that real-world evidence is particularly suited to answering questions not about safety and efficacy, but instead about how well a particular treatment
works in comparison to other possible treatments (i.e., comparative effectiveness). He also stated that questions about determining compatibility, dosage, and usage indications for combination treatments, and increasingly even for combinations during development rather than postmarketing studies, can be assessed with real-world evidence. Although postmarket evaluations have been moving in this direction, he said, evidence generation processes earlier in the product development life cycle must be able to account for the complexities seen in real-world populations. He cited an example in the increasing number of medications that will need to be tested in combination during the development phase. What is really going to change the way that industry thinks about clinical trials going forward, said Vallance, is the opportunity to incorporate the vast amount of information from electronic health records and devices that continuously collect data directly from patients and consumers outside the clinical setting. Consequently, it will be important to focus more on ensuring that such data are reliable.
Regulators
Addressing the escalating costs of bringing new treatments to market, Califf emphasized the need for a drug development system that winnows out failures as quickly as possible and, for the promising candidates, ensures that the right clinical trials are undertaken to inform decision making by all stakeholders. While maintaining that very early clinical trials will still need to be conducted in highly controlled environments so that the safety, pharmacology, and systems biology of new treatments can be carefully assessed, Califf stressed the later stage research can be integrated within the real world of clinical practice, the better the system will be at yielding results that give doctors and patients the information they need to understand what treatment options are best for them. “Sticking to the old model is a recipe for an escalating cost (of research) at a time when we need more efficient research because the questions are far outnumbering our ability to answer them,” he said.
Califf highlighted common misperceptions that FDA has encountered in discussions on the use of real-world evidence for regulatory decision making, which the agency is working to correct in its communications with stakeholders. First, he said, the source of the data should not be confused with the design of the study. A common assumption is the generation of real-world data is synonymous with observational study design; however, randomization in the context of the real world is both possible and critical. The system will not change until that distinction is broadly understood, echoed Rachel Sherman, Deputy Commissioner for Medical Products and Tobacco, FDA. Second, both Califf and Sherman emphasized that FDA’s
role does not end after approval. For example, FDA is charged with writing labels that provide accurate, instructive information on how to use approved products safely and effectively in medical practice. Continued evidence generation on effectiveness in the postmarket phase can inform FDA labeling changes as well as new indications. Indeed, Sherman noted, the demarcation between pre- and postmarket represents an outdated way of thinking about the drug development paradigm. Real-world evidence will increasingly be embedded across the process of drug discovery and development and through to the market, informing regulatory decisions across all of those phases. Finally, Califf sought to dispel the misperception that FDA regulations or guidance prohibit or inhibit use of real-world evidence for regulatory approval of new treatments.3 Real-world evidence, when considered appropriate in the views of competent experts in the field, could legitimately contribute to the legally required demonstration of substantial evidence of effectiveness of new treatments.4 As the evidence system changes, Califf said, a process will be needed for driving agreement on what constitutes substantial evidence and quality for different purposes.
Payers
When new treatments are approved, health care payers—including the Centers for Medicare & Medicaid Services (CMS), private insurers, and, increasingly, providers who participate in shared savings and capitation arrangements—base coverage determinations on their value, which is calculated by examining the net costs and evidence of benefit. While this can create tension between payers and industry, Rhonda Robinson Beale, chief medical officer, Blue Cross of Idaho, underscored the fact that when
___________________
3 The 21st Century Cures Act, signed into law on December 13, 2016, requires FDA to evaluate the use of real-world evidence to help support the approval of a new indication for a previously approved drug and to help support or satisfy postapproval study requirements. Under the direction of the Secretary of Health and Human Services, a guiding framework will be developed to implement a program within FDA that details the circumstances, standards, and methodologies for which real-world evidence can be used in medical product evaluation. This program will then guide the development of draft guidance for industry to be released for public comment, and final guidance on the use of real-world evidence for medical product evaluation by FDA (21st Century Cures Act, Public Law 114-255, 114th Cong., 2d sess. [December 13, 2016]).
4 Substantial evidence is defined in the Federal Food, Drug, and Cosmetic Act (21 U.S.C. § 355(d)) as “evidence consisting of adequate and well-controlled investigations, including clinical investigations, by experts qualified by scientific training and experience to evaluate the effectiveness of the drug involved, on the basis of which it could fairly and responsibly be concluded by such experts that the drug will have the effect it purports or is represented to have under the conditions of use prescribed, recommended, or suggested in the labeling or proposed labeling thereof.”
there is a limited pool of resources, overspending on high-cost treatments shifts the financial balance, noting that high spending on some patients could result in inadequate resources being available for the remainder of the population being serviced. In this context, payers must make coverage decisions based on limited evidence because traditional clinical trials are not designed to answer questions regarding the comparative value. Joseph Chin, deputy director, Coverage and Analysis Group, CMS, pointed out that some types of patients, such as Medicare beneficiaries, are often excluded from clinical trials, making it challenging to make coverage determinations for those populations. Chin noted that the incorporation of real-world data into evidence generation processes could assist CMS coverage determinations by rendering clinical research results more immediately translatable to the beneficiary population, both by incorporating data from a more general population than typically seen in clinical trials and by potentially creating the opportunity to apply the results obtained during approval at FDA without further need to request additional studies on efficacy for CMS. This could also motivate CMS to work with FDA on harmonizing evidence requirements.
Robinson Beale said payers see a lot of off-label use and experimentation that have little practical evidence or recognized guidelines demonstrating treatment effectiveness in clinical practice for disease areas with high mortality or morbidity. She suggested that these disease areas could be prioritized. When high-cost treatments are involved, the trial-and-error methodology often used by providers who must apply those treatments to real-life populations that do not completely match the clinical trial population is simply unaffordable and drives up health care costs, she said. She noted that this expense is affecting patients in particular, with current insurance premiums closer in cost to a house payment than a car payment.
Health Care Providers
John Carroll, professor of medicine and co-medical director of the Cardiac and Vascular Center, University of Colorado Hospital, described the completion of RCTs not as an endpoint, but as the start of a new phase of learning. Health care providers then determine how best to apply the results from RCTs, with heterogeneity of treatment effect, to individual patients, who will have different characteristics and preferences. The individualizing of care calls for experience, skills, and judgment gathered over years of practicing medicine. Carroll also pointed out that regulatory approvals are often narrowly focused, but medical products can be used by medical providers on label, near label, or off label to address patients’ needs. He emphasized the importance of learning from all of these different uses.
The process of gathering evidence to improve decision making about treatment options by clinicians and patients is limited by the slow and anecdotal process of experience acquisition from relatively small numbers of patients. Tools such as registries that enable learning from tens of thousands of patients receiving the same treatment throughout the United States have the potential to significantly accelerate knowledge generation. Tools to help health care providers and patients translate knowledge into actionable information were noted as a gap by a number of workshop participants. Even when evidence to support clinical treatment decisions exists, that evidence may not be reaching frontline providers. As a result, many patients are receiving the wrong treatment, said Robinson Beale, who characterized the existing tools providers have to support decision making, such as written clinical guidelines, as antiquated. To close the translation gap, clinical decision support tools could be embedded in the process of care to support decision making in real time. The availability of such tools could drive a shift toward more evidence-based decision making in health care. However, Carroll said, it will be important to understand their validation and what can be expected from them in terms of capturing the key elements that go into decision making.
Carroll suggested that priority focus areas to facilitate such learning health systems include the improvement of data quality and reduction of the magnitude of effort and cost currently required to gather data and translate them into actionable information. Several individual panelists agreed with Carroll that engaging health care providers in prospective research activities will add a significant burden in addition to their clinical responsibilities and thus will present a challenge to embedding research in the clinical care infrastructure.
Patients and Consumers
Naftali Zvi Frankel, a patient and consumer advocate, recognized that methodologies that generate real-world evidence are not a replacement for traditional clinical trials, but can instead be viewed as supplementing them. Such real-world studies offer new opportunities for patients with co-morbidities who are often excluded from traditional clinical trials. He illustrated this with a quote from a collaborating clinical trial investigator: “The requirement to have a clean cohort of patients in clinical trials creates a reality where drugs are tested on a universe of patients that does not reflect patients commonly seen.” He stressed that engagement with patients and consumers needs to be reciprocal. Patients and consumers can be partners in the evidence generation process by sharing their data with providers and investigators, and greater efforts are needed to give information back to them as well. Patients too often feel isolated when faced with
choosing among therapies with little awareness of available data, such as comparative effectiveness data, that could inform their treatment decisions. Frankel said a greater effort is needed to improve transparency and patient engagement throughout the product development life cycle.
POTENTIAL CROSSCUTTING PRIORITIES FOR IMPROVING REAL-WORLD DECISION MAKING
Several crosscutting priorities emerged from the Session I discussion.
Communicating Leadership Support for New Approaches to Evidence Generation
Resistance to change was noted by Vallance as a major barrier to more systematic incorporation of real-world data into evidence generation processes. Given the significant costs associated with moving new treatments through the drug discovery and development process, the adoption of new methods for evidence generation represents a real risk. Reluctance to diverge from what has worked in the past has slowed change efforts in industry and on the regulatory side. Vallance suggested that although there may be support at the top of organizations for use of real-world evidence when it is appropriate to answer a specific question, that support may not be fully communicated throughout organizations. As a result, there is a sustained misperception that real-world evidence is not acceptable to support regulatory decision making, and, he remarked, leadership intervention within industry and regulatory agencies will be needed to encourage risk taking. Califf agreed that efforts to improve communication within and outside FDA could give companies more confidence about incorporating real-world data sources and pursuing alternative endpoints.
Harmonizing Evidence Generation Processes Across Stakeholder Groups
There is a great deal of interest across the biopharmaceutical industry, regulatory agencies, and payers in harmonizing evidence generation processes as a means of improving the efficiency of the drug development process and simultaneously generating the kinds of evidence needed to support real-world decision making. Although different stakeholders might use different criteria for decision making, it could be possible for them to use the same source of evidence. For example, explained Califf, FDA and CMS might use different criteria in determining whether a treatment should be approved and whether it should be covered, respectively, but thoughtfully designed studies that yield data on effectiveness and resource usage could inform both sets of decisions, reducing the number of studies that
industry needs to undertake. This is a goal that FDA and CMS are actively pursuing. However, Sean Tunis, founder, president, and chief executive officer, Center for Medical Technology Policy, said that there are a number of other decision makers (e.g., private payers, formulary committees, guideline developers, health technology assessment organizations) who will be assessing quality and relevance of evidence in the postmarket context and therefore should be engaged on the front end of discussions on evidence generation processes.
Addressing Privacy and Confidentiality Concerns
Embedding research into the clinical care infrastructure depends on the ability to share, aggregate, and analyze patient data. But in the current cybersecurity environment, it is not possible to absolutely guarantee the security of those data, cautioned Califf, who advocated instead for a participatory environment that is endorsed by patients and consumers and includes robust procedures for ensuring data security and protecting confidentiality. These concerns could discourage patients from permitting use of their data for secondary purposes such as clinical research. There is also a common, yet unfounded, fear among patients that health-related data could be used against them, for example, in life insurance coverage decisions, explained Robinson Beale. She stressed that it would help for those collecting the data to fully explain to patients and consumers the kinds of safeguards that are in place to minimize risks, such as de-identification methods. Sherman added that stakeholders need to think more creatively about ways to protect data other than keeping it sequestered. Frankel was optimistic about the willingness of patients and consumers to share their data despite the risks, but again stressed the importance of transparency and receiving clear consent from patients, so they understand how the data may be used and the benefits and risks of those uses.
