1

Introduction¹

Randomized controlled trials (RCTs) have traditionally served as the gold standard for generating evidence about medical interventions. However, RCTs have inherent limitations and may not reflect the use of medical products in the real world (e.g., specific therapeutic interventions may perform differently within different patient cohorts based on age, gender, race, ethnicity, disease severity, comorbidities, or polypharmacy). Additionally, RCTs are expensive, time consuming, and cannot answer all questions about a product or intervention. Evidence generated from real-world use—based on sources such as patient registries, electronic health records (EHRs), and medical claims data—may provide valuable information, alongside RCTs, to inform medical product decision making. This supplemental (or complementary) information is generally based on analysis of information gathered in the “real world” of routine clinical practice outside of a tightly controlled RCT. As standards and rigor for the collection and analysis of real-world data (RWD) evolve and improve over time, stakeholders will have the opportunity to explore new areas for which RWD and real-world evidence (RWE) may be used to answer scientific questions and guide more effective and cost-efficient medical product decision making.

___________________

¹ These workshops were organized by an independent planning committee whose role was limited to planning the workshops, and the Proceedings of a Workshop Series was prepared by the workshop rapporteurs and staff as a factual summary of what occurred at the workshops. Statements, recommendations, and opinions expressed are those of individual presenters and participants, and are not necessarily endorsed or verified by the National Academies of Sciences, Engineering, and Medicine, and they should not be construed as reflecting any group consensus.

Regulatory agencies and medical product sponsors are increasingly interested in incorporating RWE into their programs. For example, the U.S. Food and Drug Administration (FDA) has committed, in both the Prescription Drug User Fee Act VI (for drugs and biologics) and the Medical Device User Fee Amendments IV (for devices), to developing policies that support the use of RWE in medical product evaluation (see remarks by Mark McClellan, director of the Duke-Margolis Center for Health Policy, in Chapter 1 and by Rachael Fleurence, executive director of the National Evaluation System for health Technology [NEST] Coordinating Center in Chapter 3). Medical product developers are already successfully using data sources such as registries and techniques such as historical comparator arms to earn marketing authorization (see remarks by Fleurence in Chapter 3; Scott Gottlieb, commissioner of FDA, in Chapter 1; Janet Woodcock, director of FDA’s Center for Drug Evaluation and Research [CDER], in Chapter 5; Steven Anderson, director of the Office of Biostatistics and Epidemiology at FDA’s Center for Biologics Evaluation and Research, in Chapter 10; Jacqueline Corrigan-Curay, director of the Office of Medical Policy at CDER, in Chapter 10; Jeff Shuren, director of FDA’s Center for Devices and Radiological Health, in Chapter 10; and speakers throughout). Furthermore, FDA has supported projects that are designed to incorporate RWE into its decision-making and safety monitoring processes, such as Sentinel (see remarks by Richard Platt, professor and chair in the Harvard Medical School Department of Population Medicine at the Harvard Pilgrim Health Care Institute, in Chapter 3). Moreover, the recently enacted 21st Century Cures Act mandates that FDA develop a framework for using RWE in support of new indications and to satisfy postapproval studies. Europe is likewise engaged in developing policies and infrastructure to support additional use of RWE. The European Medicines Agency is developing the EudraVigilance network for safety monitoring and reporting and the Adaptive Licensing pathway for product marketing authorizations (see remarks by Alasdair Breckenridge, emeritus professor of Clinical Pharmacology at the University of Liverpool, in Chapter 10). The European Union and the regulated industry are supporting several RWE initiatives through the Innovative Medicines Initiative (see remarks by Breckenridge and Pall Jonsson, associate director of Research and Development at the United Kingdom’s National Institute for Health and Care Excellence [NICE], in Chapter 10).

To explore the potential for using RWE in medical product decision making, an ad hoc workshop planning committee of the National Academies of Sciences, Engineering, and Medicine planned a three-part workshop series, sponsored by FDA and hosted by the National Academies’ Forum on Drug Discovery, Development, and Translation (see Box 1-1 for the Statement of Task). The series was designed to examine the current system of

evidence generation and its limitations, to identify when and why RWE may be an appropriate type of evidence on which to base decisions, to learn from successful initiatives that have incorporated RWE, and to describe barriers that prevent RWE from being used to its full potential. Issues regarding RWE unrelated to evidence development, use, and application were not discussed in detail during the series.

At the first workshop, held in September 2017, participants heard from stakeholders—patients, providers, and payers—about what kind of data they need to make decisions, and explored how to generate this “fit-for-purpose” evidence. Researchers who have successfully used RWE in their work were invited to share their successes, and workshop speakers and participants discussed the challenges and misaligned incentives that prevent RWE from being used more widely. Finally, speakers presented their perspectives on the shortcomings and limitations of the current system of evidence generation, and discussed how integrating RWE into the system could improve decision making for medical product development and evaluation.

The second and third workshops, held in March and July 2018, focused on illuminating when it may be appropriate to use RWE and which questions RWE may help address, and identifying key questions that stakeholders might consider when collecting or using RWE. These questions were used to draft RWE study design decision aids; the aim of these decision aids was to help stakeholders make thoughtful choices about the development and design of studies involving RWE.

The decision aids (discussed in Chapters 6 through 9) served as a starting point for discussions at the third workshop, and were informed by discussions that took place during the first and second workshops in this series. The decision aids were drafted by some individual participants of the first and second workshops, with additional input by attendees of the third workshop. The decision aids were developed to guide discussion at the third workshop, and they may also be useful in helping workshop attendees and other stakeholders think about and evaluate opportunities to use RWD and RWE for medical product decision making and make informed choices about the design of prospective or retrospective studies, primarily for regulatory review (akin to clinical decision aids that are designed to help patients make informed decisions about treatment options). There are no right or wrong answers to a given question. Instead, the decision aids lay out key questions for stakeholders to consider early on to help make thoughtful choices about the development and design of rigorous, but manageable, RWE studies that relevant parties (e.g., patients, clinicians, researchers, sponsors, regulators, payers) agree in advance will generate reliable results.

The questions in the decision aids aimed at capturing relevant information about the potential risks associated with either the treatments themselves or trade-offs associated with particular decisions; costs in terms of monetary investment, time investment, and/or patient and clinician investment; and reporting and transparency expectations for showing study methods and results. The decision aids can be further modified and refined to be broadly applicable across study types (e.g., studies on medical products to treat prevalent chronic diseases), account for different types of data sources (e.g., EHRs, claims data, patient-generated data), and “future-proofed” to accommodate new sources of data going forward (e.g., data from mobile devices, the Internet of things). The decision aids were divided into four topic areas:

1. When a particular RWD element is fit to assess study eligibility, treatment exposure, or outcomes;
2. Considerations for obscuring intervention allocation in trials to generate RWE;
3. Considerations for controlling or restricting treatment quality in real-world trials; and
4. Assessing and minimizing bias in observational comparisons.

At the third workshop, speakers and participants used the decision aids to explore the four topic areas and used case studies to explore and illuminate the practical, ethical, technological, and scientific issues that arise when designing RWE studies for decision making. Throughout the workshop series, attention was paid to how evidence is used in regulatory decision making, and how RWE may be incorporated into this process.

This workshop proceedings was prepared by rapporteurs in accordance with National Academies guidelines, and is a summary of the discussions held during the three workshops. The proceedings is divided into 10 chapters. Chapter 1 introduces the topic and frames the workshop series; Chapter 2 presents various stakeholder perspectives on RWE; Chapter 3 discusses successful examples of the use of RWE in research; Chapter 4 discusses barriers and disincentives to the use of RWD and RWE; and Chapter 5 considers RWE in the context of the current evidence-generation system. Chapters 6 through 9 focus primarily on in-depth discussions from the second and third workshops. Chapter 6 discusses the use of RWD; Chapter 7 discusses treatment quality in real-world research; Chapter 8 focuses on obscuring intervention allocation; and Chapter 9 discusses observational data research. Chapter 10 concludes the proceedings and describes topics that were discussed throughout the workshop series.

DEFINING THE TERMS

For these proceedings, definitional terms used by other organizations and descriptions of RWD and RWE that emerged over the workshop series are outlined here. Gregory Simon, senior investigator at Kaiser Permanente Washington Health Research Institute, said it is essential to establish a common language to describe RWD and RWE in order to discuss and potentially change the paradigm of evidence generation. However, he said, RWD and RWE are not easy to define, and different institutions use varying definitions (see Box 1-2). First, said Simon, it is important to differentiate between RWD and RWE: RWD is a necessary, but not sufficient, condition to produce RWE. Simon said that while RWE is sometimes thought of as “everything but a randomized trial,” this is not accurate: RWE can sometimes involve randomization, and not all non-RWE is randomized. A better way to think about RWE, said Simon, is to identify its core characteristics:

• RWE is generalizable: The answers available through RWE will generally be true if they are implemented in the future.
• RWE is relevant: It seeks to directly provide information that stakeholders need to make decisions. In other words, RWE is “fit for purpose,” meaning that the evidence is capable of answering a research question, regardless of its source.
• RWE is adaptable: When evidence is generated in the real world, by necessity it incorporates the broad heterogeneity of real patients and real providers.
• RWE is efficient: Evidence can be produced more quickly and at a lower cost than traditional methods.

Mark McClellan, director of the Duke-Margolis Center for Health Policy, offered the definitions of RWD and RWE that were developed through the work of a collaborative agreement between FDA and the Duke-Margolis Center. RWD are “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.” RWE, said McClellan, is “evidence derived from RWD through the application of research methods.” In the specific context of RWE for regulatory applications, RWE can be further defined as “clinical evidence regarding the use and potential benefits or risks of a medical product derived from analysis of RWD.”

McClellan said there is a common misconception that RWD simply means observational data. He stressed that the key characteristic of RWD is not about the research method used, but instead pertains to the provenance of the data. RWD are data that are “part of the routine delivery of care,” such as clinical records or insurance claims. In addition to these clinical

data, RWD also can include patient-generated data that are not part of an encounter with the health system, for example, data from a smartphone. Turning RWD into RWE is not an automatic process, said McClellan; it requires working with the data and applying appropriate research methods to turn the data into quality, useful evidence. Research methods may include randomized trials, prospective or retrospective approaches, observational approaches, and cluster randomization. These methods are used to turn a

deluge of data—RWD—into evidence that is fit for purpose for a specific application, RWE.

Eleanor Perfetto, senior vice president of strategic initiatives at the National Health Council, added that the difficulty in defining RWD and RWE is contributing to confusion in the patient advocacy community. Patient advocates and individual patients are interested in using RWE to make medical decisions and to advocate for regulatory and payment decisions, but the confusion over terms is making it difficult for them to fully understand what RWE is (and is not), and to understand the benefits and limitations of RWE. Rachel Sherman, principal deputy commissioner of FDA, concluded that while using clear and understandable language is important, “The goal is not to define RWD and RWE. The goal is to get better information and to do it in a more sensible way.”

THE U.S. FOOD AND DRUG ADMINISTRATION’S VISION FOR THE FUTURE

Simon opened the first workshop with an ode to Fiddler on the Roof, a 1960s musical about maintaining Jewish traditions in the face of change. Simon noted that in Fiddler on the Roof—and in scientific research—there are some traditions that are vital to the central purpose of the community, while other traditions are merely followed because they always have existed. Discerning which traditions are important and which can be let go, said Simon, is critical for moving scientific progress forward efficiently and effectively. FDA commissioner Scott Gottlieb concurred with this analogy in his keynote address, saying that expanding traditional notions of evidence generation to incorporate the use of RWE could help make the medical product development process more efficient and more cost-effective. In addition, Gottlieb said, RWE could help doctors and patients to be better informed and make better decisions, which will ultimately help achieve better health care outcomes.

Despite these potential advantages, there is uncertainty about the role that RWE should play in making regulatory decisions, said Gottlieb. RWE is already being used for decision making by many stakeholders in the medical community—payers in particular—and it is time to “close the evidence gap between the information we use to make FDA’s decisions” and the information being used by others. As the use of RWE is increasing, so is the rigor with which it is collected and the reliability of the data. One of the scientific research community’s longstanding traditions, said Gottlieb, is the hierarchy of evidence, in which randomized prospective placebo-controlled trials are at the top. Even as tools and technologies for collecting and using other forms of evidence, including RWE, have progressed, the hierarchy of evidence remains unchanged.

FDA, said Gottlieb, needs to find ways to leverage these new constructs to better inform decisions. To do so, FDA must “support the development of, and access to, appropriate forms of reliable evidence that meet our standard for approval.” Just as clinical decisions are often made based on a “mosaic of information,” regulatory decisions could likewise consider a broad range of informative sources of evidence such as RWE. Gottlieb noted that there are no statutory or regulatory barriers to incorporating RWE, and that using RWE to make decisions about product marketing would be consistent with FDA’s practice of using RWE to make decisions about safety. Data from the real world, said Gottlieb, may even be more rich, diverse, and informative than data from RCTs that “speak to a limited and rigidly constructed circumstance.”

When FDA approves a medical product, a line is by necessity drawn between safety and efficacy on one side, and risk and uncertainty on the other, said Gottlieb. However, FDA as a public health agency has a mandate to embrace the full continuum of evidence available from all sources along the entire life cycle of a product. “We can’t allow our need for a point of regulatory accountability to prevent us from looking across the line we have to draw at practical information that’s collected both before and after our point of demarcation when a product gains a license for initial market entry.” RWE, said Gottlieb, could help FDA make more informed decisions along the continuum, from providing data about the benefit–risk profile of a product, allowing for early identification and a richer understanding of safety concerns.

To enable greater adoption of RWE for regulatory decisions, FDA will need to work with the health care system to change the way that clinical information is collected, said Gottlieb. Currently, structured data within EHRs are usually geared toward billing, and clinically relevant information is often hidden in unstructured notes that are inaccessible. While data from EHRs are not perfect—for example, data might be missing or nonsensical—there are new tools and technologies that allow for electronic audits of the integrity of the data that can give FDA and other stakeholders more confidence in the quality of the data they are using. In addition to data from EHRs, there is a need to collect more information directly from patients themselves. For example, a tool could collect information about gait and physical activity directly from an individual, rather than having a doctor do a walk test on a treadmill in the office. These tools will need to be validated, and because they are relatively new, both the product developer and the regulators will need to take a “leap of faith” to get these products on the market, he said.

In conclusion, Gottlieb stated that FDA “needs to do its part to advance the use of RWE.” FDA is taking steps to provide more clarity about its approach to the use of RWE in regulatory decisions, said Gottlieb, including

final guidance that was issued in August 2017² about the use of RWE in the development of devices. Gottlieb also noted that in the past 3 years, FDA has approved or cleared at least eight new medical devices and expanded the use of at least six technologies based on evidence derived from RWD. This evidence, said Gottlieb, was generated in less time and at a lower cost than in the past, and using RWE saved up to 2 years of development time. The adoption of RWE in the context of medical device regulation is more straightforward than in the context of drug regulation, said Gottlieb, because the end user of a device is usually a clinician, who is in a position to collect information, whereas the end user of a drug is a patient, who may not be able to collect and share information as readily. However, despite these challenges, FDA is currently working on policies to support the use of RWE in the approval of new indications for already marketed drugs, which may be especially relevant for drugs for rare diseases or unmet medical needs. Although RWE is not likely to replace traditional clinical data in many cases, Gottlieb said, there is an opportunity to incorporate RWE into FDA’s entire life cycle approach to medical product development.

FIT-FOR-PURPOSE EVIDENCE

McClellan followed Gottlieb’s discussion about FDA’s vision with a presentation about a cooperative agreement between FDA and Duke-Margolis. This partnership is driven by bipartisan legislative action by Congress³ that directed FDA to further explore using RWE in the regulatory framework. The legislation specifically requires FDA to hold workshops, evaluate the potential use of RWE, and issue draft guidance by the end of 2021. As part of these activities, FDA partnered with Duke-Margolis for the purpose of exploring the use of RWE for regulatory purposes and sponsored the three-part workshop series hosted by the Forum on Drug Discovery, Development, and Translation described in this proceedings. The Forum workshop series provided an ongoing venue to discuss overarching governance issues associated with using RWE for evaluating drugs, biologics, and devices. The complementary work at Duke-Margolis focused primarily on drugs and discussed more technical issues related to RWE application. Duke-Margolis released a white paper in September 2017⁴ that proposes a framework that can be used to guide sponsors and FDA in discussions about RWE, and seeks to clarify the current landscape of RWD/RWE for regulatory use.

___________________

² See https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm513027.pdf (accessed November 6, 2018).

³ Prescription Drug User Fee Act VI and the 21st Century Cures Act.

⁴ See https://healthpolicy.duke.edu/sites/default/files/atoms/files/rwe_white_paper_2017.09.06.pdf (accessed November 6, 2018).

In particular, the white paper discusses the various considerations that affect how researchers decide to structure a study that uses RWD, including the regulatory context, the clinical context, the availability of quality data, and the research methods for the evidence needed. McClellan said this framework can be used for thinking about what data and methods would be appropriate for a specific use of RWE. For example, many different methods can be used to turn RWD into RWE, including randomized methods, prospective or retrospective observational methods, or a hybrid of methods. The method that a researcher chooses depends on the intended use of the RWE, taking into consideration the clinical question, the type of data available, and the regulatory purpose for which the evidence is being generated. McClellan noted that regardless of the chosen method, it is unlikely that “one single real-world evidence study is going to be all the evidence that exists for most of these regulatory questions.” FDA looks at the totality of the evidence when making regulatory decisions, and RWE can be a part of the overall picture.

Another area of focus of the Duke-Margolis partnership with FDA, said McClellan, has been on the challenges of data. Although there is a growing quantity of RWD, ensuring the quality of RWD is considerably more difficult, said McClellan. There are concerns about making sure the data are of good provenance, and could be traceable back to the source if questions arose. Patient-generated data are increasingly available, but turning these data into quality RWE requires patient and provider support and adoption as well as good governance and stewardship practices. McClellan said there are promising tools and technologies that may help mitigate the challenges of data, such as blockchain, which can enable more secure aggregation of patient data from a variety of devices and sources.

The partnership has also been discussing methods and how to ensure that the research methods used are appropriate for the regulatory purpose, said McClellan. Several basic good practices for RWD were identified by the partnership, including

• Developing analytic plans that are transparent and specified in advance;
• Using robust primary data sources;
• Ensuring there are enough quality data to address the endpoints and conclusions;
• Making sure that methods fit into a “totality of the evidence” approach; and
• Using randomization when possible and appropriate.

McClellan concluded that “having a clear path to regulatory acceptability” for RWE could be “a big driver toward getting more robust and

interpretable” RWD, including clinic- and patient-generated data. This move toward better data could be “synergistic with a move toward new payment models” that focus on value and patient outcome, and together these movements could generate support for infrastructure needed for RWE studies. However, said McClellan, further work is needed. The practical use of RWE is still limited, and it will take further investment by stakeholders to build a foundation for the use of RWE for regulatory purposes, as well as to improve clinical practice. McClellan noted that some stakeholders—health insurance companies in particular—are making major investments in data infrastructure, with a goal of integrating data to support better decisions for patients. These types of efforts will be highly relevant to advancing the generation of RWE for regulatory purposes, he said, and will help to integrate the regulatory framework with a framework based on patient outcomes.