3

Learning from Success

In this session of the first workshop, participants heard about successfully completed and ongoing initiatives that generated, collected, and/or analyzed real-world data (RWD) and real-world evidence (RWE). Speakers were asked to describe the features that led to success in their particular program, and to consider how these successes could be generalized and scaled for future projects.

SALFORD LUNG STUDIES

The Salford Lung Studies were two late-phase randomized controlled trials (RCTs) conducted in Salford, United Kingdom, and the surrounding areas, said Martin Gibson, chief executive officer of Northwest EHealth. Sponsored by GlaxoSmithKline (GSK), the Salford Lung Studies were the first studies in the world to evaluate the effectiveness of a prelicense medication in a real-world setting, said Gibson. There were two separate studies—one for chronic obstructive pulmonary disease (COPD) and one for asthma—evaluating the same drug (Relvar Ellipta); the studies enrolled more than 7,200 patients. Enrolled patients were monitored in near real time for safety and outcomes, using city-wide linked electronic health records (EHRs). The studies showed that the drug was effective at improving outcomes for both COPD and asthma.¹ Gibson said that tradi-

___________________

¹ See https://www.nejm.org/doi/full/10.1056/NEJMoa1608033 (accessed January 4, 2019); https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(17)32397-8/fulltext (accessed January 4, 2019).

tionally, premarket studies look at efficacy, and postmarket studies look at effectiveness. The Salford Lung Studies, however, were late Phase III trials of a premarket medication that were designed to gather information on effectiveness in real-world conditions. Gibson noted that new technologies are allowing researchers to blur the lines among traditional types of studies, and to look at real-world effectiveness earlier in the process.

Gibson explained the reasons why the Salford Lung Studies were conducted in this particular region. First, because the National Health Service (NHS) serves all UK residents, the results generated from NHS data are likely to be generalizable. Second, the general practitioners in NHS have been using EHRs since the mid-1990s, and the EHR system is a good system with high-quality data, said Gibson. Third, Salford itself is served by a single large university hospital, and the primary and secondary care data in this region have been integrated since 2002. The Salford Hospital is “regarded as the most digitally mature organization in the NHS,” which greatly facilitated the lung studies. Finally, said Gibson, there was an existing close relationship between the community of Salford and the health system. Northwest EHealth, which was established in 2008 to improve research using EHRs, had already worked in the community and there was a “connected community of care.” Gibson stressed that this element was critical to the success of the Salford studies.

Gibson gave a brief overview of the design of the COPD study (see Figure 3-1). He noted that the criteria for inclusion were much more open than they would be for a standard trial. In the trial itself, participants were randomized during an initial visit, and then had an end-of-study visit 12 months later. If assigned to the active group, patients received their drug through the usual community pharmacies, and all patients were monitored “behind the scenes” through the EHR system. Care was taken to ensure that there was minimal intervention, and that patients were truly receiving normal care during the 12 months between the initial and end-of-study visits.

When GSK first suggested these studies, nothing similar had ever been done before, said Gibson. Many questions needed to be answered to proceed:

• What study design would best achieve internal and external validity?
• What data existed?
• How good were the data?
• How could the data be accessed, managed, and validated?
• What were the evidence needs of research authorities, regulators, and payers?
• Was there a large enough population to power such a study?

Answering these questions, however, was just the “tip of the iceberg,” said Gibson. To carry out the studies, every element had to be operationalized, which required extensive forethought. Marie Kane, chief operating officer of Northwest EHealth, explained the process. Kane said that although Northwest EHealth is primarily a technology company, the biggest part of carrying out the Sanford Lung Studies was engaging with people. General practitioners, pharmacists, and specialists were critical for supporting the study day to day, while people who worked at every level of the health care organization had to buy in as well. For example, said Kane, the information technology (IT) department was not accustomed to being involved in clinical research, so care had to be taken to enlist their cooperation and support. Regulators and payers were engaged in the process in an attempt to ensure that the evidence that came out of the studies would be acceptable to these stakeholders. More than 3,000 people—physicians, nurses, pharmacists, data managers, and administrators—were trained as part of the research delivery team.

In addition to engaging with people, Northwest EHealth had to build a system to collect, manage, and validate the data, as well as a system for safety reporting. Kane presented a simplified diagram of the platform that was built for the studies (see Figure 3-2). Only structured and coded data were collected, said Kane. Data were collected from a number of sources, including Salford general practitioners, out-of-area patient episodes, 140 retail pharmacies, and data from community services. Kane said these studies required far more investment in data processing and error management than traditional clinical studies, due in part to variability in the data as they were collected as well as the scale and complexity of the data linkages required to determine patient outcomes. One of the challenges, said Kane, was that these sources used different IT systems, collected different data, or had data that were not integrated. However, by going into the “back tables” of the systems, most of the data were able to be retrieved in a usable format.

One benefit of monitoring patients in near real time, said Kane, was that serious adverse events could be detected and reported almost immediately. The safety monitoring system was set up with criteria for alerting researchers about potential safety issues, and a safety team reviewed these alerts on a daily basis. Alerts were investigated using the patient’s EHR, and when the cause of the adverse event could not be determined, the sponsor was alerted of the adverse event. The alert system, said Kane, can be “tuned up or tuned down” depending on the specific safety concerns and the needs of the sponsor.

In retrospect, said Kane, they underestimated the scale and complexity of the task of recruiting patients and staff, and of developing data systems that could collect and manage data from multiple unaligned sys-

tems. Some of the challenges were small, but consequential. For example, because the drug was prelicense, the drug name was not in the dropdown list in the EHR system, so providers had to type in the name. Mistakes or typos would have resulted in missing these data, so an algorithm was built to identify and aggregate the disparate terms. Other challenges were large. For example, pharmacy data had never been used for these purposes before, and how to best access and use these data is still an open question. Another data-related challenge, said Kane, was ensuring that they got the right data rather than just more data. Collecting too much data costs money and can create inefficiencies. “It needs to be the right data with the right provenance, with the right frequency,” said Kane. However, said Kane, the most critical component of the study was not building data systems or recruiting patients; it was building relationships among people by recruiting the right partners and ensuring that everyone understood how to fulfill their role in the study.

Based on the lessons learned from this experience, Northwest EHealth has converted the design developed for Salford into a cloud-based platform. The platform has been fully reengineered and revalidated, and has configurable and modular applications, said Kane. Depending on the needs of the researcher, he or she can use different components of the applications. There is also a master data management system, which allows a researcher to develop configurable data schema and data dictionaries. Other tools that Northwest EHealth has developed, said Gibson, include tools for recruiting and engaging patients. The system for recruitment allows researchers to screen the population and identify patients who may be appropriate subjects (using anonymized information); the system then contacts the patients’ providers to recruit them into the trial. A patient engagement tool that has been built allows patients to sign up directly to give consent for the use of their EHR and to be recruited for studies. The benefits of performing research using EHR data, said Gibson, are numerous. First, a researcher can collect information on a patient throughout her life, rather than just for a set period of time in a study. Second, using EHR data reduces the influence of research on patient’s care—patients and providers can go about their normal course of care, while data are being captured for research. Finally, data collection can be streamlined and designed for the specific protocol, saving money and increasing efficiency. Once the system for one protocol is up and running, it can be used for other things, such as safety monitoring, said Gibson.

During the discussion that followed presentations, participants discussed whether and how the Salford model could function as a “franchise” that could be exported to other health systems for other disease questions. Kane said the “franchise” part of the studies would consist of the methods developed and the lessons learned from each variation—not all studies

would need to be conducted using the same exact approach. John Graham, senior vice president, medical engagement and value evidence and outcomes of GSK, agreed that the studies had generated both a reusable infrastructure and lessons learned that could be applied in other disease areas; GSK is already applying a similar model in cardiovascular and renal disease studies in the United States. Califf added that finding a way to franchise these models—that is, to identify and scale up the common elements of successful programs—is critical for making research more efficient, but that each new project would likely have its own “regional flavor.” Gibson said that recruiting health systems to participate became easier with each study because of the positive experiences of the study participants and the investments made in building relationships as well as the desire of health systems to participate in exciting, new, and relevant studies. These experiences help to alleviate fears of systems that do not normally conduct research: “We have the tools and the capability to give them the confidence that this is something they can take part in.”

SENTINEL

The Sentinel Initiative is a national medical product monitoring system that was launched in 2008 by the U.S. Food and Drug Administration (FDA) in response to legislation that required FDA to use electronic health data to support postmarketing medical product evaluation, said Richard Platt, professor and chair, Harvard Medical School, and executive director, Harvard Pilgrim Health Care Institute. While the mandate asked FDA to assess the use, safety, and effectiveness of regulatory medical products, FDA has focused Sentinel primarily on safety. However, FDA has always intended that Sentinel would be used to support a variety of activities, including clinical research, randomized trials, and public health surveillance.

Sentinel is the product of a collaboration between FDA and a large number of organizations that bring both data and scientific expertise. In essence, Sentinel is a curated, distributed dataset that adheres to a common data model, said Platt. Sentinel is a fairly simple system, with a set of linked flat file records. The data that can be accommodated by Sentinel are wide ranging, including administrative data (e.g., age, sex, zip code, enrollment, medical and pharmacy benefits, encounter diagnoses and procedures, ambulatory pharmacy dispensing), laboratory test results, vital signs, death data, immunization records, and in-patient data (see Figure 3-3). Platt said “hundreds of millions of person years and billions of encounters” are represented in the Sentinel data, and about 10 percent of the people have some sort of laboratory test results.

Sentinel is a distributed system, which means that each of the data partners retains its own data, and the data are interrogated by the exchange of

executable programs. Each data partner can opt in or out of any individual query. Platt noted that while Sentinel is a public enterprise and a public resource, the participation of private organizations is critical to its success: Sentinel “is nothing if the data partners aren’t there.” One of the things that makes Sentinel data particularly useful, said Platt, is the fact that the data are extensively curated before use, which is not necessarily true of other large medical datasets. This curation, while it takes time and energy, means that the users of the data do not need to spend time assessing the quality of the data.

One particular advantage of Sentinel, said Platt, is that it allows for different studies to be conducted on the same research question and the same data, using different methods. Simon noted that one of the issues with observational studies is that when researchers choose different methods and get different results, it can be difficult to know which method is correct. With Sentinel, different methods can be systematically evaluated, under known conditions, said Platt.

Platt gave workshop participants six examples of how Sentinel has been used, either by itself or linked to other data sources; he noted that these examples were chosen because they have the potential to be scalable.

Sentinel Alone: Prospective Surveillance Pilot of Rivaroxaban Safety
(Bai et al., 2017; Chrischilles et al., 2018; Patel et al., 2011)

This project tested the ability to use Sentinel to do prospective surveillance, said Platt. The researchers compared data from patients on warfarin and rivaroxaban, while controlling for more than 70 confounders, including age, sex, comorbidities, usage, and diagnosis. Outcomes that were examined included gastrointestinal bleed, intracranial hemorrhage, and ischemic stroke. Platt noted that this observational study correlated well with randomized trials, which “should give us some confidence that we can be attentive to how the product is working in actual practice.” Furthermore, using Sentinel data allows researchers to explore populations that are not well represented in the clinical trial. Platt noted that “others may not agree” with using observational data in this way, but that for situations or populations where a clinical trial will not be done, there may be ways “to decide whether the observational data are credible enough.”

Sentinel with Chart Review: Risk of Intussusception After Rotavirus Vaccination
(Yih et al., 2014)

The first vaccine to prevent rotavirus infection in infants was licensed in 1998, but withdrawn in 1999 due to risk of intussusception, a form of bowel obstruction, said Platt. Alternative rotavirus vaccines showed no

increased risk in clinical trials, but postlicensure studies in other countries suggested an increased risk. In 2010, FDA began a study to quantify the possible risk among U.S. infants. Researchers used data from three Sentinel partners, and gathered data using the Current Procedural Terminology (CPT) codes for immunization and the International Classification of Diseases, Ninth Revision (ICD-9) codes for intussusception and related issues. Because CPT and ICD-9 codes are “not sufficiently specific enough to do high-quality epidemiology,” said Platt, the researchers actually reviewed the full-text medical records of patients after redacting direct identifiers. Using data on 500,000 patients, the algorithm identified potential cases, and researchers obtained full medical records for 80 percent of these. Pediatricians adjudicated the records, and they found a risk of intussusception of about 1.5 per 100,000 children immunized. The clinical trials that had been performed for these vaccines, said Platt, had enrolled only 60,000 children, which may not have been sufficient to find the actual risk. This study demonstrates the value of being able to link administrative data to full-text records, said Platt.

Sentinel Linked to Registries: Linking Mother–Infant Pairs²

Platt gave a brief overview of the difficulties of linking maternal and infant data; he noted that this gap has “bedeviled health services research” for decades. Sentinel data partners have data about linked mother–infant pairs, unlinked mothers, and unlinked infants, and state departments of health have birth certificate data. Researchers attempted to link moms and infants using these data, and were able to link more than 80 percent of the records from four data partners. Platt observed that while researchers thought linking mothers and infants would require the information from the registries (i.e., the birth certificates), nearly all of the pairs were able to be linked through information already in Sentinel (i.e., subscriber ID and last names and addresses). However, he noted, the birth certificates did include a lot of rich information that is useful for researchers, such as gestational age and smoking history.

Sentinel Linked to EHRs: PCORnet ADAPTABLE Trial³

Platt used the National Patient-Centered Clinical Research Network (PCORnet) ADAPTABLE trial as an example of how a Sentinel-like system has been used to link with data from EHRs. The ADAPTABLE trial, said

___________________

² See https://www.sentinelinitiative.org/communications/publications/2017-icpe-presentation-developing-mother-infant-cohort-sentinels-prism (accessed January 4, 2019).

³ See http://theaspirinstudy.org (accessed November 6, 2018).

Platt, is simple, with the goal of randomizing 20,000 people with coronary artery disease to either low-dose or high-dose aspirin for prevention. Follow-up of patients will occur largely through EHRs and payer data. This trial is an example of using available data to enable pragmatic research, and demonstrates how Sentinel could similarly be used.

Sentinel Linked with Patient-Reported Data: A Mobile App for Studies of Medication Safety⁴

Sentinel researchers built a mobile app to enable individuals who have data in the Sentinel system to report information that can easily be merged with the system, said Platt. This app, now known as the MyStudies smartphone app,⁵ is currently being field tested. Pregnant women with the app are asked a variety of questions, including smoking history, level of nausea, over-the-counter drug use, and other areas of interest. These patient-reported data can easily be merged with the individual’s information in Sentinel, which should make for an even richer dataset.

Sentinel for Randomized Trials: Sentinel IMPACT-AFib
(Cocoros et al., 2018)

The IMPACT-AFib trial is a pragmatic clinical trial that uses Sentinel data to test methods to improve the use of oral anticoagulants (OACs) in patients with atrial fibrillation. Working with five data partners, direct mailers were sent to 40,000 health plan members with AFib, at a high risk for stroke, and not currently taking an OAC. Mailers also went to the patients’ providers to encourage the consideration of prescribing an OAC. Outcomes will be assessed at 12 and 24 months, using Sentinel data. The primary outcome is initiation of OAC, with secondary outcomes including rates of stroke hospitalization and bleeding events. Eligibility for the trial was also determined using Sentinel data; an algorithm identified patients who were at risk and not currently being treated with an OAC. This trial, said Platt, demonstrates that it is possible to use Sentinel to identify individuals who are eligible for intervention, and also to support the implementation and follow-up of the trial.

___________________

⁴ See https://www.fda.gov/downloads/Drugs/ScienceResearch/UCM625206.pdf (accessed November 6, 2018).

⁵ See https://www.fda.gov/Drugs/ScienceResearch/ucm624785.htm (accessed January 4, 2019); https://www.fda.gov/Drugs/ScienceResearch/ucm624785.htm (accessed January 4, 2019).

DEVICE REGISTRIES

Rachael Fleurence, National Evaluation System for health Technology (NEST), told workshop participants about applying lessons learned from device registries to other treatment types. Fleurence noted that devices are sometimes “forgotten” in a conversation that focuses primarily on drugs, but that RWD and RWE are essential for early identification of problems with devices. Fleurence gave several examples of devices for which RWE could be useful: metal-on-metal hip implants, the contraceptive device Essure, and power morcellators for uterine fibroids.

Fleurence noted that there are major differences in the ways that devices and drugs are regulated by FDA, and that these differences impact the generation and use of RWE in each arena. For drugs, said Fleurence, approval is based on substantial evidence from well-controlled investigations. Approval requires disclosure of adverse events, and manufacturers must conduct postapproval safety studies. The National Drug Code ensures that the drugs that patients take can be accurately identified in claims and EHR data. On the device side, there are two pathways for approval: premarket approval (PMA) and 5-10K. For PMA, generally a single small study is required, said Fleurence. Postapproval requirements are variable, and only apply to PMAs. While a system to accurately track the use of devices (the Universal Device Identifier) was put in place in 2015, it is not compulsory for payers or health systems. As a consequence, it is difficult, but not impossible, to identify brand-specific devices in EHR and claims data, said Fleurence.

Fleurence sees a particular benefit of RWE and RWD for devices in two ways. First, RWE can help support a “pre/postmarket shift,” in which devices can receive approval more quickly, when there is confidence that there will be robust postapproval data collected on safety and effectiveness. Second, RWE could help to improve the Medical Device Reporting system, with implementation of automated surveillance methods. These methods could be used to quickly identify problems with a device so that action could be taken (e.g., pull device from the market and/or conduct further safety research). These two benefits of RWE would help patients gain access to innovative, safe, and effective devices more quickly, said Fleurence. To explore and capitalize on these potential benefits of RWE, the NEST Coordinating Center (NESTcc) was established in 2017 as a catalyst to “support timely and reliable development of high-quality real-world evidence.” To do so, NESTcc will establish partnerships with a range of stakeholders that provide data and analytics solutions, will set data quality and methods standards, and will offer products and services of value to stakeholders, said Fleurence.

Registries have historically played an important role in the regulatory space for medical devices, said Fleurence. There are some very high-quality

registries in the device space, including the Vascular Quality Initiative, the International Consortium of Orthopedic Registries, and the Transcatheter Valve Therapy Registry. In addition to these individual registries, there is a movement toward a Coordinated Registries Network, which links existing registries with other sources of data such as claims and EHR data. These registries, she said, “provide high-quality and fit-for-purpose data, and support both observational and randomized interventions, possibly at a lower cost.” In addition, algorithms can be used to assess registry data for automated safety surveillance. Registries are currently the main source of RWE decisions by FDA’s Center for Devices and Radiological Health, said Fleurence.

However, despite the utility of registries, there are challenges. First, developing and maintaining a registry is expensive, and inputting data into registries is time consuming. Second, registries cannot be developed for all devices, disease areas, and patient populations. Third, there is limited use of the Unique Device Identifier in claims and EHR data. Finally, a device outcome can depend not just on the device itself, but on the skill of the provider who is implanting the device. Differentiating among issues linked to the device itself and issues due to the experience of the person performing the intervention must be accounted for in studies of registry data, she said. In addition to these device-specific challenges, there are a number of ecosystem-wide challenges—that is, challenges with using any type of RWD and RWE. These challenges include the difficulty of ensuring the quality of data, missing data, data linkage issues, data privacy and security concerns, and issues with appropriate analytic methods. Fleurence noted in particular that there are challenges with administrative issues—for example, the length of time it takes for research studies to obtain legal review and Institutional Review Board approval. She hypothesized that the single device registries model is likely to evolve soon. Rather, there will be a new model and an expanded definition of registries; for example, a “registry” might actually be a tool for linking preexisting data such as EHRs and claims data, rather than a repository for new input of data.

In conclusion, Fleurence said tremendous progress has been made in the past decade. Technology and adoption of new tools has accelerated, and stakeholders are collaborating to change culture and to increase the involvement of patients. However, the current barriers are still real, and time, resources, and leadership will be needed to overcome them. “There is no question that the future lies in the use of RWD and generating robust RWE,” said Fleurence, but “how far off that future lies is the question.”