Building from the discussions of stakeholder needs and the potential applications of real-world data in the two preceding workshop sessions, discussions during Session III focused on opportunities and challenges for broadly adapting promising practices. Four successful use cases were discussed that showcased how alternative data sources can be used to answer real-world questions. The case studies fell into three categories of real-world evidence approaches—randomization in the clinical setting, research embedded in registries, and population-based surveillance using health system data.
RANDOMIZATION IN THE CLINICAL SETTING
Pragmatic randomized controlled trials (pRCTs) address limitations in the generalizability of results from traditional RCTs by embedding clinical research in the care delivery setting and randomizing interventions at the point of care. The Salford Lung Studies, described by Roddam, are notable as the world’s first preapproval pRCTs. The studies, which were initiated in 2012 and conducted in the Salford area of Greater Manchester in the United Kingdom, were designed to evaluate the effectiveness of medication using a once-daily combination inhaler in comparison to existing maintenance therapy for chronic obstructive pulmonary disease (COPD) and asthma. The question drove the approach: Because the comparator arm was continuing treatment with usual care, which would have involved multiple inhalers, it would not have been possible to conduct this evaluation through a traditional RCT in a controlled setting, observed Roddam. The investigators adopted broad inclusion criteria and, importantly, no other aspects of the care being rendered were changed, so the study provided a truly representative assessment of real-world effectiveness in the clinical setting in which the medication would be used. Roddam emphasized that partnerships with local health care entities (e.g., general practitioners, pharmacies) were critical to success, reiterating a point made earlier in the workshop regarding the importance of engaging providers in research. The study involved only two visits—at initiation and again at the end of the study. Between the two visits was a 12-month follow-up period, during which there was continuous real-time collection of data from EHRs and daily safety monitoring. Although the effect of the medication on asthma
is still under investigation, results for COPD endpoints have been publicly reported and a significant reduction in moderate to severe exacerbations was observed for the intervention group, with no increase in the rate of serious adverse events, as compared to usual care (Vestbo et al., 2016). Thus, through this real-world evidence approach, the first Salford Lung Study provided information that could be used to have meaningful conversations with patients about treatment options for COPD.
The pragmatic nature of the trial enabled efficiencies in patient recruitment, by leveraging the clinical system to identify eligible patients, and in data collection. The more data that can be collected from the EHR (versus an independent collection effort), the more efficient the trial becomes, said Roddam. However, although the United Kingdom has a single unique identifier for patients, there were still multiple data streams for each patient (representing different interactions with the health care system, including primary care visits, after hours/emergency care, and pharmacy data), and data collected by different health care entities varied significantly. As a result, Roddam noted, significant effort to combine data on the back end was required.
Discussing how additional efficiencies could be achieved, Laura Dember, professor of medicine, University of Pennsylvania, suggested that consideration should be given to opportunities to reuse the infrastructure for future trials. The Salford Lung Studies, for example, necessitated the training of 3,000 individuals in good clinical practice—these trained individuals now represent a resource that could be leveraged, and doing so could reduce costs for future trials.
Addressing the question of adaptability, John Hernandez of Verily Life Sciences queried Roddam on the potential for doing this kind of preapproval study in the United States. Roddam admitted that the National Health Service infrastructure in the United Kingdom, including identifiers that enabled the linking of patient records, was key to the feasibility of the study, particularly for the real-time monitoring of safety outcomes. Investigators were confident that any serious adverse event would be captured in the system. “If you need to do that in the U.S., it’s really hard,” he said. Rothman observed that Phase IV (postapproval) pragmatic trials for comparative effectiveness have been successfully carried out in the United States. However, it is not clear whether these kinds of trials would be acceptable to FDA in the preapproval setting and for labeling changes, suggesting that more regulatory guidance is needed. Sherman indicated that FDA is actively working on producing this kind of guidance, reiterating that the sharp distinction between pre- and postmarket in terms of requirements for evidence generation is outdated. There is no reason, Sherman said, “that the evidence we need to know how to use something should be any different than the evidence that we need to know whether or not to approve it or to license it.”
RESEARCH EMBEDDED IN REGISTRIES
Registries represent an efficient mechanism of collecting data for specific analytic purposes and can bring discipline to upfront data collection. Device registries have helped to advance the use of real-world evidence in the medical device world and may offer lessons for the drug development field. Michael Mack, chair, Cardiovascular Service Line, Baylor Scott & White Health, described how a public–private partnership led to the creation of a registry-based infrastructure for real-world evidence generation on the effectiveness and safety of transcatheter aortic valve replacement (TAVR). TAVR allows replacement of heart valves without surgery by delivering the valve by a catheter from an artery in the leg to the heart. In 2011, FDA developed an initiative for strengthening postmarket surveillance for devices using national device registries (FDA, 2012). The Society for Thoracic Surgeons/American College of Cardiology (STS/ACC) Transcatheter Valve Therapy (TVT) Registry evolved from that effort as a product of a partnership among FDA, STS, ACC, the Duke Clinical Research Institute, and CMS (Carroll et al., 2013). Discussing the importance of incentives, Mack underscored the critical role CMS played in the success of the TVT Registry. In its national coverage determination, CMS approved the TAVR treatment with coverage under evidence development, whereby CMS requires that services or items be provided in the context of clinical study participation or that additional clinical data be collected to support further evidence development (CMS, 2014). It also mandated participation in the TVT Registry. As a result of these conditions, data from virtually all U.S. patients receiving the device (approximately 75,000 to date) have been captured in the registry. Additionally, linkage to CMS claims data enabled evaluation of long-term (i.e., 1 year and longer) patient outcomes.
Mack outlined a number of ways the TVT Registry has supported real-world evidence generation:
- Postmarket regulatory purposes—safety reports are generated and sent to FDA quarterly, supporting postmarket surveillance, and the registry has also supported nested postapproval studies1 with three different device manufacturers.
- Premarket regulatory decision making—registry data on off-label use of the device has supported label expansion, and several inves-
___________________
1 FDA can impose at the time of device approval a requirement for medical device manufacturers to conduct postapproval studies to generate additional evidence on product safety and effectiveness in the postmarket context. For more information see http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/PostmarketRequirements/PostApprovaStudies/ucm135263.htm#q5 (accessed November 25, 2016).
-
tigational device exemption studies have been nested within the TVT Registry.
- Quality improvement—quarterly reports generated from registry data have enabled risk-adjusted benchmarking across TAVR sites.
- Research—registry data are being incorporated into several different research studies, including an evaluation of volume-to-outcome relationships that may inform the optimal number of TAVR sites, and a comparative effectiveness study comparing outcomes for surgical and non-surgical interventions.
Workshop participants discussed opportunities and challenges to scaling the TVT Registry model as a mechanism for real-world evidence generation. A primary identified challenge was sustainability, given how expensive and burdensome it is to populate and maintain a registry. The TVT Registry is populated manually, and data are collected using a case study form with 400 data fields. Mack noted that a budget of $6 million per year is required to run the registry, and that does not include the cost of full-time employees at each of the 420 TAVR sites who enter the registry data. In the case of TAVR, the device was expensive (approximately $32,000/device) and CMS was the only payer, so the agency was able to condition reimbursement on participation in the registry. How this model would work without the CMS mandate for registry participation is unclear, he said. Without the reimbursement incentive, clinicians would probably not be willing to take on this level of burden. Carroll suggested that increasing the efficiency of the process by which key data elements are extracted from the EHR will be critical to scaling the registry model. Automatic population of a registry with data from EHRs could reduce the burden and cost associated with collecting and entering data, although, Mack cautioned, given the current state of EHRs, autopopulation would likely only be able to be used to populate 20 percent of data fields. Reducing the number of required data elements would also help improve efficiency but, Mack emphasized, it can be challenging to get agreement from all stakeholder organizations, each of which has its own specific interests in the data. Jesse Berlin, vice president and global head of epidemiology, Johnson & Johnson, stressed that to overcome this issue it is important to demonstrate the value of a data field before it can be required.
An expansion of the registry model may be incentivized by payment reform efforts, observed Califf. Bundled payment might encourage the capture of long-term follow-up data in a registry-type database, but, as Carroll noted, providers entering the initial diagnosis/treatment information do not always have access to long-term follow-up data. Therefore, that kind of longitudinal data capture relies on partnerships similar to the one described for the TVT Registry, where CMS data were linked to provide 1-year out-
come data. Robinson Beale added that a shift to measuring true outcomes would be required, rather than just process measures. Additionally, she emphasized that the full potential of registries in a learning health system depends on their ability to provide actionable information to providers in real-time, for example, by connecting to decision support tools.
In the context of bringing use cases to scale, Mack showed a model proposed by John Laschinger, medical officer, FDA, representing a vision for a tiered national device registry infrastructure (see Figure 4-1). In this model, the complexity of the dataset captured in the registry and the level of multistakeholder support would vary across the tiers, depending on the intended use of the registry. The registry at each site would be fit-for-purpose, and sites would apply for a specific level of certification based on the desired level of participation. For those sites in the outermost tier where the registry would be used primarily for local quality improvement work, only the minimum dataset would be collected. This minimum dataset would be common across all tiers (captured through a standard case report form), but additional “modules” would be added sequentially as the desired capability to support studies increased (moving toward the innermost tier).
NOTE: QI = quality improvement.
SOURCE: Mack presentation, 2016.
POPULATION-BASED SURVEILLANCE USING HEALTH SYSTEM DATA
The ability to aggregate data from EHRs and link health-related data from other sources (e.g., claims) has enabled the conduct of large-scale observational research that is answering real-world questions about safety and effectiveness and providing information that informs clinical trials. As examples of the kinds of studies that can be conducted using these large linked datasets to generate real-world evidence, Lesley Curtis, professor of medicine and director for Pragmatic Health Services Research, Duke Clinical Research Institute, shared her experiences with the FDA Sentinel Initiative and Shah discussed the Observational Health Data Sciences and Informatics (OHDSI) program. Workshop attendees also heard a summary from Louis Fiore, executive director, Massachusetts Veterans Epidemiology Research and Information Center on the U.S. Department of Veterans Affairs’ framework for gathering and applying data gathered from the population of veterans in their system (see Box 4-1).
FDA Sentinel Initiative
The impetus for the Sentinel Initiative was the FDA Amendments Act of 2007, which mandated the creation of an active surveillance system for continued safety evaluation of marketed medical products. Sentinel uses private health plan data (clinical, administrative claims, and registry data) for near real-time active safety surveillance. Rather than aggregating the data into a centralized repository, Sentinel uses a distributed data network architecture so that health plans are able to keep their data behind a firewall. Executable code is sent to FDA Sentinel’s 19 health plan partners to run behind their firewalls against a common data model similar to the one used by PCORnet. Summarized data are provided to FDA. Through this approach, Sentinel has been able to access data from approximately 190 million individuals, all with private health insurance, resulting in a defined population with longitudinal data. Highlighting Sentinel’s impact, Curtis cited 4 FDA drug safety communications; 48 methods papers; 70 peer-reviewed articles; and more than 100 assessments of products, conditions, and product outcome pairs. In discussing opportunities to build from the lessons learned from Sentinel, Curtis laid out three ingredients she believed were critical to its success: engaged partners, attention to data quality, and reusable tools (see Box 4-2). Several workshop participants noted that efforts to adapt the Sentinel model are already under way. In the United States, the new National Evaluation System for health Technology (NEST) initiative will use real-world evidence to conduct postmarket evaluations of safety and performance of medical devices. Additionally, distributed data networks for safety surveillance across Europe are being considered under the auspices of the Innovative Medicines Initiative, a public–private partnership involving the European Medicines Agency and the European pharmaceutical industry.
Observational Health Data Sciences and Informatics Initiative
The OHDSI community is a multistakeholder, interdisciplinary group of investigators working collaboratively to bring out the value of observational research and generate evidence that will improve health decision making through building open-access tools, best practice methods, and a large data network. Currently, 94 different sites from across the world contribute data to the OHDSI network. Although not all sites will participate in every study, that equates to potential access to data from approximately 650 million individuals. The OHDSI suite of tools enables clinical characterization, patient-level prediction, and population-level effect estimation (causal inference). Shah described two of the tools that have been developed by the OHDSI community:
- Achilles—a database profiling tool for characterization of databases (e.g., demographics, subpopulations, and data quality assessment).
- Atlas—an integrated platform for building cohorts (e.g., for observational studies), as well as for database exploration and population-level analysis.
In addition to these and other tools, the OHDSI community provides recommendations on methods to improve quality of evidence. This includes both standard diagnostics, such as propensity models used to ensure that comparisons are valid (e.g., two drugs being compared are actually likely to have been prescribed for a given set of patients), and calibration methods using controls that should be used when looking at drug adverse-event associations.
As an example of the potential of OHDSI, Shah discussed a recently published analysis that used the data network to characterize the complement of drugs (e.g., first line, second line) prescribed for three diseases—hypertension, depression, and diabetes (Hripcsak, 2016). This kind of large-scale, real-world characterization of practice can only be done empirically. The results of such analyses can be used to inform future clinical studies—for example, comparative effectiveness studies of different second-line treatments (Vashisht et al., 2016). Figure 4-2 depicts a model that
SOURCE: Shah presentation, 2016.
shows how OHDSI can inform both clinical practice and research. In the practice setting, OHDSI’s toolsets and data networks could be leveraged to support both clinical practice and research for such a comparative effectiveness study (Longhurst et al., 2014). This model describes three distinct, yet potentially interactive, levels where questions about care are asked in OHDSI: for individual patients (top right), in practice-based or large cohorts (bottom right), and for large populations (left side). In the practice setting, there may be situations for an individual patient where an obvious first-line treatment exists, as depicted by availability of a clinical guideline at the top of the flow chart in Figure 4-2. However, for that same patient, there may not be an obvious second-line treatment, and evidence generation becomes necessary, as depicted by central rings. Thus, when no clinical guideline based on high-quality RCT data is available, health care providers could use a decision support tool, represented by the flow chart shown in Figure 4-2, that leverages OHDSI’s existing data networks to aggregate data sources—such as EHRs, claims data, and even social media data—and conduct analyses of treatment results for large cohorts of similar patients (when available in OHDSI databases) and draw on the resultant practice-based evidence to inform care decisions, depicted on the right half of Figure 4-2. As shown on the left half of Figure 4-2, continuous monitoring of such analyses being conducted by clinicians using OHDSI could, as a byproduct, inform a priority list for generating higher quality evidence through pragmatic or large simple trials.
This page intentionally left blank.
