During the closing session of the workshop, individual panelists reflected on the day’s presentations and discussions, and identified practical strategies to generate momentum toward the evidence generation paradigm of the future, in which real-world evidence is systematically incorporated into processes for the development and evaluation of medical products. Summarizing workshop participants’ comments on factors that would be critical to achieving what Hernandez called “a vibrant ecosystem for real-world evidence,” Mark McClellan, director, Margolis Center for Health Policy, Duke University, and moderator of this final session, framed these critical elements for success in the context of four themes: data availability, data methods and quality, study design, and incentives. Highlights from workshop discussions focused on each of these thematic areas are summarized in the sections below.
DATA AVAILABILITY
The systems currently in place are not yielding enough data of high enough quality to be broadly usable for research, stated McClellan. Addressing this issue of data availability, Freda Lewis-Hall, chief medical officer and executive vice president, Pfizer Inc., underscored the need for a fully digitized EHR platform that is interactive and can provide data to:
- patients—to inform decisions on how best to improve their own health;
- providers—to guide treatment decisions;
- health systems—to improve the quality of care they deliver; and
- the research community—to answer questions that can drive improvements in the system.
Achieving this vision of a learning health system could benefit from enhanced integration of the research and clinical care enterprises. Despite the opportunities to leverage synergies and improve efficiency through an
integrated system, however, the potential of practice-based systems for research has yet to be realized. Although a number of technical hurdles remain, many individual workshop participants observed that the main barriers are cultural. Califf noted that “learning in [clinical] practice doesn’t seem to be one of the fundamental attributes that’s being valued right now.” Califf’s comments were echoed by Simon, who asserted that the biggest challenge to incorporating data from health care into research is not the quality of the data, but a fundamental problem with the way health care is currently practiced and, specifically, with the recording of information on the delivery of care. Simon expressed his belief that realizing the potential of a learning health system will ultimately require health care systems and providers to be held accountable for systematically and accurately capturing a record of treatment decisions, along with the rationale for those decisions, and for evaluating the impact of the treatment approach on patient outcomes. Also underpinning the system is data sharing, as discussed in more detail in Chapters 2 and 3. The ability to share data seamlessly across systems involves broad stakeholder engagement and a commitment to transparency. It was emphasized by several individual workshop speakers, however, that robust engagement of stakeholders in a learning health system will be bolstered if data partners, whether they are patients, consumers, providers, health plans, or health systems, trust that their data will be protected and that their confidentiality and privacy needs will be responsibly addressed. Califf noted that the privacy and trust principles and data security framework developed under the auspices of the National Institutes of Health’s (NIH’s) Precision Medicine Initiative (PMI) serve as a strong foundation on which to build going forward to maintain trust in the system.
Partnership
Broadening Stakeholder Engagement
Many individual workshop participants commented that a broadening of the stakeholder engagement process would support a pivot toward an evidence generation system designed to address the needs of multiple stakeholders simultaneously. As end users, patients, clinicians, and health systems could inform the processes that will be used to generate the evidence they ultimately depend on for treatment decisions. One workshop participant observed that in initiatives funded by PCORI, representatives from these stakeholder groups participate in advisory roles, as co-investigators, or on oversight committees, ensuring that their perspectives are incorporated into the design, execution, and dissemination of research. Another workshop participant added that there are a number of other “postregulatory decision makers” (e.g., payers, formulary committees,
guidelines developers, and health technology assessment organizations) that address the quality and relevance of the evidence; there are benefits to ensuring they are at the table for early discussions on evidence generation processes (e.g., methodology, data sources).
Several individual workshop participants noted that bringing to scale the kinds of successful initiatives highlighted at the workshop will be bolstered by a significant investment in infrastructure. In discussions on how such an infrastructure could be supported, one participant asked whether those investments would be made by the government, by advocacy organizations, or by industry groups. In response, Sherman expressed her belief that the federal government can be a partner but, given the current fiscal climate, it is not going to finance the development of the infrastructure on its own. Another participant stressed that the kind of public–private partnership approach that worked for the TVT Registry needs to be expanded. In such partnerships, federal agencies can use policy tools to develop opportunities to generate and use real-world evidence, but also have the leverage to bring stakeholders (including patients and patient organizations) together in a precompetitive way to discuss infrastructure needs, priorities, and how to move beyond approaches that are obsolete and ineffective. With multiple private stakeholders, infrastructure costs can be shared, said Berlin, but conveying the value of the infrastructure to industry partners will be critical to making the business case for their participation.
Engaging Patients and Consumers as Partners
Beyond identifying the kinds of stakeholders that need to be engaged in evidence generation processes, additional points of emphasis from individual workshop participants focused on redefining what engagement means, particularly for patients and consumers. “We have really got to change the way we think about how we engage with people,” said White, noting that in the PMI’s All of Us Research Program, there are no research subjects, only research participants. Lewis-Hall outlined three aspects of meaningful partnerships with patients: engaging them in the collection of data, giving data back to them, and assessing outcomes that are meaningful to them. One workshop participant observed that measuring outcomes important to patients (e.g., improved quality of life) enables the incorporation of those outcomes into value calculations. Although patient-centric outcomes like functionality traditionally have not been a focus, it was noted that progress has been made in this area in recent years. The Cancer Moonshot Blue Ribbon Panel set as 1 of its 10 priorities the incorporation of patient-reported outcomes and quality-of-life measures into EHRs, and PCORnet has endorsed many patient-reported outcome measures from the Patient-Reported Outcomes Measurement Information System (PROMIS).
Transparency
The importance of transparency arose in two different contexts during the workshop discussions. The first was transparency with regard to those whose data are collected and analyzed. In addition to reciprocal data sharing and forthright conversations about data security discussed in Chapters 2 and 3, ethical issues arise with secondary uses of data when research is embedded into the clinical care infrastructure. One participant noted that at his institution, possible secondary uses of data are explained upfront and patients are given the option of excluding themselves.
The other area where a need for improved transparency was noted is sharing of study methods and outcomes. Progress in data sharing for clinical trials represents a model that can be more broadly applied to improve transparency, said Berlin. In addition to sharing protocols, making available the code used to generate an analysis would not only improve transparency by showing the approach investigators took, but would also help demonstrate reproducibility by enabling the testing of methods on different datasets. Both are important to building the credibility of real-world evidence, added Shah.
STUDY DESIGN, DATA METHODS, AND DATA QUALITY
When is real-world evidence good enough and for what purposes? This framing question, posed by Berger, set up a series of discussions on the use of appropriate methodological approaches focused on study design, validating data methods, and improving data quality. In the context of these discussions, a number of individual participants emphasized that evidence should be fit-for-purpose, and the optimal methodology will depend on the research question.
Study Design
Discussing the relative value of different study designs, Califf emphasized that although RCTs are not the answer to every question, randomization has a critical role to play in ensuring that clinical trials provide definitive results. Novel designs that employ randomization at the point of care, Califf continued, can advance the generation of real-world evidence while shielding results against bias. Acknowledging that pragmatic trials are a valuable means of addressing many of the methodological concerns about real-world evidence, Berger pointed out that such trials are still expensive to conduct and cannot meet all of the needs of a learning health system. In fact, results from RCTs and observational studies often are comparable, he said, citing a Cochrane report that found little evidence for significant effect
estimate differences between observational studies and RCTs (though the authors of this report also note that factors other than study design should be considered when examining reasons for lack of agreement between RCTs and observational studies) (Anglemyer et al., 2014). On the other hand, Rothman noted that pragmatic trials may increase in efficiency and decrease in cost over time with advances in technology, as evidenced by the success of the ADAPTABLE trial. For example, technological advances will take advantage of how patients are identified, contacted, and recruited, noted Rothman, as well as how data are collected and standardized. To strengthen the reliability of observational studies, Berger suggested that many of the same approaches used to ensure the validity of RCT data be adopted, including
- preregistration of studies in public registries with prespecified protocols and data analysis plans;
- checks to ensure best methodological practices were followed; and
- ensuring repeatability through multiple studies on different datasets.
McClellan added that, for validation purposes, results from observational studies can be compared to RCT results if available, as was done during the early years of FDA’s Sentinel Initiative. Recognizing that the strength of evidence from observational studies falls short of that from well-controlled clinical trials, Berger and McClellan emphasized that with proper controls, calibration, and demonstrated repeatability using independent datasets, secondary data can nevertheless be used for causal inference. Moreover, observational studies are being used when RCTs are infeasible, as in the case of rare diseases.
Data Methods and Data Quality
Data from the health care enterprise are extraordinarily complex, said McClellan, and can mean different things when coming from different sources. Thus, it is important to have a robust understanding of the sources and characteristics of data being used to generate real-world evidence on the effectiveness and value of medical products. Many individual workshop participants noted that a lack of common standards and interoperability issues necessitate labor- and cost-intensive efforts to combine information from different records (e.g., manual data entry to populate registries or time-consuming curation efforts to generate linked datasets). Hernandez pointed out that health care lags far behind other industries in terms of the capability to share and generate insights from “big data” and stressed that policy makers need to be thinking about how to ensure, and potentially even mandate, the seamless interoperability of real-world data so they can
be shared across multiple data systems in a plug-and-play fashion using tools like standard application programming interfaces. While the challenge was identified by participants as one of the more tractable ones, interoperability issues were noted as a source of great inefficiency and lost potential for electronic health data.
Several individual workshop participants pointed to the increasing use of common data models to transform data into a standard format on the back end as a partial solution. Although there can be variability in the models themselves, this can be attributed to the different purposes for which the models were created, and reflects ongoing innovation. Common data models need to evolve to accommodate new kinds of data being generated. Once data have been mapped to a common data model, however, it is much easier to map those data to each other. Significant progress has been achieved using such back-end data curation methods, but considerable opportunity remains for the implementation of standards and common terminology for prospective data collection, as called for in the Nationwide Interoperability Roadmap published recently by the ONC (2015). Standardization on the front end has the potential to dramatically reduce the effort required for back-end data curation.
Shah elaborated on the importance of method validation given the higher standard of evidence and level of rigor required for causal inference. Stakeholders need confidence in the data models, statistics, and comparison methods that go into generating the evidence they will use to inform their decisions. Noting that common data models have been validated by application to a wide range of datasets used to conduct real-world studies, Shah suggested that a shared standard dataset—on the scale of millions of patients—could be used for benchmarking to facilitate more rapid progress in methods development in the real-world evidence realm. McClellan added that with more validated methods, there may be less need to share or aggregate sensitive raw data and more evidence could be generated using “cloud-based” approaches as described for Sentinel and OHDSI, where patient-level data are retained behind the firewalls of data partners.
INCENTIVES
Incentives are powerful drivers of change. Lewis-Hall observed that a shift from traditional separate evidence generation systems for clinical practice and research to an integrated model that addresses the real-world questions of all stakeholders will require a realignment of incentives, adding that it will be important to consider the different kinds of incentives that will work for different stakeholders. Shah added that the infrastructure for a real-world evidence generation platform should include a data strategy
that defines the incentives to ensure data are made available for clinical research over time.
Incentivizing Patients and Providers to Participate in Clinical Research
Many individual workshop participants emphasized that if the path forward for real-world evidence is to embed research in the clinical care infrastructure, the key elements of incentivizing health care providers and health systems to capture high-quality data and participate in prospective research will require a significant culture change. Tunis underscored the need to understand what motivates clinicians to engage. The following possible incentives for health care providers were suggested by individual workshop participants:
- Communicate the importance of provider participation—If health care providers are asked to take on additional responsibilities beyond their clinical burden to assist with patient recruitment and data collection for research purposes, it will be important for them to understand why their participation is critical and how the research could benefit their patients.
- Ask questions that interest providers and health systems—Providers and health systems will be more willing to engage in studies that are interesting to them and address the real-world questions they face, such as how to provide better care for their patients and how to reduce costs. Engaging providers in the development of questions early in the process can improve the relevance of the study to their needs. Asking questions that health systems care about could provide more opportunities to leverage the clinical care infrastructure and resources for research. For example, health system administrators may encourage providers to participate in research that helps answer questions related to performance improvement. Feeding information on study results back to providers could bolster their ongoing participation in research.
- Emphasize prestige—It is unethical to offer financial incentives to health care providers or institutions for participating in research studies, but recognition and prestige can be a powerful driver. Opportunities for recognition as a primary investigator and publication in peer-reviewed journals may be attractive to providers, particularly for those in academic institutions.
Similar incentives were suggested as possible motivators for engaging patients in research. Patients may be more likely to enroll in studies or share their data if researchers are asking questions and measuring outcomes
that matter to them and communicating how the results can help them make better decisions on treatment options. One participant reminded the audience of the simple power of a thank you and added that even in cases where there is no direct benefit to the patient, many patients will be willing to participate if they can understand how the research may help others.
The Role of Regulators
Steven Galson, senior vice president for Global Regulatory Affairs and Safety, Amgen Inc., noted that regulators have a number of tools that could be employed to incentivize the incorporation of real-world evidence into evidence generation processes. A relatively simple means of encouraging industry to take the risks perceived to be associated with real-world evidence generation is through improved communication and coordination efforts. Guidance on the use of real-world evidence for regulatory purposes like that recently published by FDA for medical devices (FDA, 2016) can help to increase confidence in expanding beyond traditional evidence collection processes. Furthermore, the 21st Century Cures Act, signed into law on December 13, 2016, directs FDA to produce a framework, program, and guidance on the use of real-world evidence to help support the approval of a new indication for a previously approved drug and to help support or satisfy postapproval study requirements (21st Century Cures Act, Public Law 114-255, 114th Cong., 2d sess. [December 13, 2016]). Sherman noted that, in addition to guidance, shifting the dialogue between FDA and industry to earlier in the development pathway would provide more opportunity for FDA to influence decisions on use of real-world data sources and capture of patient-centric outcome measures. FDA is also working on internal education regarding the appropriate use of real-world evidence. Another key leverage point FDA has, Sherman pointed out, is its labeling authority. Companies may be incentivized to generate real-world evidence if they knew it would be going into FDA labeling. As an example of how FDA is using many of its tools (e.g., grants, guidance, and coalition building) to advance the use of real-world evidence for the evaluation of medical devices, McClellan cited the NEST initiative.
The Role of Payers
Aligning payments for drugs and devices with payment reforms that are taking place in the health care industry would put drug and device manufacturers “on the hook” along with providers for demonstrating better patient population outcomes and lower total cost of care, said McClellan. Payers can employ a number of levers to realign payment-based incentives, including pay-for-performance and pay-for-certification. Robinson Beale
added that payers can pay for provider participation within organized and sound entities using big data for decision support. To facilitate this, it is important that payers know which of those big data products and methodologies are sound and will produce the best evidence to drive practice-level outcomes and help providers to be more effective, stressed Robinson Beale.
SOME PRACTICAL NEXT STEPS
McClellan brought the workshop to a close by asking panelists to identify practical next steps that can be initiated now that would significantly advance the generation and application of real-world evidence and our understanding of how medical products—those on the market today and those in the pipeline—perform in the real world.
Connecting Existing Systems and Initiatives
At the federal level, different agencies are spending significant energy and resources creating separate systems and initiatives to address similar and connected goals (see Box 5-1). Several individual workshop participants suggested that an important next step is to connect these existing systems into a national evidence generation platform. Califf emphasized that this is not about creating a single system, but rather a federation that works together under a common set of rules and facilitates data sharing to answer questions efficiently. Galson added that in the process, some systems and initiatives may need to be abandoned so that the limited resources available are being directed to those that are going to result in the greatest advances.
Developing End-to-End Use Cases
The reshaping of a national evidence generation system is a systems problem to be examined in an end-to-end fashion, so that standards, methods, and incentives are aligned across the whole. Instead of attempting to address each element of the system piecemeal, one practical next step, suggested Lewis-Hall, could be the development of end-to-end use cases. The process would start by asking the question: What would be the plan for using real-world data at discrete points and under specific circumstances for developing a new therapeutic? Such use cases would enable, for example, the identification of points in the process where incentives are not optimally aligned so that decisions could be made on how best to incentivize the stakeholders at those points. Iterated several times, the process will start to solve the system problem, observed Lewis-Hall.
Creating a Standard Process for Stakeholder Engagement
Several workshop participants expressed frustration with the current sequential process for evidence generation, which is inefficient and does not meet the needs of all stakeholders, often necessitating post-hoc efforts to address gaps. The evidence generated to support regulatory approval, for example, may not meet payer requirements for coverage determinations, resulting in a need for additional studies. Hernandez suggested that a formal process for stakeholder engagement be developed to guide efforts on working together in a collaborative fashion end to end. The goal of companies soliciting input from stakeholders (including patients, providers, payers, and regulators) early in the process would be to generate a single development plan that meets multistakeholder needs.
This page intentionally left blank.
