Examining the Impact of
Real-World Evidence on
Medical Product Development
PROCEEDINGS OF A WORKSHOP SERIES
Erin Hammers Forstag, Benjamin Kahn, Amanda Wagner Gee, and Carolyn Shore, Rapporteurs
Forum on Drug Discovery, Development, and Translation
Board on Health Sciences Policy
Health and Medicine Division
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by contracts between the National Academy of Sciences and AbbVie Inc.; American Diabetes Association; Amgen Inc. (Contract No. GHC-COPS-CSARF-126570); Association of American Medical Colleges; AstraZeneca; Burroughs Wellcome Fund (Contract No. 1017664); Critical Path Institute; Eli Lilly and Company; FasterCures; Foundation for the National Institutes of Health; Friends of Cancer Research; GlaxoSmithKline; Johnson & Johnson; Merck & Co., Inc. (Contract No. CMO180308-0002499); National Institutes of Health (Contract No. HHSN263201200074I; Task Order No. HHSN26300093): National Cancer Institute, National Center for Advancing Translational Sciences, National Institute of Allergy and Infectious Diseases, National Institute of Mental Health, National Institute of Neurological Disorders and Stroke, Office of the Director; New England Journal of Medicine; Pfizer Inc.; Sanofi (Contract No. 33215505); Takeda Pharmaceuticals (Contract No. 53108); and the U.S. Food and Drug Administration (Grant No. 5R13FD005496-04 and Contract No. HHSP2233201400020B, Order No. HHSP23337055): Center for Drug Evaluation and Research, Office of the Commissioner. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-48829-7
International Standard Book Number-10: 0-309-48829-X
Digital Object Identifier: https://doi.org/10.17226/25352
Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2019 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2019. Examining the impact of real-world evidence on medical product development: Proceedings of a workshop series. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/25352.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.
Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.
Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.
For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.
PLANNING COMMITTEE ON EXAMINING THE IMPACT OF REAL-WORLD EVIDENCE ON MEDICAL PRODUCT DEVELOPMENT1
MARK MCCLELLAN (Co-Chair), Robert J. Margolis Professor of Business, Medicine, and Policy, and Director, Duke-Margolis Center for Health Policy, Duke University
GREGORY SIMON (Co-Chair), Senior Investigator, Kaiser Permanente Washington Health Research Institute
JEFF ALLEN, President and Chief Executive Officer, Friends of Cancer Research
ANDREW BINDMAN, Professor of Medicine, Epidemiology, and Biostatistics, University of California, San Francisco
JOHN GRAHAM, Senior Vice President, Medical Engagement and Value Evidence & Outcomes, GlaxoSmithKline (from March 2018)
ADAM HAIM, Chief, Treatment and Preventive Intervention Research Branch, National Institute of Mental Health, National Institutes of Health
MICHAEL HORBERG, Executive Director, Research, Community Benefit, and Medicaid Strategy, Mid-Atlantic Permanente Medical Group, and Director, Mid-Atlantic Permanente Research Institute
PETRA KAUFMANN, Director, Office of Rare Diseases Research and Division of Clinical Innovation, National Center for Advancing Translational Sciences, National Institutes of Health (until May 2018)
RICHARD KUNTZ, Senior Vice President and Chief Scientific, Clinical & Regulatory Officer, Medtronic, Inc.
ELLIOT LEVY, Senior Vice President, Global Development, Amgen Inc.
DAVID MADIGAN, Professor of Statistics, and Dean, Faculty of Arts and Sciences, Columbia University
DEVEN MCGRAW, Chief Regulatory Officer, Ciitizen
JOHN DAVID NOLEN, Physician Executive, Clinical Strategy, Cerner Corporation (until October 2017)
RICHARD PLATT, Professor and Chair, Department of Population Medicine, and Executive Director, Harvard Pilgrim Health Care Institute, Harvard Medical School
PATRICK VALLANCE, President, Research and Development, GlaxoSmithKline (until March 2018)
___________________
1 The National Academies of Sciences, Engineering, and Medicine’s planning committees are solely responsible for organizing the workshop, identifying topics, and choosing speakers. The responsibility for the published Proceedings of a Workshop Series rests with the workshop rapporteurs and the institution.
JOANNE WALDSTREICHER, Chief Medical Officer, Johnson & Johnson
MARCUS WILSON, Co-Founder and President, HealthCore, Inc.
Health and Medicine Division Staff
CAROLYN SHORE, Senior Program Officer (from August 2017)
ANNE CLAIBORNE, Senior Program Officer (until August 2017)
AMANDA WAGNER GEE, Program Officer
BEN KAHN, Associate Program Officer (from October 2017)
JONATHAN PHILLIPS, Associate Program Officer (until October 2017)
MELVIN JOPPY, Senior Program Assistant
ANDREW M. POPE, Director, Board on Health Sciences Policy
Consultant
ERIN HAMMERS FORSTAG, Science Writer
FORUM ON DRUG DISCOVERY, DEVELOPMENT, AND TRANSLATION1
RUSS B. ALTMAN (Co-Chair), Kenneth Fong Professor of Bioengineering, Genetics, Medicine, and (by courtesy) Computer Science, Stanford University
ROBERT M. CALIFF (Co-Chair), Vice Chancellor for Health Data Science, Duke University, and Scientific Advisor, Verily Life Sciences
CHRISTOPHER P. AUSTIN, Director, National Center for Advancing Translational Sciences, National Institutes of Health
LINDA BRADY, Director, Division of Neuroscience and Basic Behavioral Science, National Institute of Mental Health, National Institutes of Health
TANISHA CARINO, Executive Director, FasterCures
RICHARD DAVEY, Deputy Clinical Director, National Institute of Allergy and Infectious Diseases, National Institutes of Health
JAMES H. DOROSHOW, Deputy Director, Clinical and Translational Research, National Cancer Institute, National Institutes of Health
JEFFREY M. DRAZEN, Editor-in-Chief, New England Journal of Medicine
STEVEN K. GALSON, Senior Vice President, Global Regulatory Affairs and Safety, Amgen Inc.
CARLOS GARNER, Vice President, Global Regulatory Affairs, Eli Lilly and Company
JULIE L. GERBERDING, Chief Patient Officer and Executive Vice President, Strategic Communications, Global Public Policy, and Population Health, Merck & Co., Inc.
LYNN D. HUDSON, Chief Science Officer, Critical Path Institute
JEFF HURD, Regional Clinical Account Director, AstraZeneca
S. CLAIBORNE JOHNSTON, Dean, Dell Medical School, The University of Texas at Austin
KATHARINE KNOBIL, Chief Medical Officer, GlaxoSmithKline
FREDA C. LEWIS-HALL, Executive Vice President & Chief Medical Officer, Pfizer Inc.
ALLISON MCELVAINE, Vice President, Research & Scientific Programs, American Diabetes Association
ROSS MCKINNEY, JR., Chief Scientific Officer, Association of American Medical Colleges
JOSEPH P. MENETSKI, Deputy Director, Research Partnerships, Foundation for the National Institutes of Health
___________________
1 The National Academies of Sciences, Engineering, and Medicine’s forums and roundtables do not issue, review, or approve individual documents. The responsibility for the published Proceedings of a Workshop Series rests with the workshop rapporteurs and the institution.
BERNARD H. MUNOS, Founder, Innothink Center for Research in Biomedical Innovation
MICHAEL SEVERINO, Executive Vice President, Research & Development, and Chief Scientific Officer, AbbVie Inc.
RACHEL E. SHERMAN, Principal Deputy Commissioner, Office of the Commissioner, U.S. Food and Drug Administration
ELLEN V. SIGAL, Chair and Founder, Friends of Cancer Research
LANA R. SKIRBOLL, Vice President, Science Policy, Sanofi
BRIAN L. STROM, Chancellor, Rutgers Biomedical and Health Sciences, Rutgers University
AMIR TAMIZ, Director, Division of Translational Research, National Institute of Neurological Disorders and Stroke, National Institutes of Health
PAMELA TENAERTS, Executive Director, Clinical Trials Transformation Initiative, Duke University
JOHN WAGNER, Senior Vice President and Head of Clinical and Translational Sciences, Takeda Pharmaceuticals
JOANNE WALDSTREICHER, Chief Medical Officer, Johnson & Johnson
CARRIE WOLINETZ, Associate Director of Science Policy, Office of Science Policy, National Institutes of Health
JANET WOODCOCK, Director, Center for Drug Evaluation and Research, U.S. Food and Drug Administration
Forum Staff
CAROLYN SHORE, Forum Director
REBECCA ENGLISH, Program Officer
AMANDA WAGNER GEE, Program Officer
SYLVIA NCHA, Associate Program Officer
EESHAN KHANDEKAR, Research Associate
MELVIN JOPPY, Senior Program Assistant
ANDREW M. POPE, Director, Board on Health Sciences Policy
Reviewers
This Proceedings of a Workshop Series was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published proceedings as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the process.
We thank the following individuals for their review of this proceedings:
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the proceedings nor did they see the final draft before its release. The review of this proceedings was overseen by DAN MASYS, University of Washington. He was responsible for making certain that an independent examination of this proceedings was carried out in accordance with standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the rapporteurs and the National Academies.
This page intentionally left blank.
Contents
The U.S. Food and Drug Administration’s Vision for the Future
2 PERSPECTIVES ON REAL-WORLD EVIDENCE
4 BARRIERS AND DISINCENTIVES TO THE USE OF REAL-WORLD EVIDENCE AND REAL-WORLD DATA
The Use of Real-World Evidence and Real-World Data in Product Development
Making Choices About Research Design
Opportunities to Integrate Real-World Data and Real-World Evidence in Research
5 GETTING UNSTUCK: MYTHBUSTING THE CURRENT SYSTEM
Integrating the New with the Old
Discussion: Characterizing Real-World Data and Real-World Evidence
“Fit for Purpose” and Relevance of Data
Discussion: Real-World Data Concerns for Future Research
8 OBSCURING INTERVENTION ALLOCATION IN TRIALS TO GENERATE REAL-WORLD EVIDENCE: WHY, WHO, AND HOW?
9 GAINING CONFIDENCE IN OBSERVATIONAL COMPARISONS
Presentations: Observational Studies and Bias
Discussion: The Future of Observational Studies
Real-World Evidence to Improve Health Technology Assessment
Real-World Evidence to Turn Patients into Partners
Real-World Evidence to Transform Research and Development
This page intentionally left blank.
Boxes, Figures, and Tables
BOXES
1-1 Workshop Series Statement of Task
1-2 Definitions of Real-World Data and Real-World Evidence from Various Stakeholders
5-1 A Patient’s Perspective on Randomized Controlled Trials (RCTs) as Presented by McCollister-Slipp
6-1 Feedback on the Decision Aid as Discussed by Individual Workshop Participants
6-2 The Importance of Patient-Generated Data as Discussed by Individual Workshop Participants
6-3 Potential Sources of Bias in Real-World Data as Discussed by Individual Workshop Participants
7-1 Patient Compliance as Discussed by Individual Workshop Participants
7-3 Regulatory Decision Making Regarding Clinical Strategy as Discussed by Schneeweiss
8-2 Feedback on the Decision Aid as Discussed by Individual Workshop Participants
8-3 Considerations Around Uncertainty in Study Design
9-1 Feedback on the Decision Aid as Discussed by Individual Workshop Participants
10-1 Key Messages Identified by Individual Speakers
FIGURES
2-1 Platform for Engaging Everyone Responsibly (PEER)
3-1 Salford Lung Studies design
3-2 Salford Lung Studies platform diagram
3-3 Data elements available in Sentinel
4-1 The importance of integrated data
4-2 Transformation from traditional to new approach of evidence generation
5-1 Learning health care system
5-2 Broad sources of real-world data and broad uses of real-world evidence
6-1 Possible sources of real-world data
6-2 Real-world overall survival
6-5 Linked, de-identified data sources at OptumLabs
6-6 Decision process with an existing real-world data asset
6-7 Effect of medical procedure or surgery on resting heart rate
8-1 Study design for the INVESTED trial
8-3 Blinding not warranted when utilization factors are unknown
9-1 Sources of data for research
9-2 A priori confidence in validity of study findings
9-3 Prematched herpes zoster vaccinated and unvaccinated cohorts
9-4 Postmatched herpes zoster vaccinated and unvaccinated cohorts
10-1 Acceptability of real-world evidence (RWE) across Europe
10-2 monARC Bionetworks Integrated Learning Platform
10-3 Hypothetical example of traditional design versus Bayesian adaptive design
10-4 Standard separate trials for drugs A and B
10-5 Combined platform trial for drugs A and B
10-6 Use of real-world data in the evaluation of drugs for rare diseases
TABLES
6-1 Real-World Endpoints in Six Datasets
6-2 Correlation Between Real-World Overall Survival and Real-World Extracted Endpoints
6-3 Patient-Generated Health Data for a Study of Multiple Sclerosis (MS)
This page intentionally left blank.
Acronyms and Abbreviations
ACE |
angiotensin converting enzyme |
AE |
adverse event |
AFib |
atrial fibrillation |
aNSCLC |
advanced non-small cell lung cancer |
ARIA |
Active Risk Identification and Analysis |
ARISTOTLE |
Apixaban for Reduction in Stroke and Other Thromboembolic Events in Atrial Fibrillation |
BEST |
Biologics Effectiveness and Safety |
BLA |
biologics license application |
Cal INDEX |
California Integrated Data Exchange |
CBER |
Center for Biologics Evaluation and Research |
CDER |
Center for Drug Evaluation and Research |
CDRH |
Center for Devices and Radiological Health |
CE |
European conformity |
CIOMS |
Council for International Organizations of Medical Sciences |
COPD |
chronic obstructive pulmonary disease |
CPT |
Current Procedural Terminology |
CRISP |
Chesapeake Regional Information System for our Patients |
CSR |
Clinical Study Report |
CTTI |
Clinical Trials Transformation Initiative |
EHR |
electronic health record |
EMA |
European Medicines Agency |
ENGAGE AF-TIMI 48 |
Effective Anticoagulation with Factor Xa Next Generation in Atrial Fibrillation Thrombolysis in Myocardial Infarction 48 |
FDA |
U.S. Food and Drug Administration |
GCP |
good clinical practice |
GSK |
GlaxoSmithKline |
ICD |
International Statistical Classification of Diseases and Related Health Problems |
ICH |
International Conference on Harmonisation |
IMDRF |
International Medical Device Regulators Forum |
IMI |
Innovative Medicines Initiative |
INR |
international normalized ratio |
InterSePT |
International Suicide Prevention Trial |
INVESTED |
INfluenza Vaccine to Effectively Stop Cardio Thoracic Events and Decompensated heart failure |
IRB |
Institutional Review Board |
JHHS |
Johns Hopkins Health System |
KP |
Kaiser Permanente |
MAPPs |
Medicines Adaptive Pathways to Patients |
MDEpiNet |
Medical Device Epidemiology Network |
MI |
myocardial infarction |
NEST |
National Evaluation System for health Technology |
NESTcc |
NEST Coordinating Center |
NHS |
National Health Service |
NICE |
National Institute for Health and Care Excellence (United Kingdom) |
NOAC |
novel oral anticoagulant |
NSAID |
non-steroidal anti-inflammatory drug |
OAC |
oral anticoagulant |
OHDSI |
Observational Health Data Sciences and Informatics |
OS |
overall survival |
PCORnet |
National Patient-Centered Clinical Research Network |
PEER |
Platform for Engaging Everyone Responsibly |
PFS |
progression-free survival |
PGHD |
patient-generated health data |
PMA |
premarket approval |
PPRN |
patient-powered research network |
PRO |
patient-reported outcome |
PROTECT |
Pharmacoepidemiological Research Outcomes of Therapeutics by European Consortium |
PTSD |
posttraumatic stress disorder |
PXE |
pseudoxanthoma elasticum |
RCT |
randomized controlled trial |
RE-LY |
Randomized Evaluation of Long-term Anticoagulation Therapy |
ROCKET AF |
Rivaroxaban Once Daily Oral Direct Factor Xa Inhibition Compared with Vitamin K Antagonism for Prevention of Stroke and Embolism Trial in Atrial Fibrillation |
RWD |
real-world data |
RWE |
real-world evidence |
SOC |
standard of care |
SOP |
standard operating procedure |
SSRI |
selective serotonin reuptake inhibitor |
TAVR |
transcatheter aortic valve replacement |
TTNT |
time to next treatment |
TTP |
time to progression |
TTTD |
time to treatment discontinuation |
VA |
U.S. Department of Veterans Affairs |
This page intentionally left blank.