Examining the Impact of
Real-World Evidence on
Medical Product Development

PROCEEDINGS OF A WORKSHOP SERIES

Erin Hammers Forstag, Benjamin Kahn, Amanda Wagner Gee, and Carolyn Shore, Rapporteurs

Forum on Drug Discovery, Development, and Translation

Board on Health Sciences Policy

Health and Medicine Division

THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This activity was supported by contracts between the National Academy of Sciences and AbbVie Inc.; American Diabetes Association; Amgen Inc. (Contract No. GHC-COPS-CSARF-126570); Association of American Medical Colleges; AstraZeneca; Burroughs Wellcome Fund (Contract No. 1017664); Critical Path Institute; Eli Lilly and Company; FasterCures; Foundation for the National Institutes of Health; Friends of Cancer Research; GlaxoSmithKline; Johnson & Johnson; Merck & Co., Inc. (Contract No. CMO180308-0002499); National Institutes of Health (Contract No. HHSN263201200074I; Task Order No. HHSN26300093): National Cancer Institute, National Center for Advancing Translational Sciences, National Institute of Allergy and Infectious Diseases, National Institute of Mental Health, National Institute of Neurological Disorders and Stroke, Office of the Director; New England Journal of Medicine; Pfizer Inc.; Sanofi (Contract No. 33215505); Takeda Pharmaceuticals (Contract No. 53108); and the U.S. Food and Drug Administration (Grant No. 5R13FD005496-04 and Contract No. HHSP2233201400020B, Order No. HHSP23337055): Center for Drug Evaluation and Research, Office of the Commissioner. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.

International Standard Book Number-13: 978-0-309-48829-7
International Standard Book Number-10: 0-309-48829-X
Digital Object Identifier: https://doi.org/10.17226/25352

Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Copyright 2019 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2019. Examining the impact of real-world evidence on medical product development: Proceedings of a workshop series. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/25352.

PLANNING COMMITTEE ON EXAMINING THE IMPACT OF REAL-WORLD EVIDENCE ON MEDICAL PRODUCT DEVELOPMENT¹

MARK MCCLELLAN (Co-Chair), Robert J. Margolis Professor of Business, Medicine, and Policy, and Director, Duke-Margolis Center for Health Policy, Duke University

GREGORY SIMON (Co-Chair), Senior Investigator, Kaiser Permanente Washington Health Research Institute

JEFF ALLEN, President and Chief Executive Officer, Friends of Cancer Research

ANDREW BINDMAN, Professor of Medicine, Epidemiology, and Biostatistics, University of California, San Francisco

JOHN GRAHAM, Senior Vice President, Medical Engagement and Value Evidence & Outcomes, GlaxoSmithKline (from March 2018)

ADAM HAIM, Chief, Treatment and Preventive Intervention Research Branch, National Institute of Mental Health, National Institutes of Health

MICHAEL HORBERG, Executive Director, Research, Community Benefit, and Medicaid Strategy, Mid-Atlantic Permanente Medical Group, and Director, Mid-Atlantic Permanente Research Institute

PETRA KAUFMANN, Director, Office of Rare Diseases Research and Division of Clinical Innovation, National Center for Advancing Translational Sciences, National Institutes of Health (until May 2018)

RICHARD KUNTZ, Senior Vice President and Chief Scientific, Clinical & Regulatory Officer, Medtronic, Inc.

ELLIOT LEVY, Senior Vice President, Global Development, Amgen Inc.

DAVID MADIGAN, Professor of Statistics, and Dean, Faculty of Arts and Sciences, Columbia University

DEVEN MCGRAW, Chief Regulatory Officer, Ciitizen

JOHN DAVID NOLEN, Physician Executive, Clinical Strategy, Cerner Corporation (until October 2017)

RICHARD PLATT, Professor and Chair, Department of Population Medicine, and Executive Director, Harvard Pilgrim Health Care Institute, Harvard Medical School

PATRICK VALLANCE, President, Research and Development, GlaxoSmithKline (until March 2018)

___________________

¹ The National Academies of Sciences, Engineering, and Medicine’s planning committees are solely responsible for organizing the workshop, identifying topics, and choosing speakers. The responsibility for the published Proceedings of a Workshop Series rests with the workshop rapporteurs and the institution.

JOANNE WALDSTREICHER, Chief Medical Officer, Johnson & Johnson

MARCUS WILSON, Co-Founder and President, HealthCore, Inc.

Health and Medicine Division Staff

CAROLYN SHORE, Senior Program Officer (from August 2017)

ANNE CLAIBORNE, Senior Program Officer (until August 2017)

AMANDA WAGNER GEE, Program Officer

BEN KAHN, Associate Program Officer (from October 2017)

JONATHAN PHILLIPS, Associate Program Officer (until October 2017)

MELVIN JOPPY, Senior Program Assistant

ANDREW M. POPE, Director, Board on Health Sciences Policy

Consultant

ERIN HAMMERS FORSTAG, Science Writer

FORUM ON DRUG DISCOVERY, DEVELOPMENT, AND TRANSLATION¹

RUSS B. ALTMAN (Co-Chair), Kenneth Fong Professor of Bioengineering, Genetics, Medicine, and (by courtesy) Computer Science, Stanford University

ROBERT M. CALIFF (Co-Chair), Vice Chancellor for Health Data Science, Duke University, and Scientific Advisor, Verily Life Sciences

CHRISTOPHER P. AUSTIN, Director, National Center for Advancing Translational Sciences, National Institutes of Health

LINDA BRADY, Director, Division of Neuroscience and Basic Behavioral Science, National Institute of Mental Health, National Institutes of Health

TANISHA CARINO, Executive Director, FasterCures

RICHARD DAVEY, Deputy Clinical Director, National Institute of Allergy and Infectious Diseases, National Institutes of Health

JAMES H. DOROSHOW, Deputy Director, Clinical and Translational Research, National Cancer Institute, National Institutes of Health

JEFFREY M. DRAZEN, Editor-in-Chief, New England Journal of Medicine

STEVEN K. GALSON, Senior Vice President, Global Regulatory Affairs and Safety, Amgen Inc.

CARLOS GARNER, Vice President, Global Regulatory Affairs, Eli Lilly and Company

JULIE L. GERBERDING, Chief Patient Officer and Executive Vice President, Strategic Communications, Global Public Policy, and Population Health, Merck & Co., Inc.

LYNN D. HUDSON, Chief Science Officer, Critical Path Institute

JEFF HURD, Regional Clinical Account Director, AstraZeneca

S. CLAIBORNE JOHNSTON, Dean, Dell Medical School, The University of Texas at Austin

KATHARINE KNOBIL, Chief Medical Officer, GlaxoSmithKline

FREDA C. LEWIS-HALL, Executive Vice President & Chief Medical Officer, Pfizer Inc.

ALLISON MCELVAINE, Vice President, Research & Scientific Programs, American Diabetes Association

ROSS MCKINNEY, JR., Chief Scientific Officer, Association of American Medical Colleges

JOSEPH P. MENETSKI, Deputy Director, Research Partnerships, Foundation for the National Institutes of Health

___________________

¹ The National Academies of Sciences, Engineering, and Medicine’s forums and roundtables do not issue, review, or approve individual documents. The responsibility for the published Proceedings of a Workshop Series rests with the workshop rapporteurs and the institution.

BERNARD H. MUNOS, Founder, Innothink Center for Research in Biomedical Innovation

MICHAEL SEVERINO, Executive Vice President, Research & Development, and Chief Scientific Officer, AbbVie Inc.

RACHEL E. SHERMAN, Principal Deputy Commissioner, Office of the Commissioner, U.S. Food and Drug Administration

ELLEN V. SIGAL, Chair and Founder, Friends of Cancer Research

LANA R. SKIRBOLL, Vice President, Science Policy, Sanofi

BRIAN L. STROM, Chancellor, Rutgers Biomedical and Health Sciences, Rutgers University

AMIR TAMIZ, Director, Division of Translational Research, National Institute of Neurological Disorders and Stroke, National Institutes of Health

PAMELA TENAERTS, Executive Director, Clinical Trials Transformation Initiative, Duke University

JOHN WAGNER, Senior Vice President and Head of Clinical and Translational Sciences, Takeda Pharmaceuticals

JOANNE WALDSTREICHER, Chief Medical Officer, Johnson & Johnson

CARRIE WOLINETZ, Associate Director of Science Policy, Office of Science Policy, National Institutes of Health

JANET WOODCOCK, Director, Center for Drug Evaluation and Research, U.S. Food and Drug Administration

Forum Staff

CAROLYN SHORE, Forum Director

REBECCA ENGLISH, Program Officer

AMANDA WAGNER GEE, Program Officer

SYLVIA NCHA, Associate Program Officer

EESHAN KHANDEKAR, Research Associate

MELVIN JOPPY, Senior Program Assistant

ANDREW M. POPE, Director, Board on Health Sciences Policy

Reviewers

This Proceedings of a Workshop Series was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published proceedings as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the process.

We thank the following individuals for their review of this proceedings:

Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the proceedings nor did they see the final draft before its release. The review of this proceedings was overseen by DAN MASYS, University of Washington. He was responsible for making certain that an independent examination of this proceedings was carried out in accordance with standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the rapporteurs and the National Academies.

This page intentionally left blank.

Contents

ACRONYMS AND ABBREVIATIONS

1 INTRODUCTION

Defining the Terms

The U.S. Food and Drug Administration’s Vision for the Future

Fit-for-Purpose Evidence

2 PERSPECTIVES ON REAL-WORLD EVIDENCE

Payer Perspective

Delivery Systems Perspective

Patient Perspective

Discussion

3 LEARNING FROM SUCCESS

Salford Lung Studies

Sentinel

Device Registries

4 BARRIERS AND DISINCENTIVES TO THE USE OF REAL-WORLD EVIDENCE AND REAL-WORLD DATA

The Use of Real-World Evidence and Real-World Data in Product Development

Making Choices About Research Design

Accelerating Evidence Generation Through Defragmentation

Evidence Hierarchies

Opportunities to Integrate Real-World Data and Real-World Evidence in Research

Discussion

5 GETTING UNSTUCK: MYTHBUSTING THE CURRENT SYSTEM

From Precision to Reliability

Integrating the New with the Old

Regulatory Perspective

6 WHEN IS A REAL-WORLD DATA ELEMENT FIT FOR ASSESSMENT OF ELIGIBILITY, TREATMENT EXPOSURE, OR OUTCOMES?

Real-World Data Elements

Illustrative Examples

Discussion: Characterizing Real-World Data and Real-World Evidence

Decision Aid

“Fit for Purpose” and Relevance of Data

Patient-Generated Data

Discussion: Real-World Data Concerns for Future Research

7 HOW TIGHTLY SHOULD INVESTIGATORS ATTEMPT TO CONTROL OR RESTRICT TREATMENT QUALITY IN A PRAGMATIC OR REAL-WORLD TRIAL?

Illustrative Examples

Decision Aid

Patient-Centered Research

Context of the Decision

Obligations to Patients

8 OBSCURING INTERVENTION ALLOCATION IN TRIALS TO GENERATE REAL-WORLD EVIDENCE: WHY, WHO, AND HOW?

Illustrative Examples

Decision Aid

Discussion

9 GAINING CONFIDENCE IN OBSERVATIONAL COMPARISONS

Illustrative Examples

Discussion: Observational Studies and Randomization

Decision Aid

Presentations: Observational Studies and Bias

Discussion: The Future of Observational Studies

10 LOOKING AHEAD

Real-World Evidence to Improve Health Technology Assessment

Real-World Evidence to Turn Patients into Partners

Real-World Evidence to Transform Research and Development

Real-World Evidence to Inform Regulatory Decisions

Final Thoughts

REFERENCES

APPENDIXES

A Related Resources

B Workshop One Agenda

C Workshop Two Agenda

D Workshop Three Agenda

This page intentionally left blank.

Boxes, Figures, and Tables

BOXES

1-1 Workshop Series Statement of Task

1-2 Definitions of Real-World Data and Real-World Evidence from Various Stakeholders

4-1 Barriers to Adoption of Real-World Evidence and Real-World Data in Research as Discussed by Individual Workshop Participants

5-1 A Patient’s Perspective on Randomized Controlled Trials (RCTs) as Presented by McCollister-Slipp

6-1 Feedback on the Decision Aid as Discussed by Individual Workshop Participants

6-2 The Importance of Patient-Generated Data as Discussed by Individual Workshop Participants

6-3 Potential Sources of Bias in Real-World Data as Discussed by Individual Workshop Participants

7-1 Patient Compliance as Discussed by Individual Workshop Participants

7-2 Features of the ArthritisPower Research Registry, with Smartphone Application and User-Friendly Interface

7-3 Regulatory Decision Making Regarding Clinical Strategy as Discussed by Schneeweiss

8-1 Other Examples of Blinded and Non-Blinded Studies Discussed Throughout the Workshop Series as Presented by Individual Workshop Participants

8-2 Feedback on the Decision Aid as Discussed by Individual Workshop Participants

8-3 Considerations Around Uncertainty in Study Design

9-1 Feedback on the Decision Aid as Discussed by Individual Workshop Participants

10-1 Key Messages Identified by Individual Speakers

FIGURES

2-1 Platform for Engaging Everyone Responsibly (PEER)

3-1 Salford Lung Studies design

3-2 Salford Lung Studies platform diagram

3-3 Data elements available in Sentinel

4-1 The importance of integrated data

4-2 Transformation from traditional to new approach of evidence generation

4-3 Mosaic methodologies to blend randomized controlled trial (RCT) and real-world evidence (RWE) approaches

5-1 Learning health care system

5-2 Broad sources of real-world data and broad uses of real-world evidence

5-3 Effect estimates from published observational studies on antidepressants and the risk of preeclampsia

5-4 Effect estimates from published observational studies on all disease states, all treatments, and all causal effects

5-5 Systematically generated evidence from observational data, comparing all depression treatments on all outcomes of interest

6-1 Possible sources of real-world data

6-2 Real-world overall survival

6-3 Real-world data (RWD) decision aid

6-4 Decision tree to identify possible data sources

6-5 Linked, de-identified data sources at OptumLabs

6-6 Decision process with an existing real-world data asset

6-7 Effect of medical procedure or surgery on resting heart rate

7-1 Cumulative probability of experiencing a significant suicide attempt or hospitalization to prevent suicide

7-2 Decision aid on questions to consider regarding participant safety and investigator control of treatments in a trial taking place in a community care setting

8-1 Study design for the INVESTED trial

8-2 Decision aid on questions to consider regarding when and whether to obscure intervention allocation (commonly known as blinding) in trials intended to generate real-world evidence

8-3 Blinding not warranted when utilization factors are unknown

9-1 Sources of data for research

9-2 A priori confidence in validity of study findings

9-3 Prematched herpes zoster vaccinated and unvaccinated cohorts

9-4 Postmatched herpes zoster vaccinated and unvaccinated cohorts

9-5 Decision aid on questions to consider to assess and minimize bias in non-randomized observational comparisons

10-1 Acceptability of real-world evidence (RWE) across Europe

10-2 monARC Bionetworks Integrated Learning Platform

10-3 Hypothetical example of traditional design versus Bayesian adaptive design

10-4 Standard separate trials for drugs A and B

10-5 Combined platform trial for drugs A and B

10-6 Use of real-world data in the evaluation of drugs for rare diseases

TABLES

6-1 Real-World Endpoints in Six Datasets

6-2 Correlation Between Real-World Overall Survival and Real-World Extracted Endpoints

6-3 Patient-Generated Health Data for a Study of Multiple Sclerosis (MS)

7-1 Considerations for Study Design Restrictions

10-1 Hypothetical Examples of Cost Savings

This page intentionally left blank.

Acronyms and Abbreviations

This page intentionally left blank.