National Academies Press: OpenBook
« Previous: Appendix B: Workshop One Agenda
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×

Appendix C

Workshop Two Agenda

Workshop Two: Practical Approaches

March 6–7, 2018

National Academy of Sciences Building, Room 120
2101 Constitution Avenue, NW, Washington, DC 20418

The National Academies of Sciences, Engineering, and Medicine is convening a three-part workshop series examining how real-world evidence development and uptake can enhance medical product development and evaluation. The workshops will advance discussions and common knowledge about complex issues relating to the generation and usage of real-world evidence, including fostering development and implementation of the science and technology of real-world evidence generation and usage.

  • Workshop One (September 19–20, 2017) focused on how to align incentives to support collection and use of real-world evidence in health product review, payment, and delivery. Incentives need to address barriers impeding the uptake of real-world evidence, including barriers to transparency.
  • Workshop Two (March 6–7, 2018) will illuminate what types of data are appropriate for what specific purposes and suggest practical approaches for data collection and evidence use by developing and working through example use cases.
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
  • Workshop Three (July 17–18, 2018) will examine and suggest approaches for operationalizing the collection and use of real-world evidence.

DAY 1: MARCH 6, 2018

8:30 a.m. Breakfast Available Outside Room 120
8:40 a.m. Welcome and Opening Remarks
GREGORY SIMON, Workshop Series Co-Chair

Investigator

Kaiser Permanente Washington Health Research Institute

Session I: When Can We Rely on Real-World Data?

Session Discussion Questions:

  • When can we have confidence in EHR data from real-world practice to accurately assess study eligibility, key prognostic factors, and study outcomes?
  • When can we have confidence in data generated outside of clinical settings (e.g., mobile phones, connected glucometers, connected blood pressure monitors)?
  • When does adjudication or other postprocessing of real-world data add value?
Moderator: Gregory Daniel, Duke-Margolis Center for Health Policy
Session Discussants

JESSE BERLIN

Vice President and Global Head, Epidemiology

Johnson & Johnson

ANDY BINDMAN

Professor of Medicine

University of California, San Francisco

9:00 a.m. Introduction and Background to Inform the Discussion: Novel Oral Anticoagulants in Comparison with Warfarin
ADRIAN HERNANDEZ

Vice Dean for Clinical Research

Duke University School of Medicine

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
9:20 a.m. Open Discussion with Audience
  • What questions can characterize the utility of any real-world data source and signal reliability before a study is performed (examples below)?
    • When is accuracy good enough to reasonably and consistently identify the right population?
    • When is accuracy good enough to reasonably and consistently assess the exposure or intervention?
    • When is accuracy good enough to reasonably and consistently assess the right outcome?
    • Are there any big safety issues that would be missed?
    • Are there concerns about data collection or entry, particularly in relation to creating systemic bias?
    • When is expert adjudication necessary to confirm that the recorded data are reliable and/or reasonably complete?
  • What information is needed to answer such questions?
10:40 a.m. BREAK (Workgroup Participants Gather to Synthesize Audience Feedback)
11:00 a.m. Workgroup Presents Synthesis of Audience Feedback

Session II: When Can We Rely on Real-World Treatment?

Session Discussion Questions:

  • When conducting research in a real-world setting, are there situations that would require special guidance, knowledge, or experience in order for clinicians to adequately monitor participant safety and respond appropriately to adverse events?
  • When does variation between comparison groups (socioeconomic, demographic, etc.); in treatment fidelity; in provider behavior and preferences; or in adherence yield a valid signal about real-world effectiveness, and when is it just noise?
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Moderator: Khaled Sarsour, Genentech
Session Discussants

MICHAEL HORBERG

Executive Director, Research, Community Benefit, and Medicaid Strategy

Executive Director, Mid-Atlantic Permanente Research Institute

Kaiser Permanente Mid-Atlantic Permanente Medical Group

GREGORY SIMON

Investigator

Kaiser Permanente Washington Health Research Institute

ROBERT TEMPLE

Deputy Director for Clinical Science

Center for Drug Evaluation and Research

U.S. Food and Drug Administration

11:15 a.m. Introduction and Presentation to Inform Discussion on Participant Monitoring: Study on Lithium for Suicidal Behavior in Mood Disorders
IRA KATZ

Senior Consultant for Program Evaluation

U.S. Department of Veterans Affairs Office of Mental Health and Suicide Prevention

11:35 a.m. Open Discussion with Audience
  • What conditions make self-monitoring and reporting acceptable?
  • Does this vary for treatments at different stages of product development or with different baseline knowledge about use in varied patient types and treatment conditions?
  • Can we draw any generalizable lessons about cases in which self-monitoring is acceptable and safe?
12:15 p.m. Introduction and Presentation to Inform Discussion on Signal Detection: Novel Oral Anticoagulants in Comparison with Warfarin
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
12:30 p.m. Open Discussion with Audience
  • What conditions and training prepare clinical care providers to monitor patient safety outside a tightly controlled environment?
  • How does this vary for treatments at different stages of product development or with different baseline knowledge about use in varied patient types and treatment conditions?
  • How do you decide which variables require strict adherence to “protocol” and which can be allowed to vary?
1:00 p.m. BREAK (Lunch Available Outside Room 120)

(Workgroup participants gather to synthesize audience feedback)

2:00 p.m. Workgroup Presents Synthesis of Audience Feedback

Session III: When Can We Learn from Real-World Treatment Assignment

Session Discussion Questions:

  • When can we have confidence in inference from cluster-randomized or stepped-wedge study designs?
  • Under what conditions can we trust inference from observational or naturalistic comparisons?
  • How could we judge the validity of observational comparisons in advance, rather than waiting until we have observed the result?
Moderator: Richard Platt, Harvard Medical School
Session Discussants

ROBERT CALIFF

Vice Chancellor, Health Data Science, Duke University Scientific Advisor, Verily Life Sciences

DAVID MADIGAN

Professor of Statistics

Dean, Faculty of Arts and Sciences

Columbia University

DAVID MARTIN

Associate Director for Real-World Evidence Analytics

U.S. Food and Drug Administration

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
2:20 p.m. Introduction and Presentation to Inform the Discussion: Health Care Database Analyses of Medical Products for Regulatory Decision Making
SEBASTIAN SCHNEEWEISS

Professor of Medicine and Epidemiology

Harvard Medical School

Brigham & Women’s Hospital

2:50 p.m. Open Discussion with Audience
  • Random assignment is always preferable, but when is the cost (in time, money, infrastructure, patient exposure) truly necessary?
  • How can we know that the effects from unmeasured confounders are not so large that they would change a decision based on information from an observational study?
  • What are some of the conditions under which there is more confidence in inference from non-randomized comparisons (examples of some conditions below)?
    • Expectation of large effects
    • Outcome proximal to treatment
    • High degree of similarity between comparison groups
    • Pathway from treatment to outcome is relatively clear, and without lots of complexity or reciprocal effects
    • Treatment allocation method is relatively transparent
3:40 p.m. BREAK
4:00 p.m. Open Discussion with Audience and Reflections from Panelists
5:00 p.m. ADJOURN WORKSHOP DAY 1
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×

DAY 2: MARCH 7, 2018

8:30 a.m. Breakfast Available Outside Room 120

Session IV: Synthesizing the Use Cases

Session Objectives:

  • Discuss key considerations presented in each session on Day 1
  • Consider components of a potential “checklist” for using real-world evidence
9:00 a.m. Welcome and Recap of Day 1
MARK MCCLELLAN, Workshop Series Co-Chair

Director

Duke-Margolis Center for Health Policy

GREGORY SIMON, Workshop Series Co-Chair

Investigator

Kaiser Permanente Washington Health Research Institute

9:20 a.m. Open Discussion with Audience of Outputs from Day 1 and Potential Components to a “Checklist” for Using Real-World Evidence
10:40 a.m. BREAK
11:00 a.m. Open Discussion with Audience of Outputs from Day 1 and Potential Components to a “Checklist” for Using Real-World Evidence
12:30 p.m. ADJOURN WORKSHOP DAY 2
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×

Future Workshop Objectives

WORKSHOP THREE: Examine and suggest approaches for operationalizing the collection and use of real-world evidence (July 17–18, 2018, Washington, DC)

  • Applications for using real-world evidence to supplement traditional clinical trials, pragmatic/effectiveness trials, or routine clinical application.
  • Mechanisms for determining which discrete types of real-world evidence could support regulatory decisions.
  • Operational challenges and barriers for generating and incorporating real-world evidence in the context of a learning health system and how clinicians can best be involved in the collection and usage/utilization of real-world evidence.
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 191
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 192
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 193
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 194
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 195
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 196
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 197
Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 198
Next: Appendix D: Workshop Three Agenda »
Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series Get This Book
×
Buy Paperback | $65.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Randomized controlled trials (RCTs) have traditionally served as the gold standard for generating evidence about medical interventions. However, RCTs have inherent limitations and may not reflect the use of medical products in the real world. Additionally, RCTs are expensive, time consuming, and cannot answer all questions about a product or intervention. Evidence generated from real-world use, such as real-world evidence (RWE) may provide valuable information, alongside RCTs, to inform medical product decision making.

To explore the potential for using RWE in medical product decision making, the National Academies of Sciences, Engineering, and Medicine planned a three-part workshop series. The series was designed to examine the current system of evidence generation and its limitations, to identify when and why RWE may be an appropriate type of evidence on which to base decisions, to learn from successful initiatives that have incorporated RWE, and to describe barriers that prevent RWE from being used to its full potential. This publication summarizes the discussions from the entire workshop series.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!