Appendix C
Workshop Two Agenda
Workshop Two: Practical Approaches
March 6–7, 2018
National Academy of Sciences Building, Room 120
2101 Constitution Avenue, NW, Washington, DC 20418
The National Academies of Sciences, Engineering, and Medicine is convening a three-part workshop series examining how real-world evidence development and uptake can enhance medical product development and evaluation. The workshops will advance discussions and common knowledge about complex issues relating to the generation and usage of real-world evidence, including fostering development and implementation of the science and technology of real-world evidence generation and usage.
- Workshop One (September 19–20, 2017) focused on how to align incentives to support collection and use of real-world evidence in health product review, payment, and delivery. Incentives need to address barriers impeding the uptake of real-world evidence, including barriers to transparency.
- Workshop Two (March 6–7, 2018) will illuminate what types of data are appropriate for what specific purposes and suggest practical approaches for data collection and evidence use by developing and working through example use cases.
- Workshop Three (July 17–18, 2018) will examine and suggest approaches for operationalizing the collection and use of real-world evidence.
DAY 1: MARCH 6, 2018
8:30 a.m. | Breakfast Available Outside Room 120 |
8:40 a.m. | Welcome and Opening Remarks |
GREGORY SIMON, Workshop Series Co-Chair
Investigator Kaiser Permanente Washington Health Research Institute |
Session I: When Can We Rely on Real-World Data?
Session Discussion Questions:
- When can we have confidence in EHR data from real-world practice to accurately assess study eligibility, key prognostic factors, and study outcomes?
- When can we have confidence in data generated outside of clinical settings (e.g., mobile phones, connected glucometers, connected blood pressure monitors)?
- When does adjudication or other postprocessing of real-world data add value?
Moderator: | Gregory Daniel, Duke-Margolis Center for Health Policy |
Session Discussants
JESSE BERLIN Vice President and Global Head, Epidemiology Johnson & Johnson |
|
ANDY BINDMAN
Professor of Medicine University of California, San Francisco |
|
9:00 a.m. | Introduction and Background to Inform the Discussion: Novel Oral Anticoagulants in Comparison with Warfarin |
ADRIAN HERNANDEZ
Vice Dean for Clinical Research Duke University School of Medicine |
9:20 a.m. |
Open Discussion with Audience
|
10:40 a.m. | BREAK (Workgroup Participants Gather to Synthesize Audience Feedback) |
11:00 a.m. | Workgroup Presents Synthesis of Audience Feedback |
Session II: When Can We Rely on Real-World Treatment?
Session Discussion Questions:
- When conducting research in a real-world setting, are there situations that would require special guidance, knowledge, or experience in order for clinicians to adequately monitor participant safety and respond appropriately to adverse events?
- When does variation between comparison groups (socioeconomic, demographic, etc.); in treatment fidelity; in provider behavior and preferences; or in adherence yield a valid signal about real-world effectiveness, and when is it just noise?
Moderator: | Khaled Sarsour, Genentech |
Session Discussants
MICHAEL HORBERG Executive Director, Research, Community Benefit, and Medicaid Strategy Executive Director, Mid-Atlantic Permanente Research Institute Kaiser Permanente Mid-Atlantic Permanente Medical Group |
|
GREGORY SIMON
Investigator Kaiser Permanente Washington Health Research Institute |
|
ROBERT TEMPLE
Deputy Director for Clinical Science Center for Drug Evaluation and Research U.S. Food and Drug Administration |
|
11:15 a.m. | Introduction and Presentation to Inform Discussion on Participant Monitoring: Study on Lithium for Suicidal Behavior in Mood Disorders |
IRA KATZ
Senior Consultant for Program Evaluation U.S. Department of Veterans Affairs Office of Mental Health and Suicide Prevention |
|
11:35 a.m. |
Open Discussion with Audience
|
12:15 p.m. | Introduction and Presentation to Inform Discussion on Signal Detection: Novel Oral Anticoagulants in Comparison with Warfarin |
12:30 p.m. |
Open Discussion with Audience
|
1:00 p.m. |
BREAK (Lunch Available Outside Room 120)
(Workgroup participants gather to synthesize audience feedback) |
2:00 p.m. | Workgroup Presents Synthesis of Audience Feedback |
Session III: When Can We Learn from Real-World Treatment Assignment
Session Discussion Questions:
- When can we have confidence in inference from cluster-randomized or stepped-wedge study designs?
- Under what conditions can we trust inference from observational or naturalistic comparisons?
- How could we judge the validity of observational comparisons in advance, rather than waiting until we have observed the result?
Moderator: | Richard Platt, Harvard Medical School |
Session Discussants
ROBERT CALIFF Vice Chancellor, Health Data Science, Duke University Scientific Advisor, Verily Life Sciences |
|
DAVID MADIGAN
Professor of Statistics Dean, Faculty of Arts and Sciences Columbia University |
|
DAVID MARTIN
Associate Director for Real-World Evidence Analytics U.S. Food and Drug Administration |
2:20 p.m. | Introduction and Presentation to Inform the Discussion: Health Care Database Analyses of Medical Products for Regulatory Decision Making |
SEBASTIAN SCHNEEWEISS
Professor of Medicine and Epidemiology Harvard Medical School Brigham & Women’s Hospital |
|
2:50 p.m. |
Open Discussion with Audience
|
3:40 p.m. | BREAK |
4:00 p.m. | Open Discussion with Audience and Reflections from Panelists |
5:00 p.m. | ADJOURN WORKSHOP DAY 1 |
DAY 2: MARCH 7, 2018
8:30 a.m. | Breakfast Available Outside Room 120 |
Session IV: Synthesizing the Use Cases
Session Objectives:
- Discuss key considerations presented in each session on Day 1
- Consider components of a potential “checklist” for using real-world evidence
9:00 a.m. | Welcome and Recap of Day 1 |
MARK MCCLELLAN, Workshop Series Co-Chair
Director Duke-Margolis Center for Health Policy |
|
GREGORY SIMON, Workshop Series Co-Chair
Investigator Kaiser Permanente Washington Health Research Institute |
|
9:20 a.m. | Open Discussion with Audience of Outputs from Day 1 and Potential Components to a “Checklist” for Using Real-World Evidence |
10:40 a.m. | BREAK |
11:00 a.m. | Open Discussion with Audience of Outputs from Day 1 and Potential Components to a “Checklist” for Using Real-World Evidence |
12:30 p.m. | ADJOURN WORKSHOP DAY 2 |
Future Workshop Objectives
WORKSHOP THREE: Examine and suggest approaches for operationalizing the collection and use of real-world evidence (July 17–18, 2018, Washington, DC)
- Applications for using real-world evidence to supplement traditional clinical trials, pragmatic/effectiveness trials, or routine clinical application.
- Mechanisms for determining which discrete types of real-world evidence could support regulatory decisions.
- Operational challenges and barriers for generating and incorporating real-world evidence in the context of a learning health system and how clinicians can best be involved in the collection and usage/utilization of real-world evidence.