Examining the Impact of Real-World Evidence on Medical Product Development Proceedings of a Workshop Series (2019) / Chapter Skim
Currently Skimming:
9 Gaining Confidence in Observational Comparisons
9 Gaining Confidence in Observational Comparisons
Pages 133-152
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 133... ...
• Replicating randomized clinical trial results with observational databases can help establish a process and criteria for conduct ing observational studies more widely. (Franklin)
|
From page 134... ...
Throughout the presentations and discussions, workshop participants also heard an alternative perspective from some speakers who emphasized the continuing importance of randomization and discussed methods to make randomization easier in real-world settings. ILLUSTRATIVE EXAMPLES To explore the issues surrounding bias in observational comparisons, speakers at the second and third workshops presented case studies as illustrative examples of the considerations that go into designing and conducting observational data analyses.
|
From page 135... ...
SOURCES: Schneeweiss presentation, March 6, 2018; Franklin and Schneeweiss, 2017. gories: electronic health records, laboratory results, and administrative data such as insurance claims.
|
From page 136... ...
Schneeweiss emphasized that "sweat and tears" go into the design of an RCT, but the actual analysis is quite straightforward compared to the complex analytics used in database studies. Despite these advantages of RCTs, Schneeweiss offered his perspective on when a researcher might feel comfortable conducting a database study instead: • First, an active comparator is preferred.
|
From page 137... ...
• Is the data analysis plan based on epidemiologic study design principles? • Was balance in confounding factors among treatment groups achieved?
|
From page 138... ...
The researchers, said Izurieta, used Cox regression models to estimate the risks of herpes zoster and postherpetic neuralgia in the vaccinated and unvaccinated populations, adjusted for the main known characteristics, and measured the risk at different time intervals because vaccine protection varies over time. After this basic overview of the study, Izurieta went through the draft decision aid (see Figure 9-5 later in this chapter)
|
From page 139... ...
that combines propensity scores and Mahalanobis metric matching. This approach, said Izurieta, allowed adjustment for heterogeneity between variables and controls, using a broad list of covariates that are plausibly related to herpes zoster, while generating cohorts that are closely matched on a subset of key covariates.
|
From page 140... ...
FIGURE 9-4 Postmatched herpes zoster vaccinated and unvaccinated cohorts. SOURCE: Izurieta presentation, July 17, 2018.
|
From page 141... ...
The Medicare Current Beneficiary Survey is distributed to a random sample of beneficiaries every year, and asks questions regarding health usage, vaccine history, education level, and frailty. The researchers compared approximately 900 herpes zoster vaccine recipients with about 900 non-vaccinated individuals from the survey respondents, and used multiple imputation to reanalyze the vaccine effectiveness after linking the data to the survey.
|
From page 142... ...
Paul Rosenbaum (Rosenbaum, 2010) has extended this idea further, said Madigan, by attempting to account not just for known factors that affect probability of exposure, but also for the unknown factors that are not included in the propensity score.
|
From page 143... ...
Although results in the published literature may point to a certain finding, Califf said, "We have no idea how many things were looked at that were never published." Observational data can now be analyzed with the push of a button because of automated programs; this ease of analysis increases the chances that the published results are a highly selective sample of all of the results compared with clinical trials. He added that there is a need for principles and systems to address this issue for observational studies.
|
From page 144... ...
The second would be to allow analyses to be unlimited, but require that all analyses be reported and compared. Schneeweiss disagreed with this second approach, saying that "you just get a mess of data and you have no idea how to interpret it." Several other individual speakers emphasized the importance of prespecification of the analytic plan; Mark van der Laan, professor of biostatistics and statistics at the University of California, Berkeley, called the problem of multiple unplanned analyses "the biggest problem in observational studies." Reproducibility Several individual participants noted problems related to non- reproducibility of database studies.
|
From page 145... ...
Predictive Analytics and Machine Learning Javier Jimenez, vice president and global head for Real-World Evidence and Clinical Outcomes at Sanofi, suggested that new tools such as machine learning and predictive modeling may be useful for analyzing RWD. In addition to other uses, machine learning and predictive modeling present an opportunity to evaluate unmeasured confounders through proxies from other information that has been collected, he said.
|
From page 146... ...
Provide two tables that report covariate balance before and after matching or weighting, respectively. After matching or weighting for balance, do the analytic cohorts appear to represent clinically meaningful groups for study (e.g., has utility or generalizability been sacrificed)
|
From page 147... ...
DISCUSSION DRAFT FIGURE 9-5 Decision aid on questions to consider to assess and minimize bias in non-randomized observational comparisons. NOTE: This decision aid was drafted by some individual workshop participants based on the discussions of individual workshop participants at the first and second workshops in the real-world evidence series.
|
From page 148... ...
This model, said Jimenez, could be used to predict the probability of a specific outcome for a patient, using all of the available information. van der Laan outlined another modern approach for conducting observational studies using computer systems to learn from data, known as "targeted machine learning." This approach, he said, is always based on a roadmap of causal inference, with defined steps that are followed for every analysis: • The first step is understanding the question of interest from a causal perspective -- that is, what are the outcomes, interventions, and other variables of interest -- and developing a causal model describ ing the causal relations among the variables.
|
From page 149... ...
van der Laan described an approach called "targeted learning," which combines causal modeling, state-of-the-art machine learning, and deep statistical learning to get more precise answers for causal questions of interest, while providing formal statistical inference in terms of confidence intervals and p-values. Targeted learning is a technique to minimize estimation bias and to maximize precision in observational studies.
|
From page 150... ...
Statisticians at CDRH recently developed a streamlined procedure for designing premarket observational studies, said Li. This procedure uses propensity score methodology to balance baseline covariates, and uses an "outcome-free" design principle.
|
From page 151... ...
For example, she pointed to Schneeweiss's presentation about which characteristics lead to a more valid observational study: an active comparator, a new user design, well-specified outcomes, and data sources with good longitudinal exposure measurement. In addition to these, propensity scores and sensitivity analyses can be useful, she said.
|
From page 152... ...
For example, in RCTs, hypotheses must be registered, and if a hypothesis changes, this needs to be reported. Berger advocated for similar requirements and practices for observational studies: "We need to bring observational data up to the same level of scrutiny as we have for RCTs" before discussing when and whether RWD are fit for purpose.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.