The strengths and weaknesses of observational studies and randomized trials are reviewed. Also reviewed are well-recognized limitations of observational studies due to the potential for confounding by a variety of factors as well as their limited capacity to assess short-term or acute risks. Although randomized controlled trials (RCTs) have the advantage of minimizing confounding, RCTs are often constrained by higher costs, shorter duration of follow-up, and limited applicability to populations of greatest clinical relevance. However, mixed experiences with different investigative approaches do not argue for total cessation of any one approach in favor of another. Rather, as the authors in this chapter suggest, the research community needs to be more receptive to the use of alternative methodologies to generate insights into clinical effectiveness, and we need to determine which approach we use for a given question with full recognition of what is right for particular research circumstances. Collectively these experiences suggest the availability of a powerful array of methods, and when results are combined they produce more nuanced information needed to guide treatment decisions. Opportunities to strengthen these methods are discussed and, overall, greater attention is needed to define state-of-the-art methods so the quality of research is readily discernible regardless of study approach. In addition to methods, data and data system improvements are needed. Electronic health records and data registry approaches offer the opportunity to better systematically capture, track, and report outcomes. Moreover, there is the suggestion that a mix of research approaches, using the best advantages of particular designs, offers untapped promise and that researchers should be more open to adopting such approaches. Greater engagement by the healthcare system is imperative in the evaluation of effectiveness.
JoAnn E. Manson from Harvard Medical School reviews the divergent results of observational studies and RCTs, evaluating the effect of menopausal hormone replacement therapy (HRT) on coronary heart disease (CHD). Despite this divergence, both have contributed critically important information on the therapies’ effectiveness and implications for healthcare decision making. Building on this experience, Manson discusses factors that might have contributed to the different findings. She suggests that because the short- and long-term effects of a clinical intervention may differ, both observational studies and clinical trial design must have benefits to offer researchers. Perhaps, she says, we should consider research findings in the context of all of the available evidence and design studies to complement and extend existing data. Large-scale studies involving networks of electronic databases could facilitate evidence development. Due to the high cost and generally short duration of clinical trials, information about long-term risk may rely heavily on observational sources.
The Food and Drug Administration’s (FDA’s) Ashley B. Boam recounts