Clinical research is changing, although perhaps not fast enough to meet the challenges and seize the opportunities presented. The constantly increasing diversity and sophistication of health care interventions hold great promise to provide gains in health but also raise substantial challenges to the pace and nature of research on the effectiveness of treatments. New tools are also emerging, however. These tools have the potential to accelerate the research process and to tailor it more to the question being asked, to allow, in effect, a diverse, portfolio-based approach to clinical research that applies the most appropriate methods, given the specific requirements of the situation. This approach includes the conduct of randomized controlled trials (RCTs) as well as the leveraging of the information collected in the process of delivering care through observational studies to drive processes for continuous improvement, which is at the heart of a learning health system.
All research methods have advantages and disadvantages; therefore, the role of specific methods in contributing to a learning health system varies according to the framing of the questions being asked and the context in which the research is being carried out. Although RCTs have strong internal validity, their use of well-defined test and control populations limits their applicability to patients in the real world, who often have characteristics
1 The planning committee’s role was limited to planning the workshop, and the workshop summary has been prepared by the workshop rapporteurs as a factual summary of what occurred at the workshop. Statements, recommendations, and opinions expressed are those of individual presenters and participants and are not necessarily endorsed or verified by the Institute of Medicine, and they should not be construed as reflecting any group consensus.
such as comorbidities that would disqualify them from most RCTs. In addition, because of their extended timelines and costs, which can run in the range of $300 million to $600 million (IOM, 2009), RCTs are an impractical approach to address many important questions.
Observational studies face issues of bias, but when they are used correctly, they can provide information on the effectiveness of therapies in real-world clinical practice. Observational studies can detect signals about the benefits and risks of various therapies in the general population, identify rare side effects and benefits that are beyond the reach of RCTs, and provide community-level data that can lead to new hypotheses that can then be tested in clinical trials. In addition, observational studies can be used in conjunction with RCTs to test the external validity of the RCTs by expanding the clinical settings to a more representative population and to assess the heterogeneity of the treatment response. These approaches will not replace RCTs but can complement them in building the body of evidence on which health decisions can be made.
Today, the most rapidly growing resource for scientific progress in health and health care is the nation’s clinical data infrastructure. The increased adoption of electronic health records, with 44 percent of hospitals and 40 percent of physicians’ offices having at least a basic system (Robert Wood Johnson Foundation, 2013), and the proliferation of mobile sensors and devices, such as the FitBit activity tracker and wireless-enabled scales, have created a wealth of health data that can serve as a resource for learning. This resource, which can be thought of as a new form of public utility, coupled with advances in scientific and statistical methods, makes an examination of the role of observational studies in a learning health system timely.
The purpose of the workshop described here was to identify the leading approaches to observational studies, chart the course for the use of this growing utility, and guide and grow their use in the most responsible fashion possible.
An effective and efficient health care system requires a continually evolving evidence base to guide clinical decisions at the patient level and policy decisions at the level of the population. To meet this need, the methods used must be rigorous and the evidence must be valid and generalizable. Approaches to generating the kind of evidence needed to guide decisions will vary on the basis of the question asked, access to data or research subjects, the availability of resources, and the ultimate use of the results. When the evidence needs of the health system as a whole are considered,
the need for a diversity of approaches to match the wide array of situations and needs is clear.
Clinical research methods can be differentiated in several different ways, with the most fundamental differentiator being whether an approach is observational or experimental. This distinction is defined primarily by whether randomization is employed. Randomization can confer protection from certain biases but can involve logistical and even ethical challenges. Methods that do not employ randomization and that rely on data collected as part of other processes, such as the delivery of care, face analytical challenges but are theoretically easier to carry out and are more likely to produce results that are representative of the environment in which they are being used to inform decision making.
Given this differentiation, several challenges and opportunities are associated with the use of observational studies to generate evidence and inform decision making. The challenges include biases related to the measurement methods used, the population selected for study, the time available for the study, and confounding by medical indication. Because interventions are not randomly assigned in observational studies and the study environment is not tightly controlled, causal relationships are harder to draw. However, innovations in statistical methods, such as instrumental variable (IV) and propensity scores, have allowed progress in addressing these challenges to be made. Data quality, in particular, for studies done with data not collected for research purposes, is an additional challenge.
Observational studies also provide opportunities for clinical research. They require fewer resources and in some cases require the collection of minimal to no additional data beyond those that are routinely collected, and they can often be done more expeditiously than clinical trials. Because observational studies require minimal modification of routine processes, they can provide insight into real-world processes and effects that may more closely mirror those in the decision-making environment in which their results are used. In addition, the broader inclusiveness of observational studies means that the population studied is more likely to mirror the population of patients whose care their findings can inform.
One area of promise for observational studies is provision of a better understanding of heterogeneity in the responses to treatment and the effects of treatment. Although attempts to obtain an understanding of this heterogeneity were made through the analysis of subgroups of individuals participating in RCTs, the size of observational studies provides a greater power to detect and understand heterogeneity among subsets of a larger study population. Finally, data collected in the context of routine care or for observational studies can be used to develop predictive models to, for example, help clinicians and patients make health care decisions on the basis of data from patients who are the most like the patients being treated.
Since its founding at the Institute of Medicine in 2006, the Roundtable on Value & Science-Driven Health Care has brought together leaders from throughout the health system to accelerate the development of a continuously learning health system. A learning health system is one in which science, informatics, incentives, and the culture of the health care system are aligned to create a continuous learning loop. In a learning health care system, evidence and best practices are embedded in health and health care services and new knowledge is routinely captured as a by-product from each interaction with the system (see Figure 1-1). Tocaptured achieve this ambitious goal, the Roundtable convenes meetings of key leaders in health care, holds public workshops, stewards collaborative projects that advance a learning system, and authors reports and related publications.
During the past 7 years, the Roundtable has produced 14 publications, including this one, in its Learning Health System series. The topics covered
FIGURE 1-1 Schematic of a learning health system.
SOURCE: IOM, 2012.
in these publications span a number of the elements necessary for transformation of the system, including clinical research, the digital infrastructure, engaging patients and the public, focusing on value and financial incentives, and applying lessons from other industries to health and health care. The publications have explored stakeholder perspectives on each issue and have identified priorities for advancement and areas in need of collaborative action.
Another vehicle for the Roundtable’s work is a series of Innovation Collaboratives in which key leaders in health and health care participate in collaborative activities that advance the science and increase the value of the health system. The Innovation Collaboratives currently focus on six overlapping and complementary areas: clinical effectiveness research, digital infrastructure, best practices, evidence communication, value, and systems approaches to improving health. These Collaboratives foster information sharing and cooperation across the health and health care system, explore emerging issues facing particular sectors of the health system, and harness the talent and expertise of the participants in practical efforts to advance the field.
The workshop described in this publication was designed to initiate a comprehensive evaluation of the role of observational studies in contributing to the body of evidence for decision making in a learning health system (see Box 1-1). In setting the workshop’s agenda, the planning committee intended individual participants to focus on
• Exploring the role of observational studies in the generation of evidence to guide clinical and health policy decisions in a learning health system with a focus on the care of the individual patient;
• Considering concepts of observational study design and analysis, emerging statistical methods, the opportunities and challenges of observational studies to supplement evidence from experimental methods, identify the heterogeneity of treatments, and provide estimates of effectiveness tailored to individual patients;
• Identifying stakeholder needs and the barriers hindering the broader application of evidence generated from observational studies to decision making;
• Engaging colleagues from disciplines typically underrepresented in discussions of clinical evidence; and
• Suggesting strategies for accelerating progress in the appropriate use of observational studies for the generation of evidence.
Statement of Task
An ad hoc committee will plan and guide the development of a 2-day public workshop to identify and explore issues, attitudes, and approaches to engaging expert stakeholders in exploring the role of observational studies in patient-centered and clinical effectiveness research. The purpose of the workshop is to initiate a comprehensive evaluation of the complementary roles of randomized controlled trials (RCTs) and observational studies and the use of patient-reported data, while looking ahead to the potential of very large sets of data from observational studies to transform the evidence generation needs of a continuously learning health care system. The committee will steer development of the agenda for the workshop, including the selection and invitation of speakers and discussants, and will moderate the discussions. The discussions will highlight fundamental questions defining real-world design; will discuss appropriate analytical approaches for a spectrum of studies, including those RCTs that closely resemble observational studies; and will consider policies, strategies, and procedures for data collection. The heterogeneity of patient responses to treatment will also be considered as it relates to the development of guidelines for individualized clinical care.
Through a series of expert presentations and discussions, workshop participants addressed how observational studies can be made to be more rigorous and internally valid, how to deal with bias, the use of observational studies to generalize the findings from RCTs to broader populations, and the prospects for the use of data from observational studies to evaluate treatment heterogeneity. In addition, presentations at the workshop considered whether observational studies have a role in predicting the response of individuals to treatment, which lies at the heart of personalized medicine. The final workshop session elicited perspectives from speakers with a variety of backgrounds and workshop participants on what the strategies going forward should be.
This publication summarizes the discussions that occurred throughout the workshop, highlighting the key lessons presented, practical strategies, and the needs and opportunities for the use of observational studies in conjunction with RCTs in the context of a learning health care system. Chapter 2 discusses the role that observational studies can play in patient-centered outcomes research. Chapter 3 considers bias, Chapter 4 addresses key issues involved in the generalization of the results of RCTs to the
broader population, and Chapter 5 highlights the role of observational studies in detecting the heterogeneous effects of treatment. Chapter 6 considers the prediction of individual treatment responses, and Chapter 7 discusses some of the common themes that emerged from the workshop discussions and strategies going forward.
IOM (Institute of Medicine). 2009. Transforming Clinical Research in the United States: Challenges and Opportunities: Workshop Summary . Washington, DC: The National Academies Press.
IOM. 2012. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America . Washington, DC: The National Academies Press.
Robert Wood Johnson Foundation. 2013. Health Information Technology in the United States: Better Information Systems for Better Care . Princeton, NJ: Robert Wood Johnson Foundation. http://www.rwjf.org/content/dam/farm/reports/reports/2013/rwjf406758 (accessed June 1, 2013).