Redesigning the Clinical Effectiveness Research Paradigm: Innovation and Practice-Based Approaches: Workshop Summary (2010)

Clinical effectiveness research (CER) serves as the bridge between the development of innovative treatments and therapies and their productive application to improve human health. Building on efficacy and safety determinations necessary for regulatory approval, the results of these investigations guide the delivery of appropriate care to individual patients. As the complexity, number, and diversity of treatment options grow, the provision of clinical effectiveness information is increasingly essential for a safe and efficient healthcare system. Currently, the rapid expansion in scientific knowledge is inefficiently translated from scientific lab to clinical practice (Balas and Boren, 2000; McGlynn, 2003). Limited resources play a part in this problem. Of our nation’s more than $2 trillion investment in health care, an estimated less than 0.1 percent is devoted to evaluating the relative effectiveness of the various diagnostics, procedures, devices, pharmaceuticals, and other interventions in clinical practice (AcademyHealth, 2005; Moses et al., 2005).

The problem is not merely a question of resources but also of the way they are used. With the information and practice demands at hand, and new tools in the works, a more practical and reliable clinical effectiveness research paradigm is needed. Information relevant to guiding decision making in clinical practice requires the assessment of a broad range of research

questions (e.g., how, when, for whom, and in what settings are treatments best used?), yet the current research paradigm, based on a hierarchical arrangement of study designs, assigns greater weight or strength to evidence produced from methods higher in the hierarchy, without necessarily considering the appropriateness of the design for the particular question under investigation. For example, the advantages of strong internal validity, a key characteristic of the randomized controlled trial (RCT)—long considered the gold standard in clinical research—are often muted by constraints in time, cost, and limited external validity or applicability of results. And, although the scientific value of well-designed clinical trials has been demonstrated, for certain research questions, this approach is not feasible, ethical, or practical and may not yield the answer needed. Similarly, issues of bias and confounding inherent to observational, simulation, and quasi-experimental approaches may limit their use and enhancement, even for situations and circumstances requiring a greater emphasis on external validity.

Especially given the growing capacity of information technology to capture, store, and use vastly larger amounts of clinically rich data and the importance of improved understanding of an intervention’s effect in real-world practice, the advantages of identifying and advancing methods and strategies that draw research closer to practice become even clearer.

Against the backdrop of the growing scope and scale of evidence needs, limits of current approaches, and potential of emerging data resources, the Institute of Medicine (IOM) Roundtable on Evidence-Based Medicine, now the Roundtable on Value & Science-Driven Health Care convened the Redesigning the Clinical Effectiveness Research Paradigm: Innovation and Practice-Based Approaches workshop. The issues motivating the meeting’s discussions are noted in Box S-1, the first of which is the need for a deeper and broader evidence base for improved clinical decision making. But also important are the needs to improve the efficiency and applicability of the process. Underscoring the timeliness of the discussion is recognition of the challenges presented by the expense, time, and limited generalizability of current approaches, as well as of the opportunities presented by innovative research approaches and broader use of electronic health records that make clinical data more accessible. The overall goal of the meeting was to explore these issues, identify potential approaches, and discuss possible strategies for their engagement.

Participants examined ways to expedite the development of clinical effectiveness information, highlighting the opportunities presented by innovative study designs and new methods of analysis and modeling; the size and expansion of potentially interoperable administrative and clinical datasets; and emerging research networks and data resources. The presentations and discussion emphasized approaches to research and learning that had the potential to supplement, complement, or supersede RCT findings and

suggested opportunities to engage these tools and methods as a new generation of studies that better address current challenges in clinical effectiveness research. Consideration also was given to the policies and infrastructure needed to take greater advantage of existing research capacity.

Starting points for the workshop’s discussion reside in the presentation of what has come to be viewed as the traditional clinical research model, depicted as a pyramid in Figure S-1. In this model, the strongest level of evidence is displayed at the peak of the pyramid: the randomized controlled double blind study. This is often referred to as the “gold standard” of clinical research, and is followed, in a descending sequence of strength or quality, by randomized controlled studies, cohort studies, case control studies, case series, and case reports. The base of the pyramid, the weakest evidence, is reserved for undocumented experience, ideas, and opinions. A brief overview of the range of clinical effectiveness research methods is presented in Table S-1. Approaches are categorized into two groups: experimental and nonexperimental. Experimental studies are those in which the choice and assignment of the intervention is under control of the investigator; the results of a test intervention are compared to the results of an alternative approach by actively monitoring the respective experiences of either individuals or groups receiving or not receiving the intervention. Nonexperimental studies are those in which manipulation or randomiza-

FIGURE S-1 The classic evidence hierarchy.

SOURCE: DeVoto, E., and B. S. Kramer. 2005. Evidence-Based Approach to Oncology. In Oncology an Evidence-Based Approach. Edited by A. Chang. New York: Springer. Modified and reprinted with permission of Springer SBM.

tion is generally absent, the choice of an intervention is made in the course of clinical care, and existing data collected in the course of the care process are used to draw conclusions about the relative impact of different circumstances or interventions that vary between and among identified groups, or to construct mathematical models that seek to predict the likelihood of events in the future based on variables identified in previous studies.

Noted at the workshop was the fact that, as currently practiced, the randomized controlled and blinded trial is not the gold standard for every circumstance. While not an exhaustive catalog of methods, Table S-1 provides a sense of the range of clinical research approaches that can be used to improve understanding of clinical effectiveness. Each method has the potential to advance understanding of the various aspects of the spectrum of questions that emerge throughout a product’s or intervention’s lifecycle in clinical practice. The issue is therefore not whether internal or external validity should be the overarching priority for research, but rather which approach is most appropriate to the particular need. In each case, careful attention to design and execution studies are vital.

Recent methods development, along with the identification of problems in generalizing research results to broader populations than those enrolled in tightly controlled trials, as well as the impressive advances in the potential availability of data through expanded use of electronic health records, have all prompted re-consideration of research strategies and opportuni-

ties (Kravitz, 2004; Liang, 2005; Rush, 2008; Schneeweiss, 2004; Agency for Healthcare Research and Quality [AHRQ] CER methods and registry issues).

This emerging understanding about limitations in the current approach, with respect to both current and future needs and opportunities, sets the stage for the workshop’s discussions.

Formed in 2006 as the Roundtable on Value & Science-Driven Health Care brings together key stakeholders from multiple sectors—patients, health providers, payers, employers, health product developers, policy makers, and researchers—for cooperative consideration of the ways that evidence can be better developed and applied to drive improvements in the effectiveness and efficiency of U.S. medical care. Roundtable participants have set the goal that “by the year 2020, 90 percent of clinical decisions will be supported by accurate, timely, and up-to-date clinical information, and will reflect the best available evidence.” To achieve this goal, Roundtable members and their colleagues identify issues and priorities for cooperative stakeholder engagements. Central to these efforts is the Learning Healthcare System series of workshops and publications. The series collectively characterizes the key elements of a healthcare system that is designed to generate and apply the best evidence for healthcare choices of patients and providers. A related purpose of these meetings is the identification and

engagement of barriers to the development of the learning healthcare system and the key opportunities for progress. Each meeting is summarized in a publication available through The National Academies Press. Workshops in this series include

• The Learning Healthcare System (July 20–21, 2006)

• Judging the Evidence: Standards for Determining Clinical Effectiveness (February 5, 2007)

• Leadership Commitments to Improve Value in Healthcare: Toward Common Ground (July 23–24, 2007)

• Redesigning the Clinical Effectiveness Research Paradigm: Innovation and Practice-Based Approaches (December 12–13, 2007)

• Clinical Data as the Basic Staple of Health Learning: Creating and Protecting a Public Good (February 28–29, 2008)

• Engineering a Learning Healthcare System: A Look to the Future (April 28–29, 2008)

• Learning What Works: Infrastructure Required for Learning Which Care Is Best (July 30–31, 2008)

• Value in Health Care: Accounting for Cost, Quality, Safety, Outcomes and Innovation (November 17–18, 2008)

This publication summarizes the proceedings of the fourth workshop in the Learning Healthcare System series, focused on improving approaches to clinical effectiveness research.

The Roundtable’s work is predicated on the principle that “to the greatest extent possible, the decisions that shape the health and health care of Americans—by patients, providers, payers, and policy makers alike—will be grounded on a reliable evidence base, will account appropriately for individual variation in patient needs, and will support the generation of new insights on clinical effectiveness.” Well-conducted clinical trials have and will continue to contribute to this evidence base. However, the need for research insights is pressing, and as data are increasingly captured at the point of care and larger stores of data are made available for research, exploration is urgently needed on how to best use these data to ensure care is tailored to circumstance and individual variation.

The workshop’s intent was to provide an overview of some of the most promising innovations and approaches to clinical effectiveness research. Opportunities to streamline clinical trials, improve their practical application, and reduce costs were reviewed; however, particular emphasis was placed on reviewing methods that improve our capacity to draw upon data collected at the point of care. Rather than providing a comprehensive review of methods, the discussion in the chapters that follow uses examples

to highlight emerging opportunities for improving our capacity to determine what works best for whom.

A synopsis of key points from each session is included in this chapter; more detailed information on session presentations and discussions can be found in the chapters that follow. Day one of the workshop identified key lessons learned from experience (Chapter 2) and important opportunities presented by new tools and techniques (Chapter 3) and emerging data resources (Chapter 4). Discussion and presentations during day two focused on strategies to better plan, develop, and sequence the studies needed (Chapter 5) and concluded with presentations on opportunities to better align policy with research opportunities and a panel discussion on organizing the research community for change (Chapter 6). Keynote presentations provided overviews of the evolution and opportunities for clinical effectiveness research and provided important context for workshop discussions. These presentations and a synopsis of the workshop discussion are included in Chapter 1. The workshop agenda, biographical sketches of the speakers, and a list of workshop participants can be found in Appendixes A, B, and C, respectively.

The Redesigning the Clinical Effectiveness Research Paradigm workshop featured speakers from a wide range of perspectives and sectors in health care. Although many points of view were represented, certain themes emerged from the 2 days of discussion, as summarized below and in Box S-2²:

• Address current limitations in applicability of research results. Because clinical conditions and their interventions have complex and varying circumstances, there are different implications for the evidence needed, study designs, and the ways lessons are applied: the internal and external validity challenge. In particular given our aging population, often people have multiple conditions—co-morbidities—yet study designs generally focus on people with just one condition, limiting their applicability. In addition, although our assessment of candidate interventions is primarily through premarket studies, the opportunity for discovery extends throughout the lifecycle of an intervention—development, approval, coverage, and the full period of implementation.

• Counter inefficiencies in timeliness, costs, and volume. Much of current clinical effectiveness research has inherent limits and inefficiencies related to time, cost, and volume. Small studies may have insufficient reliability or follow-up. Large experimental studies may be expensive and lengthy but have limited applicability to practice circumstances. Studies sponsored by product manufacturers have to overcome perceived conflicts and may not be fully used. Each incremental unit of research time and money may bring greater confidence but also carries greater opportunity costs. There is a strong need for more systematic approaches to better defying how, when, for whom, and in what setting an intervention is best used.

• Define a more strategic use to the clinical experimental model. Just as there are limits and challenges to observational data, there are limits to the use of experimental data. Challenges related to the scope of possible inferences, to discrepancies in the ability to detect near-term versus long-term events, to the timeliness of our insights and our ability to keep pace with changes in technology and procedures, all must be managed. Part of the strategy challenge is choosing the right tool at the right time. For the future of clinical effectiveness research, the important issues relate not to whether randomized experimental studies are better than observational studies, or vice versa, but to what’s right for the circumstances (clinical and economic) and how the capacity can be systematically improved.

• Provide stimulus to new research designs, tools, and analytics. An exciting part of the advancement process has been the development of new tools and resources that may quicken the pace of our learning and add real value by helping to better target, tailor, and refine approaches. Use of innovative research designs, statistical techniques, probability, and other models may accelerate the timeliness and level of research insights. Some interesting approaches using modeling for virtual intervention studies may hold prospects for revolutionary change in certain clinical outcomes research.

• Encourage innovation in clinical effectiveness research conduct. The kinds of “safe harbor” opportunities that exist in various fields for developing and testing innovative methodologies for addressing complex problems are rarely found in clinical research. Initiative is needed for the research community to challenge and assess its approaches—a sort of meta-experimental strategy—including those related to analyzing large datasets, in order to learn about the purposes best served by different approaches. Innovation is also needed to counter the inefficiencies related to the volume of studies conducted. How might existing research be more systematically summarized or different research methods be organized, phased, or coordinated to add incremental value to existing evidence?

• Promote the notion of effectiveness research as a routine part of practice. Taking full advantage of each clinical experience is the theoretical goal of a learning healthcare system. But for the theory to move closer to the practice, tools and incentives are needed for caregiver engagement. A starting point is with the anchoring of the focus of clinical effectiveness research planning and priority setting on the point of service—the patient–provider interface—as the source of attention, guidance, and involvement on the key questions to engage. The work with patient registries by many specialty groups is an indication of the promise in this respect, but additional emphasis is necessary in anticipation of the access and use of the technology that opens new possibilities.

• Improve access and use of clinical data as a knowledge resource. With the development of bigger and more numerous clinical data sets, the potential exists for larger scale data mining for new insights on the effectiveness of interventions. Taking advantage of the prospects will require improvements in data sharing arrangements and platform compatibilities, addressing issues related to real and perceived barriers from interpretation of privacy and patient protection rules, enhanced access for secondary analysis to federally sponsored clinical data (e.g., Medicare part D, pharmaceutical,

clinical trials), the necessary expertise, and stronger capacity to use clinical data for postmarket surveillance.

• Foster the transformational research potential of information technology. Broad application and linkage of electronic health records hold the potential to foster movement toward real-time clinical effectiveness research that can generate vastly enhanced insights into the performance of interventions, caregivers, institutions, and systems—and how they vary by patient needs and circumstances. Capturing that potential requires working to better understand and foster the progress possible, through full application of electronic health records, developing and applying standards that facilitate interoperability, agreeing on and adhering to research data collection standards by researchers, developing new search strategies for data mining, and investing patients and caregivers as key supporters in learning.

• Engage patients as full partners in the learning culture. With the impact of the information age growing daily, access to up-to-date information by both caregiver and patient changes the state of play in several ways. The patient sometimes has greater time and motivation to access relevant information than the caregiver, and a sharing partnership is to the advantage of both. Taking full advantage of clinical records, even with blinded information, requires a strong level of understanding and support for the work and its importance to improving the quality of health care. This support may be the most important element in the development of the learning enterprise. In addition, the more patients understand and communicate with their caregivers about the evolving nature of evidence, the less disruptive will be the frequency and amplitude of public response to research results that find themselves prematurely, or without appropriate interpretative guidance, in the headlines and the short-term consciousness of Americans.

• Build toward continuous learning in all aspects of care. This foundational principle of a learning healthcare system will depend on system and culture change in each element of the care process with the potential to promote interest, activity, and involvement in the knowledge and evidence development process, from health professions education to care delivery and payment.

Of particular prominence throughout the workshop discussion was the notion of closing the gap between research and practice. Participants emphasized the challenges of ensuring that research is structured to provide

information relevant to real-world decisions faced by patients, providers, and policy makers; ensuring the rigor of the research design and execution; and monitoring the safety and effectiveness of new products, with more attention to point-of-care data.

The multifaceted, practice-oriented approach to clinical effectiveness research discussed at the workshop complements and blends with traditional trial-oriented clinical research and may be represented as a continuum in which evidence is continuously produced by a blend of experimental studies with patient assignment (clinical trials); modeling, statistical, and observational studies without patient assignment; and monitored clinical experience (Figure S-2). The ratio of the different approaches varies with the nature of the intervention, as does the weight given to available studies. This enhanced flexibility and range of research resources is facilitated by the development of innovative design and analytic tools, and by the growing potential of electronic health records to allow much broader and structured access to the results of the clinical experience. The ability to draw on real-time clinical insights will naturally improve over time.

FIGURE S-2 Evidence development in the learning healthcare system.

The workshop presentations and discussions by experts from many areas of health care detailed the current state of clinical effectiveness research, provided examples of promising approaches, and proposed some key challenges and opportunities for improvement. Keynote addresses opened the 2 days of the workshop, previewing and underscoring the conceptual background, issues, and themes. IOM President Harvey V. Fineberg reviewed the evolution of clinical effectiveness research, and Carolyn M. Clancy offered meeting participants a vision for research that is better matched to evidence needs. Workshop discussions and presentations are briefly summarized; expanded discussions are included in the chapters that follow.

In his keynote presentation, Fineberg briefly traced the evolution of clinical effectiveness research. From early efforts such as James Lynd’s evaluation of treatments for scurvy to 20th-century developments in statistics that strengthen scientific studies to the establishment of RCTs as the standard of Food and Drug Administration (FDA) in making judgments about efficacy in the early 1970s, clinical effectiveness research has developed rapidly and has helped to transform medical care. However, Fineberg suggested that the resulting research paradigm, with randomized controlled double blind trials at the pinnacle, has often left important evidence needs unmet when combined with the costs, complexity, and lack of generalizability of RCTs. These gaps in evidence, he noted, prompt a reevaluation of the current application and require movement toward a strategy that takes better advantage of the range of methodologies to develop evidence that meets the particular need.

Important to the redesign of the research paradigm is the consideration of a set of prior questions that could better shape research design and conduct. For example, understanding the purpose of and vantage point from which these clinical questions are being asked helps to put into perspective the roles and contributions of the study designs that might be employed. Thinking critically about what is being evaluated and mapping what is appropriate, effective, and efficient for the various types of questions is an ongoing and important challenge in clinical effectiveness research. These efforts will also combat the central paradox in health care today: Despite the overwhelming abundance of information available, there is an acute shortage of information that is relevant, timely, appropriate, and useful to clinical decision making.

The critically central question for clinical effectiveness research is what works best in clinical care for the individual patient at the time care is needed. Answering this type of question will require the transformation of current

approaches to a system that combines point-of-care focus with an electronic health record (EHR) data system coupled to systems for assembling evidence in a variety of ways. Such a system would combine several components: evaluation of learning what works and what does not work in a patient, weighing benefits and costs; decision support for the patient that would compensate for the overwhelming amount of evidence and make relevant determinations from the information available while increasing the pool of potentially useful information; meaningful continuing education for health professionals—real time, in time, practical, and applied—moving beyond the traditional lecture to learning in place, in real time, in the course of clinical care; and quality improvement systems that synthesize information from the three components above—evaluation, decision support, and meaningful continuing education.

Fineberg proposed a meta-experimental strategy, in which researchers not only focus on how well a certain method evaluates a particular kind of problem, in a specific class of patient, from a particular point of view, but also on determining the array of experimental methods that collectively perform in a manner that enables us to make better decisions for the individual and for society. With this approach, future learning opportunities can be structured to provide insights on what works for a particular kind of patient as well as how that strategy of evaluation can be employed to achieve a health system driven by evidence and based on value.

In the second day’s keynote address, Carolyn M. Clancy, director of the Agency for Healthcare Research and Quality (AHRQ), shared perspectives in two broad areas: emerging methods that might be applied to meet current challenges in clinical effectiveness research and approaches to turning evidence into action. Stressing the importance of not producing better evidence for its own sake, Clancy challenged researchers to focus on the goal of achieving an information-rich, patient-focused system. Building toward a healthcare system in which actionable information is made available to clinicians and patients and in which evidence is continually refined as a byproduct of healthcare delivery will require a broadening of the investigative approaches and methodologies that constitute the research arsenal.

Clancy noted that the traditional evidence hierarchy is being increasingly challenged, in part because it is inadequate to meet the current decision-making needs in health care, prompting calls for a rigorous reassessment of the appropriate roles for randomized and nonrandomized evidence. Recognizing that an intervention can work is a necessary, but not sufficient, requirement for making a treatment decision for an individual patient or for promoting it for a broad population. Even the most rigorously designed randomized trials have limitations, and research methods are needed to

explore critical questions related to important trade-offs between risks and benefits of treatments for individual patients.

Although some circumstances may always require randomized trials, nonrandomized studies can complement and extend insights from RCTs in various ways—for example, tracking longer term outcomes, monitoring replicability in broader populations and community settings, and expanding information on potential benefits and harms of a given intervention. From a practical perspective, these approaches can help to evidence match the pace of rapid change and innovation found, for example, in surgical procedures and medical device development. Promising advances include (1) practical clinical trials, in which trial design is based on information needed to make a clinical decision and conduct is embedded into healthcare delivery systems; and (2) the use of cohort study registries to explore heterogeneity of treatment effects due to setting, practitioner, and patient variation, and consequently to turn evidence into action.

Observational studies offer an alternative when trials are impractical or infeasible and also help to accelerate translation of evidence into practice and aid risk management and minimization efforts. The promotion of more transparent, consistent approaches to the assessment of evidence and increased emphasis on the quality of study design and conduct over the type of method used are trends toward research that fit evidence needs, as is the focus on new and improved research methods. Finally, Clancy emphasized that to ensure research impacts practice, attention also is needed to improve approaches to turning evidence into action, and recent efforts by AHRQ and others underscore the research community’s commitment to the creation of a system focused on the patient and improving health outcomes.

The second chapter summarizes workshop discussions of case examples of high-profile issues—some linked to application of effective treatments or to premature adoption of unwarranted treatments—from which important lessons might be drawn about the design and interpretation of clinical effectiveness studies. The experiences recounted show that, from randomized trials to observational studies, each investigative approach has limitations. These limitations argue against using a particular approach and suggest that the research community needs more experience with the array of methodologies used to generate insights into clinical effectiveness and structured decision rules to guide the study design choice for particular research circumstances. Improvements can be made across the process, including careful consideration of the methods most appropriate to the question being asked; careful development and conduct of trials or studies to ensure they reflect the

“state of the art”; clear communication of results; and exploration of new approaches, such as using a hybrid mix of research approaches.

The first case was presented by JoAnn E. Manson from Harvard Medical School on the impact of hormone replacement therapy (HRT) on health. Both observational studies and clinical trials have contributed critically important information to elucidate the health effects of HRT with estrogen and progestin and to inform decision making, and they constitute a model suggesting that research findings should be considered in the context of the totality of available evidence and that studies should be designed to complement and extend existing data. Manson noted that observational studies and randomized clinical trials of menopausal HRT and coronary heart disease (CHD) have produced widely divergent results. Observational studies had suggested a 40–50 percent reduction in the risk of CHD among women taking HRT, whereas randomized trials suggested a neutral or even elevated risk of coronary events. Well-recognized limitations of observational studies, including the potential for confounding by lifestyle practices, socioeconomic status, education, and access to medical care, as well as selection factors related to “indications for use,” can explain only some of the discrepancies. Other methodologic factors that may help to explain the differences include the limitations of observational studies in assessing short-term or acute risks, which led to incomplete capture of early clinical events after therapy began, and the predominance of follow-up time among compliant long-term users of HRT. In contrast to the greater weighting of long-term use in observational studies, clinical trial results tend to reflect shorter term use. Given that CHD risks related to HRT are highest soon after initiation of therapy, these differences may contribute substantially to the discrepancies observed.

Methodologic differences between observational studies and clinical trials, however, may not fully elucidate the basis for the discrepancies observed. The findings of observational studies and clinical trials are remarkably concordant for other health outcomes, including stroke, venous thromboembolism (blood clot), breast cancer, colorectal cancer, and fracture—outcomes that also should be affected by confounding and selection biases. Indeed, an emerging body of research suggests that the age of menopause or time since menopause critically influences the relationship between HRT and CHD outcomes.

Importantly, observational studies should be designed to capture both short- and long-term risks and should have frequent updating of exposure variables of interest (electronic health and pharmacy records may be useful). Clinical trials must be powered adequately to assess clinically relevant

subgroups and to address the possibility of a modulating effect of key clinical variables. Consideration of absolute risks in research presentation and interpretation is critically important. Finally, it may be helpful to incorporate intermediate and surrogate markers into study designs, although such markers can never fully replace clinical event ascertainment.

Research to date strongly suggests that further understanding of and solutions to the safety issues concerning drug-eluting stents (DES) will likely come from a mix of randomized trials and observational registries conducted in both the premarket and postmarket arenas and a collaborative effort among regulators, industry, and academia. As Ashley B. Boam from the FDA recounted, coronary drug-eluting stents have dramatically changed interventional cardiology practice since their introduction in the United States in 2003. These products—a combination of a metal stent and an antiproliferative drug—have significantly reduced the need for reintervention compared to the previous standard of care, bare metal stents. This substantial improvement has led to widespread adoption of these products and use in patients outside those enrolled in the initial pivotal clinical studies. The desire to bring additional DES technology to the market quickly drove research into the identification of potential surrogate markers for effectiveness. While at least two measures obtained from angiography (i.e., late loss and percentage diameter stenosis of vessel) have been identified as biomarkers with a strong correlation to clinical effectiveness—specifically, the need for a reintervention—no such marker has been identified as a possible surrogate for safety outcomes, such as death or myocardial infarction.

In the last half of 2006 and through 2007, however, the emergence of stent thrombosis—the occurrence of a clot within the stent that often leads to myocardial infarction or death—shifted the focus of DES research significantly from effectiveness to safety. Recent meta-analyses and research from centers in Europe and the United States have indicated that late stent thrombosis may be an ongoing risk to DES patients. Low event rates, less than 1 percent, and late-term occurrence (beyond 1 year postimplantation) have complicated efforts to understand the true incidence and etiology of this noteworthy complication. Confounding the picture is the lack of appropriate studies to optimize prescription of mandatory adjunctive dual antiplatelet therapy, early interruption of which is one known risk factor for stent thrombosis. The issue is challenging in part because what is under consideration is a low frequency event with a late-term appearance, which generally mandates very large and long studies. Randomization, long-term follow-up, concurrent device iterations and new platforms, on-label versus

broad clinical use considerations, and the role of resistance to acetylsalicylic acid and/or clopidogrel are among the other challenges.

Population-based registries—appropriately funded and constructed with clinician engagement—offer a compromise of strengths and limitations and may be the most effective tool for evaluating emerging healthcare technology, argued David R. Flum of the University of Washington. The dichotomy between “effectiveness” and “efficacy” is particularly relevant in evaluating surgical interventions where characteristics of the surgeon (experience, training, and specialty), variations in technical performance, patient selection, practice environments, and publication bias all influence the understanding of healthcare interventions. Case series, often authored by experts in the field, dominate the surgical literature. Despite their limitations, they are strong influences on clinical and policy decisions.

Bariatric interventions include a group of operations that have become increasingly popular with the advent of less-invasive surgical approaches and epidemic obesity. Understanding the safety and efficacy of bariatric interventions has come almost exclusively through single-center case series. A research group based at the University of Washington has worked to expand knowledge in this field through the use of retrospective population-level cohorts using administrative data, clinical registries, and longitudinal prospective cohorts that work to assess effectiveness. These safety data have been helpful in coverage decisions by payers in assessing quality improvement opportunities and in providing more realistic assessments of these interventions. Inherent limitations in effectiveness research include tradeoffs among numbers of patients and details on patients, the granularity and accuracy of the data, and limits to the types of outcomes that can be evaluated (i.e., no quality-of-life or functional data).

As shown in recent research to ensure that antipsychotic medication use is clinically effective, advances in study designs, databases, and analytic methods provide a toolbox of complementary techniques from which researchers can draw, suggested Phillip S. Wang of the National Institute of Mental Health (NIMH). Antipsychotic medications are now widely used by patients and account for a large proportion of pharmaceutical spending, particularly in public healthcare programs. However, there is a paucity of evidence to help guide clinical, purchasing, and policy decisions regarding antipsychotic medications. The recently completed, NIMH-sponsored comparative effectiveness trials of antipsychotic medications in patients

with schizophrenia (the CATIE trial), for example, blends features of efficacy studies and large, simple trials to create a pragmatic trial to provide extensive information about antipsychotic drug effectiveness over a drug course of at least 18 months. Recent advances in databases, designs, and methods can also be brought to bear to improve antipsychotic effectiveness. New study populations and databases have been developed, including practice-based networks that look at psychiatric care. Large administrative datasets are also available, such as Medicaid and useful health maintenance organization (HMO) databases, which are ideal for studying primary care in the setting where most mental health care is actually received.

Other approaches can be employed when trial data are not available. Researchers can use clinical epidemiologic data and methods, which are often a useful addition to the literature. When trials and quasi-experimental and even epidemiologic studies are not possible, researchers can use simulation methods. In addition, researchers have developed new means to deal with threats to validity—both threats to external validity, as in the development of effectiveness research, and threats to internal validity. These include new analytic methods, propensity score adjustments, and instrumental variable techniques.

Developing comparative effectiveness information about screening tests is a complex undertaking, as demonstrated by Peter B. Bach of Memorial Sloan-Kettering Cancer Center. The general rationale for screening in the context of clinical medicine or typical practice is that clinical disease usually “presents”—patients arrive with symptoms or signs that define a population that can be screened. Such circumstances actually make a strong argument for looking for preclinical conditions. That is what screening is intended to do—essentially scan an unaffected population to look for people who are at risk for developing some condition. The underlying principle of such an investigation is that theoretically we can decrease morbidity and mortality and other negative outcomes by looking for patients with preclinical conditions. Screening is widely encouraged, and one could argue that it is the dominant activity in much of primary care. Most medical journals, for example, regularly publish tables of screening evaluations, which are lists of questions that physicians should ask their patients to determine their risk for given diseases.

How these screenings impact a patient’s health need to be understood better. A recent study found that screening for lung cancer with low-dose computed tomography may increase the rate of lung cancer diagnosis and treatment, but may not meaningfully reduce the risk of advanced lung cancer or death. Until more conclusive data are available, asymptomatic

individuals should not be screened outside of clinical research studies that have a reasonable likelihood of further clarifying the potential benefits and risks. There are similar examples—in prostate cancer, in breast cancer with mammography, in renal cell cancer, and in melanoma. A paradoxical reality of the surrogate end-points often used to evaluate effectiveness in these cases is that they are readily available, but can be misleading. Simply stated, screening can often pick up pseudodisease, conditions that are not significant, yet trigger interventions or conditions that cannot be cured that are too often quickly characterized as “early” or “curable.” Refuting the principle of “catch it early” is difficult, but there are many approaches to this goal, each with its own advantages and disadvantages.

Clinical effectiveness research can be improved and expedited through better use of existing methods and attention to emerging tools and techniques that enhance study design, conduct, and analysis. Chapter 3 presents discussion of some key opportunities for advancement in effectiveness research, including improved efficiency and result applicability of trials and studies; innovative statistical approaches to analyses of large databases; capture and use of the wealth of data generated in genomic research; and the promise of simulation and predictive modeling.

As each paper notes, the full benefits of these tools and techniques have yet to be fully realized. To enhance clinical effectiveness research, attention is needed in part to developing a shared understanding of these various approaches and clarity on the insights each offer the research enterprise, both alone and in synergy with other approaches. Essential to this discussion will be careful consideration of circumstances and questions for which a particular approach is best suited.

Clinical trials play an important role in assessing the effects of medical interventions, in particular where observational studies are often inadequate, such as the detection of modest treatment effects or when the risk of an invalid answer is substantial. Robert Califf from Duke University emphasizes the importance of focusing discussion about medical evidence on a serious examination of ways to improve the operational methods of both approaches and of building human systems that take advantage of the power of modern informatics on improving both RCTs and observational studies. In particular, the design and conduct of RCTs needs to evolve to take further advantage of modern informatics and to provide a more flexible and practical tool for clinical effectiveness research. Improvements in

the structure, strategy, conduct, analysis, and reporting of RCTs can help to address their perceived limitations related to cost, timeliness, and reduced generalizability.

Innovative approaches discussed including the conduct of trials within “constant registries” and targeting the standard set of rules for the conduct of trials to make them more adaptable and customized to meet research needs. A relevant model is the clinical practice database found at The Society of Thoracic Surgeons. This database has been used for quality reporting and increasingly to evaluate operative issues and technique. An extension of this model would be to develop constant disease registries capable of drawing on multiple EHR systems. The conduct of RCTs within the database would allow researchers to revolutionize the timeframe and costs of clinical trials. Trials also could take better advantage of “natural units of care” with cluster randomization, or provide information more relevant to practice by focusing on research questions based on gaps in clinical practice guidelines or being conducted in real-world practice (e.g., pragmatic clinical trials). Research networks offer the opportunity to enable the needed sharing of protocols, data structures, and other information.

A promising initiative for improvements in the quality and efficiency of clinical trials is the FDA Critical Path public–private partnership: Clinical Trials Transformation Initiative (CTTI). A collaboration of the FDA, industry, academia, patient advocates, and nonacademic clinical researchers, CTTI is designed to enhance regulations that improve the quality of clinical trials, eliminate guidances and practices that increase costs but provide no value, and conduct empirical studies of the value of guidances and practices. Primary barriers to innovation include the lack of appropriately structured financial incentives and the caution toward change that comes with a highly regulated market. To contend with this substantial barrier to ensuring that innovative approaches are implemented in practice, the research community should adopt a model from business of establishing “envelopes of creativity,” or environments in which researchers could innovate with a certain creative freedom, and where they would have appropriate financial incentives.

Califf concludes that smarter trials that provide timely information on outcomes that matter most to patients and clinicians at an acceptable cost will become an integral part of practice in learning health systems as trials increasing become embedded into the information systems that form the basis for clinical practice. These systems also will provide the foundation for integrating modern genomics and molecular medicine into the framework of care.

Because healthcare databases record drug use and some health outcomes for increasingly large populations, they will be a useful data resource for timely, comparative analyses that reflect routine care. Confounding is one of the biggest issues facing effectiveness research in the analyses of these large-claims databases. While recognizing that instrumental variable analyses have the drawback of producing certain levels of untestable assumptions, Sebastian Schneeweiss from Harvard Medical School proposes that their use can lead to substantial research improvements, particularly in situations with strong confounding and where it is likely that important confounders remain unmeasured in a data source. Several developments may bring the field closer to acceptable validity, including approaches that exploit the concepts of proxy variables using high-dimensional propensity scores and exploiting provider variation in prescribing preference using instrumental variable analysis.

Epidemiologists have a number of techniques that can control for confounding by measured factors, but instrumental variables are a promising approach to address unmeasured confounders because they are an unconfounded substitute for the actual treatment. In this approach, instead of modeling treatment and outcome, researchers model the instrument—which is unconfounded on the outcome—and then correct the estimate for the correlation between the instrumental variable and the actual treatment. In this respect valid results require the identification of a quasi-random treatment assignment in the real world, such as interruption in medical practice. Recent work has also demonstrated the potential of provider treatment preference as a random component in the treatment choice process, providing an additional instrument worth consideration for comparative drug effectiveness studies.

Instrumental variable analysis is an underused, but very promising, approach compared to effectiveness research using nonrandomized data, and researchers should routinely explore whether an instrument variable analysis is possible in a particular setting. Additional work is underway to develop better methods to assess its validity and to develop systematic screens for instrument candidates.

Adaptive and, particularly, Bayesian approaches to study design offer opportunities to improve on randomization and to facilitate new ways of learning in health care. Donald A. Berry from the University of Texas M.D. Anderson Cancer Center contends that these approaches can be used to make RCTs more flexible by using data developed during a study to

guide its conduct and to incorporate different sources of information to strengthen findings related to comparative effectiveness. The historical, frequentist approach, in which a design must be completely designed in advance and the study must be complete before inferences can be made, impedes the ability to continually assess and alter a course of study based on accrued learning. In contrast, the Bayesian approach’s ability to calculate probabilities of future observations based on previous observations enables an “online learning” ideal for developing adaptive study designs. Prospective building of adaptive study designs is critical and, except in the simplest situations, requires simulation.

Adaptive designs increase the flexibility of RCTs by enabling modifications—based on interim or other study results—including stopping the study early, changing eligibility criteria, expanding or extending accrual, dropping or adding study arms or doses, switching between clinical phases, or shifting focus to subsets of patient populations. These adaptations not only enable rapid learning about relative therapeutic benefits but also improve the overall efficiency of research. Flexibility with respect to patient accrual, for example, may enable a needed increase in study sample size, potentially minimizing the need for additional follow-on studies.

Inherently a synthetic approach, Bayesian analysis can also enhance our capacity to appropriately aggregate information from multifarious sources. For instance, in addition to use in meta-analyses, this approach has been used recently to help answer complex questions, such as the proportional attribution of mammographic screening and adjuvant treatment with tamoxifen and chemotherapy in a drop in breast cancer mortality in the United States. The results of the Bayesian models used to explore this question were consistent with non-Bayesian models as well as those derived in clinical trials.

In conclusion, Berry notes that although the rigor and inflexibility of the current research paradigm has been important to establishing medicine as a science, new approaches such as Bayesian thinking and methodologies can help to move the field even further by making research more nimble and applicable to patient care, while maintaining scientific rigor.

Certain research questions or evidence gaps will be difficult or impractical to answer using clinical trial methods. Although physiology-based or mechanistic models have been used only recently in medicine, as noted by Mark S. Roberts from the University of Pittsburgh and representing Archimedes, Inc., physiology-based models, such at the Archimedes model, have the potential to address these gaps by extending results beyond the

narrow spectrum of disease or practice found in most trials. These trials can also reduce the cost and time required to complete RCTs.

Physiology-based models aim to replicate disease processes at a biological level—from individual variables to system-level, whole-organ relationships. The behavior of these elements and effect on health outcomes are modeled using equations derived from and calibrated with data from empirical sources. When properly constructed and independently validated, these models not only can serve as useful tools to identify, set priorities in, or facilitate the design of new trials, but also can be engaged to conduct virtual comparative effectiveness trials. When time, cost, or other factors make doing a trial impossible, an independently validated, physiology-based model provides a useful alternative.

At the forefront of discovery research, genomewide association studies permit examination of inherited genetic variability at an unprecedented level of resolution. As described by Teri Manolio of the National Human Genome Research Institute, given 500,000 or even a million SNPs (single-nucleotide polymorphism or differences among individuals within species) scattered across the genome, researchers can capture as many as 95 percent of variations in the population. This capacity enables “agnostic” genome-wide evaluations, whereby a researcher does not need to preformulate hypotheses or to limit examination to specific candidate genes, but rather can scan the entire genome. Following the availability of high-density genotyping platforms, the pace of genomic discovery has accelerated dramatically for an increasingly broad array of traits and diseases. However, examples of genomewide association studies do have drawbacks. Given the large number of comparisons per study, there is an unprecedented potential for false-positive results. Validation of findings through replication of results generally requires expanding studies from a small initial set of individuals to as many as 50,000 participants.

Two prototypes for applying genomic information from genomewide association studies to clinical effectiveness research are genetic variants related to two traits—Type 2 diabetes risk and warfarin dosing. Though both have sufficient scientific foundations and clinical availability, they remain many steps away from clinical application. Gaining more from these types of insights from genomic research will require additional epidemiologic and genomic information and evidence of impact on outcomes of importance. This evidence can be derived by linking genotypic data to phenotypic characteristics in clinical or research databases, an approach being explored by a number of biorepositories. The National Human Genome Research Institute’s eMERGE network is applying genotyping to

subsets of participants in a number of biorepositories with electronic health records. If the phenotypic measures derived from the EHRs are standardized to increase reliability, these types of linked databases hold significant promise for clinical effectiveness research.

Capacity for research, including the testing and interpretation of results, will require significant laboratory infrastructure, including a valid, readily available, FDA-certified, affordable test, conducted under the auspices of a CLIA (Clinical Laboratory Improvement Amendments)-certified laboratory. Robust electronic health records will also be critical to receive data and provide real-time performance feedback so that patients who receive abnormal results can be given suggestions for how to process that information and proceed from that point. In addition, tools for identifying emerging genomic information with potential clinical applications are needed. In this respect, a useful model is the database of Genotype and Phenotype (dbGaP), an accessible but secure large-scale database that receives, archives, and distributes results of studies of genotype–phenotype associations.

Also of vital importance is infrastructure related to policy and educational needs. For example, there is a pressing need to ensure confidentiality and privacy protection, specifically because the potential for discrimination by employers and insurers might occur if they have access to genomic information. To that end, the recently approved Genetic Information Non-Discrimination Act will be helpful. The research community also needs consensus on what should be reported to patients when abnormalities appear—what to tell them, when to tell them, and how to tell them—and adequate consent policies and procedures as well as consistent Institutional Review Board approaches. A flexible approach to genetic counseling is also needed, including the ability for patients to obtain adequate counseling from someone other than a certified genetic counselor. Also needed is a better educational infrastructure to ensure that these issues are discussed both by physicians and by patients, even during the course of ongoing genomic research; this would include better reporting guidelines for both patients and physicians. Education is needed in medical schools and nursing schools, at professional conferences, and in ongoing professional development and training to ensure that caregivers are “genomically literate.” At the same time, we have a responsibility to also educate the general population, so that patients can develop a deeper understanding of genomics. By learning how genomics affects their lives and their health care, patients will know what questions to ask.

Vastly larger, electronically accessible health records and administrative and clinical databases currently under development offer previously

unimagined resources for researchers and have significant and yet untapped potential to inform clinical effectiveness research. Mining data to expand the base of relevant knowledge and to determine what works for individual patients will use some of the techniques identified in Chapter 3 but will also require tools for organizing and improving these data collections. Chapter 4 provides an overview of the potential for data sources to improve effectiveness research, and identifies opportunities to better define rules of engagement and ensure these emerging data sources are appropriately harnessed. EHR and point-of-care data, enhanced administrative datasets, clinical registries, and distributed data networks are discussed. Collectively these papers illustrate how these approaches can be applied to improve the efficiency and quality of clinical practice; provide meaningful complementary data to existing research findings; accelerate the capture and dissemination of learnings from innovation in practice; and offer a means to process complex information—derived from multiple sources and formats—and develop information that supports clinical practice and informs the research enterprise.

Ronald A. Paulus reported on Geisinger’s use of EHRs to transform care delivery to support his contention that there is more potential than currently exploited at the nexus of point-of-care systems and research. When systematically captured, the data produced by these systems can be used to inform and improve clinical practice. Demonstrating the potential of EHRs for impact beyond practice standardization and decision support mechanisms, Geisinger has used these resources in the production of “delivery-based evidence.” EHRs capture data directly relevant to real work practice and can therefore provide extensive, longitudinal data. When coupled with an integrated data warehouse, the creation of a unique data resource can be mined for both clinical and economic insights. These data can also facilitate observational studies that address issues of clinical relevance to complement and fill gaps in RCT data. In developing such models, it is expected that more thought needs to be given to aggregating, transforming, and normalizing data in order to conduct productive analysis that will bridge the knowledge creation gap.

To illustrate the power inherent in linking data to clinical care, Paulus reviewed Geisinger’s work to enhance performance improvement (PI) initiatives. EHRs and associated data aggregation and analysis to complement PI initiatives are increasingly being adopted in the healthcare setting. At Geisinger, PI has evolved into a continuous process involving data generation, performance measurement, and analysis to transform clinical practice.

Underlying this transformation is an EHR platform fully used across the system. Geisinger’s integrated database, including EHR, financial, operational, claims, and patient satisfaction data, serves as the foundation of a Clinical Decision Intelligence System (CDIS) and is used to inform and document the results of PI efforts. PI Architecture draws upon CDIS and other inputs (e.g., evidence-based guidelines, third-party benchmarks) and leverages this information via decision support applications to help the organization answer important questions that could not be addressed before.

Key goals supported by PI Architecture include (1) assessment of PI initiatives’ returns on investment (ROIs); (2) simultaneous focus on quality and efficiency; (3) development and refinement of reusable components and modules to support future PI efforts; and (4) elimination of any unnecessary steps in care, automating processes when safe and effective to do so, delegating care to the least cost, competent caregiver, and activating the patient as a participant in her own self-care. The PI Architecture enhances and refines the traditional Plan-Do-Study-Act (PDSA) cycle and yields advantages of reduced cycle time, increased relevance, increased sustainability, increased focus on ROI, and enhanced research capabilities. Key features include (1) use of local data to document the current state of practice and to direct focus on areas of greatest potential improvement; (2) use of electronic record review and simulation to confirm hypotheses and to project the benefits of varying avenues and degrees of change; (3) testing on a small scale, using an iterative approach that builds toward a strategy of rapid escalation; and (4) leveraging of reusable parts from past initiatives to build core infrastructure and accelerate future work.

As described by Alexander M. Walker of Worldwide Health Information Science Consultants and the Harvard School of Public Health, data from health insurance claims form the backbone of many health analytic programs. Although administrative databases are being used more effectively for research, their development and especially their application for generating insights into clinical effectiveness require careful consideration and attention to potential methodologic pitfalls and hazards. Nonetheless, there is extraordinary promise in these resources.

Insurance claims data, derived from government payers or independent health insurers, are comprehensive, population based, and well structured for longitudinal analysis. All services and therapeutics for which there is a payment enter the data system, with easy linkage across providers. The regional or employment-based nature of the populations covered includes medicine as actually provided, not just the care that reflects best practice. The need for multiple providers to interact with multiple insurers or with

a government mandate has led to highly standardized data structures supported by regular audit. Finally, these data are available for large numbers of individuals; the largest database with complete information is estimated to include data on 20 million patients.

Although claims data are excellent resources for answering many questions in health services research, they are not always sufficient for clinical research. For example, although labs ordered are recorded, the outcomes or results are not. To address this limitation, research groups have begun to augment their core files—adding laboratory and consumer data, creating the infrastructure for medical record review, implementing methods for automated and quasi-automated examination of masses of data, developing “rapid-cycle” analyses to circumvent the delays of claims processing and adjudication, and opening initiatives to collaborations that respect patients’ and institutions’ legitimate needs for privacy and confidentiality. These enhanced databases provide the information that allows researchers to trace back to specific patients or providers for additional information.

Enhanced claims databases that have been used to support surveillance programs along with automated and quasi-automated database review provide potential decision support tools for clinical safety and efficacy. Basic issues of confounding remain, however, and much attention is needed on these emerging tools to capture the full potential of these databases.

The dynamic and highly innovative character of healthcare technologies has been important to improvements in health; however, because intervention capacities often evolve due to iterative improvements or expanded use in practice, assessing their effectiveness presents a substantial challenge to researchers and policy makers. Because clinical registries capture information important to understanding the use of diagnostic and therapeutic interventions throughout their lifecycle, they are particularly valuable resources for assessing real-world health and economic outcomes. Alan J. Moskowitz and Annetine Gelijns from Columbia University suggest that in addition to providing information important to guiding decision making for patient care and setting policy, registries are valuable for assessing the performance of physicians and institutions, such as through the use of risk-adjusted volume–outcome relationship studies, and for increasing the efficiency of RCTs.

Several examples of findings derived from registry data on left ventricular assist devices (LVADs) were presented by Moskowitz to illustrate the potential of registries to improve effectiveness research. He pointed out that in contrast to efficacy trials or administrative databases, registries are able to keep pace with the dynamic process of medical innovation. The

premarket setting has limits on what can be learned because of the nature of efficacy trials, which are usually short and conducted in narrow populations and under ideal conditions. Once interventions are introduced into general practice, they are used in broader patient populations and under different practice circumstances. For example, only 4 percent of patients treated with coronary artery bypass grafts in practice meet the eligibility criteria of the initial trials (the elderly, females, and those with co-morbidities were excluded). Expanded use of interventions in clinical practice creates a locus for learning and innovation, with the frequent discovery of new and unexpected indications for use, as well as the accrual of knowledge on the appropriate integration of technology into the care of particular patients. Characterized as an “organized system using observational study methods to collect uniform data to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves a predetermined scientific, clinical, or policy purpose,” clinical registries collect data information important to learning and innovation. Examples include long-term outcomes and rare adverse events and outcomes achieved when technology is used by a broadened set of providers or patients. Clinical registries also provide comparative effectiveness information.

Registries offer a powerful means to capture innovation and downstream learning that take place in practice and to develop information complementary and supplementary to that produced by randomized trials. Enhancing the value of registries for clinical research requires improving the quality of data obtained, while decreasing costs and other barriers to data access. Special attention needs to be paid to the definition and standardization of target populations and outcomes (e.g., adverse events); efforts to address bias; measures to ensure representative capture of the population; and sound analytical approaches. Incorporating data collected in the usual course of patient care may help to reduce the burden and cost of registries.

Opportunities to address the traditional weaknesses of registries are presented by advances in informatics, analytical techniques, and new models of financing. The potential of registries to improve the efficiency of randomized trials also must be addressed. The development of investment incentives for stakeholders is important to improving the viability of clinical registries. Although registries have been created by public, not-for-profit, or private organizations, public–private partnerships offer a new model for registry support.

The variety of information created by healthcare delivery has the potential to provide insights to improve care and support clinical research.

Increasingly, these data are held by many organizations in different forms. Although the aggregation of disparate databases into a super dataset may seem desirable for the improved study power and strength of findings provided by larger numbers, such efforts also face significant challenges due to privacy concerns and the proprietary nature of some data. Richard Platt, from Harvard University illustrated how distributed research models circumvent these issues and minimize security risks by allowing the data repositories of multiple parties to remain separately owned and controlled. These models also provide an interface to these stores of highly useful data that allows them to function as a large combined dataset. This approach also takes advantage of local expertise needed to interpret content.

Work at the HMO Research Network has demonstrated the feasibility of this approach for biosurveillance and vaccine safety networks, and several additional models are currently in development. The FDA is calling for the development of a sentinel network to support postmarketing safety research, AHRQ has written a contract to develop a prototype distributed network that will be targeted at comparative effectiveness and safety research, the Robert Wood Johnson Foundation has funded the development of a distributed network that will evaluate quality measures, and the National Institutes of Health (NIH) clinical translational science centers are trying to develop ways for the centers to collaborate with one another in a wide range of clinical research activities. Opportunities to build and use these types of resources also extend to the private sector.

Effective governance models are most pressing for the development and use of these networks. Attention must be given to policies, funding, and further research that supports their development, and to make such work more efficient. Consideration should also be given to developing common standards for distributed research networks so that a single infrastructure might serve multiple functions, even as networks accommodate different kinds of interfaces and governance systems. Overall, governance of such networks, as well as funding for them, will be nontrivial concerns.

In the face of expanding options for diagnostic and treatment options, focus is needed on the development of information that can help guide clinical decision making. Not only are the number of research questions increasing, but current approaches to developing comparative effectiveness information are impractical—making the need to take better advantage of new sources of data and other opportunities to produce evidence relevant to clinical practice more urgent. Many participants noted that in research today, knowledge is expanding much faster than we can effectively translate

and process it in ways that can impact patient care. Thus there is a sense that to address gaps in evidence and newly emerging research challenges, the research community needs to support and nurture efforts to develop new approaches to study design and strategies.

In a session focused on moving to the next generation of studies, participants considered a set of interrelated questions: What are the key opportunities and needed advancements to improve our approach to clinical effectiveness research? How might we take better advantage of emerging resources to plan, develop, and sequence studies that are more timely, relevant, efficient, and generalizable? How can we account for lifecycle variation of the conditions and interventions at play? A variety of innovations were presented, including new mathematical models, new ideas for observational studies and hybrid studies, tools for assessing the roots of genetic variation, cooperative research networks, and even innovation in incentives.

Data are increasingly generated and captured throughout the research, development, and delivery of healthcare products and services. Sharon-Lise Normand of Harvard Medical School contends that the availability of large data streams holds potential to enhance our capacity to produce clinically useful information, but under the current evidence paradigm, these types of information are often wasted. By treating these data sources as silos of information, opportunities are lost for insights on issues such as treatment heterogeneity, or multiple outcomes and patient subgroups. New analytic strategies are needed to more effectively deploy these resources and improve the efficiency with which the research community produces information.

Taking better advantage of large data streams is important in moving to the next generation of studies. The development and implementation of pooling algorithms are needed, as are inferential tools to detect relationships among diverse data sources and to appropriately combine them. Possible applications include comparative effectiveness research through the enhancement of trial results or inferences derived from single data sources. Moreover, although the research community has some experience in pooling observational data with trial data, study designs that exploit features of the emerging diverse data sources—such as hybrid designs, preference-based designs, and quasi-experimental designs—have not yet been exploited to the full potential.

The number of data sources will only continue to expand, and infrastructure is needed to enable their optimal use, including support for the development of new, innovative analytic strategies and mechanisms for

data sharing, documentation, and quality control. Approaches are needed to assess data-pooling strategies, minimize false discoveries, and validate findings. More opportunities are needed to explore approaches to pooling different data sources and employing various study designs. Finally, we need to educate policy makers and researchers in the interpretation of results from the new designs.

Observational studies add value to the research paradigm through their ability to address the dilemma presented by the costs, slow pace, and other logistical difficulties of conducting RCTs. Additionally, suggested Wayne A. Ray of Vanderbilt University, this approach is essential for answering important clinical questions in which RCTs are not appropriate. Although findings of observational studies are intrinsically more prone to uncertainty than those from randomized trials, abiding by some fundamental epidemiologic principles will allow clinicians to better exploit the wealth of available observational data. Common errors include elementary design mistakes; failure to identify a clinically meaningful t₀, or start to follow-up; exposure and disease misclassification; use of overly broad endpoints for safety studies; confounding by the “healthy drug-user effect”; and marginal sample size. These sources of bias and error can easily lead to the design of a “false-negative” safety study or a “false-positive” efficacy study; upon examination, many controversial or misleading results of observational studies result from such suboptimal methodology.

Although the design of observational studies is a complex subject, opportunities exist to improve their capacity to contribute to clinical effectiveness research. The notion that study design and analysis are quick and inexpensive is misleading. Resources and expertise are needed to support state-of-the-art observational studies—in which consideration is given at the outset of the question at hand and how the various biases might apply. For analyses examining safety, limitations that lead to false results are fairly easy to identify and counteract, and a greater challenge is to address conflicts of interest concerns related to their conduct. For efficacy, a first step is to ensure that the expected benefits of therapy exist for the population as a whole, if not for the individual. Underpinning all of these needed advancements is improved education and training of researchers. Epidemiologists need to become more familiar with the clinical and pharmacological principles that affect the use of observational data and to ensure that these guide the design and conduct of proposed and published studies.

Basic scientific research continues to generate advances in treatment options, but bringing a product to market is not enough to ensure its optimal, appropriate, and safe use in clinical practice. In terms of improving health outcomes and use of resources, research to determine how, for whom, when, and in what context a treatment is best used deserves higher emphasis in funding and prioritization. As illustrated by John Rush of the University of Texas Southwestern Medical Center, a variety of methods are available for answering such complex and clinically important questions—such as using observational data obtained when systemic practices are employed (registry/cohort studies), effectiveness trials (practical clinical trials), and hybrid designs. Also of note are new study designs (e.g., equipoise stratified randomized designs; adaptive treatment studies) and posthoc data analyses (e.g., moderator analyses).

The results of a multisite NIMH trial, Sequenced Treatment Alternatives to Relieve Depression (STAR*D), illustrate how a hybrid trial can be employed to provide valuable insights on the optimal use of interventions in clinical practice as well as to improve future research analyses. This investigation helped to answer a host of clinically critical questions, such as the time needed to see a response in treatment; effects of sequence when a course of multiple treatment steps are involved; and effectiveness in a treatment population expanded beyond those in the efficacy trials to include patients with concurrent general medical conditions.

Expanding the evidence base to answer a broader set of questions important to delivering appropriate care—such as those examined in STAR*D—will require the design and use of cost-efficient, rapidly executed studies and the prioritization of research questions. Some key considerations include research that will change practice, enhance outcomes or understanding of the disorder, improve cost efficiency, and/or make treatments safer. Defining and answering these key questions will require the input of relevant stakeholders and care systems reengineered to support research. Finally, once the questions are defined, designs must be identified or developed to obtain the answers. NIH leadership will be essential, as will input and support from clinicians, patients, investigators, and payers. Without these commitments, how, when, for whom, and in what setting a treatment is best will remain the “art of medicine,” rather than the science.

Genomics is poised to have a significant impact on clinical care, and likewise, the medical system could, if adequately harnessed, dramatically transform our understanding of the role of genetic variation on disease development and progression. A significant threat to genomic medicine lies in its potential to generate a large number of spurious findings. To contend with this issue, the research community seeks approaches that garner a large number of patients needed to obtain reproducible linkages between disease characteristics and rare events or weak effects measured for common genetic variants. Given the potential offered by large-scale genomic studies, developing efficient and inexpensive approaches to obtain data of needed quality and quantity is of utmost importance. Issac S. Kohane of Harvard Medical School suggests three prongs of instrumentation of the health system in particular that will help to efficiently produce the large N needed: high-throughput genotyping, phenotyping, and sample acquisition.

Although the costs of high-throughput genotyping are rapidly dropping, the cost of assembling, phenotyping, and studying large populations is an estimated $3 billion for 500,000 individuals. Fortunately, the informational by-products of routine clinical care can be used to bring phenotyping and sample acquisition to the same high-throughput, commodity price point as is currently true of genotyping costs. Kohane discussed recent efforts to contend with challenges related to identifying relevant patient populations and obtaining biosamples from any phenotyped population. First, advances in automated natural language processing has allowed the evaluation of online health record text to quickly, reproducibly, and accurately stratify 96,000 out of 2.5 million patients for disease severity, pharmacoresponsiveness, and exposures. If expanded across many delivery systems, high-throughput phenotyping will be achievable at the national level. Second, a system currently being pilot-tested takes advantage of the many biosamples collected by laboratories in the course of care that are usually discarded. These samples present the opportunity to develop a potentially rich set of clinically relevant information (e.g., genomic composition, identification of biomarkers, effects of new treatments) that can be linked to previously phenotyped populations. A focus on instrumenting the health enterprise will not only contribute to advances in genomic studies but also provide an opportunity to learn from care delivery, if the appropriate security procedures are assured. The detection of cardiovascular risk in patients taking Vioxx was enabled by the development of large-scale databases and datamarts and is one example of how the healthcare system can be used for both discovery research and surveillance.

The most important step toward this goal is to create a safe harbor

for methodological testing that challenges researchers to experiment with large datasets analysis. Open and transparent discussion is needed about the strengths and weaknesses of various methodological approaches; data should be made more broadly available so that researchers can test the data and methodologies and replicate findings. Progress will require increased investment in information technology—particularly to increase the quality of secondary uses of electronically captured data; addressing, through policy and education, various aspects of the Health Insurance Portability and Accountability Act (HIPAA) that prevent broader implementation of these systems and approaches; and the development of an informatics-savvy healthcare research workforce that understands relationships among health information, genomics, and biology.

Clinical effectiveness research draws on experience gained in the post-market setting. Because this research is distinct from that required for FDA approval and market entry, innovative policies that encourage and facilitate these types of investigations will be needed. Wade Aubry from the Center for Medical Technology Policy described coverage with evidence development (CED) as an example of how policy can be used to contend with an essential problem in medical care—that for many clinical situations, evidence is insufficient to inform decision making. A brief history of how concepts evolved since CED’s initial use in the 1990s, as well as some lessons learned, is discussed.

An important early example of CED is of the support of commercial payers such as Blue Cross Blue Shield plans for patient care costs of high-priority, National Cancer Institute (NCI)-sponsored randomized clinical trials evaluating high-dose chemotherapy with autologous bone marrow transplantation. Other examples in Medicare include the FDA/Centers for Medicare & Medicaid Services (CMS) interagency agreement allowing for coverage of Category B investigational devices, coverage of lung volume reduction surgery for bullous emphysema under an NIH protocol, and the Medicare clinical trials policy, under which qualifying clinical trials receive Medicare coverage for patient care costs under an approved protocol. Over the past 4 years, Medicare CED has been formalized by CMS with guidance documents for CED policies on (1) implantable cardioverter defibrillators for prevention of sudden cardiac death, and (2) positron emission tomography for diagnosis of Alzheimer’s disease.

The application of CED to commercial health plans has grown in interest over the past year as part of the debate over whether a national comparative effectiveness institute should be established. Despite this progress, significant barriers to further development of this concept in the private

sector remain. These include health plan Evidence of Coverage (EOC) language defining medical necessity, ethical issues, the difficulty in achieving multistakeholder consensus, lack of a clear definition of “adequate” evidence compared to “ideal” evidence, timing of CED in regard to existing coverage without restrictions, and limitations of the number of studies that can be implemented under CED.

Successful initiatives such as the Cooperative Oncology Groups, HMO Research Network, Center for Education and Research in Therapeutics, the Framingham Heart Study, and others provide useful models for how the clinical effectiveness paradigm might be redesigned to be more timely, relevant, efficient, and generalizable. As noted by Eric B. Larson from the Group Health Cooperative, similar emerging research networks embedded in the healthcare system provide an important anchoring element for a new generation of effectiveness studies. Certain characteristics of research networks make them particularly suited to produce good research on which to base clinical decisions.

Research conducted in a functioning delivery system is generally population based, with greater generalizability and relevance than research in convenience samples or highly selected, specialized populations. This research infrastructure also allows for better accounting for ecological factors in a real, live organization that can affect applicability of research. A well-constructed healthcare system not only enables more efficient conduct of trials and studies but also continuously pioneers technological and structural advancements to improve care. Such advancements are not the focus of traditional research but often significantly improve the quality of care.

Working in a functioning delivery system promotes another key characteristic: the “bidirectionality” of research. Research ideas often emerge from advances in more rarefied research settings when they may be ready for application to practice. Research ideas often surface because of real-world problems that need solutions or because innovative ideas emerge from the practice environment that need refinement and testing. Rather than thinking of only bench to bedside, the best research networks will perform bidirectional research and follow-up, so that there is, in essence, both an afferent and efferent limb to the research. Research networks are ideal for pragmatic and efficient clinical trials and, if population based, assessment of generalizability is ideal. That said, research networks placed in well-constructed healthcare systems will need to (and can) develop other, robust methods such as cluster randomized trials, disease registries, inception cohort studies and time series, and quasi-experimental use of observational data that can be helpful to a learning healthcare organization. Such research will also

inform the field in general, especially if population based, allowing one to infer its generalizabilty. Research networks in well-characterized populations that are enrolled in an organized delivery system are ideal to reduce the time from research to translation, particularly in emerging research areas such as genomic/personalized medicine and use of electronic health records to efficiently determine phenotypes.

We are beginning to realize the potential of bringing together research networks in integrated healthcare systems with university-based scientists. One outcome of the NIH Roadmap has been development of NIH-funded programs under the Clinical and Translational Science Awards (CTSAs). The CTSA program aims to “develop a national system of interconnected clinical research networks capable of more quickly and efficiently mounting large-scale studies.” One consequence of this effort is a nascent culture change and, in places, work by institutions choosing to “reengineer” their clinical and translational research programs. A redesign of the clinical effectiveness research paradigm ideally would address challenges the NIH will face as it aims to reengineer the massive U.S. biomedical research enterprise. To achieve this culture change, both human and technical factors must be addressed: For CTSAs to work together, we will need a baseline level of interoperability to facilitate data exchange. It is also important to consider how CTSAs might seamlessly partner with other networks. Additionally, the research community needs to articulate and conscientiously attend to the challenges of building and sustaining strong collaborative teams.

Reform in clinical effectiveness research will require action beyond the development of new and improved methodologies. The healthcare sectors contain substantial talent and leadership, and many opportunities to better harness and direct these resources to improvement in clinical care might emerge with broader engagement and stakeholder commitment to these efforts. Opportunities for cross-sectoral collaboration are necessary to create a focus and set priorities for these efforts, to clarify the questions that must be addressed, and to marshal the resources that reform requires. Chapter 6 identifies some policy changes that can drive innovative research and explores how government, industry, and consumers can contribute to and build support for clinical effectiveness work.

Clinical systems increasingly capture data on the experience of each patient and clinician in a structured and quantifiable manner. These data resources present a major opportunity for “rapid learning” about the effec-

tiveness of various treatments in the clinical practice setting. Greg Pawlson from the National Committee for Quality Assurance notes that although these data offer the opportunity to bridge the chasm between clinical practice and clinical or health services investigations, much work is needed to make these data available and useful for research. Key barriers to progress include insufficient funding for the development of the health information technology (HIT) infrastructure needed for rapid learning; the influence of HIPAA protections and the proprietary nature of these data on the research questions and approaches pursued; and the structure of the EHR.

Important to enabling the full potential of EHR and other course-of-care data resources will require greater focus on designing and developing EHR systems for research—beginning with the development of data standards and increased funding for the development of HIT infrastructure to facilitate access and use of data. Suggested policy interventions include:

• Better coordinating how the private and public sectors fund research, clinical learning, and HIT development;

• Increasing the proportion of funding dedicated to improving the quality and quantity of secondary database analyses;

• Creating incentives for collecting and structuring data useful for research;

• Providing more open and affordable access to data held by health plans and others;

• Engaging both the private and public sectors in an effort to set standards for how and what data are entered and retrieved from EHRs;

• Modifying HIPAA regulations to remove the major barriers imposed on research while balancing privacy concerns; and

• Improving medical education to better prepare health professionals to use individual and aggregated data for the care of patients.

As the healthcare system becomes more complex, the pharmaceutical industry is increasingly challenged to meet regulatory, payer, and patient demands for demonstration of the value of their products. Such assessments of risk–benefit, long-term safety, and comparative effectiveness often require postmarket clinical trial and database commitments. Peter Honig of Merck reflected on the difficult balance between the data transparency and data access needed to support the necessary epidemiologic, pharmacovigilance, and outcomes research with the increasingly commoditized and proprietary nature of data sources. Additional barriers to efficient use of

these data include the decentralized nature of most utilization and claims outcome data and needed improvements in study and trial methods.

These issues are of particular interest to the industry in light of the high risk and high costs of the drug development process. Honig discusses several important initiatives underway to address these challenges. The FDA’s Critical Path initiative is advocating for public–private partnerships in the precompetitive space to address challenges in drug discovery and development, comparators other than placebo are being increasingly incorporated into clinical postapproval (Phase IV) trials, and structured methods are being developed to provide risk–benefit information that aids clinicians, payers, and regulators in making important decisions about safety, indicated use, and effectiveness. Industry acceptance of an ongoing, learning paradigm for evidence development has increased. Increasing rigor in pharmacoepidemiologic and risk–benefit standards is an asset to the field and the patient. The use of registries, both sentinel and population based, may provide a better method for pharmacovigilance. Increasing use of Bayesian statistical approaches, use of spontaneous and population-based data mining for postmarketing surveillance, and development of sophisticated data analysis tools to improve database output are all advances toward a smarter data collection system. To ensure these data add value to how care is delivered, educational efforts are also needed to improve the translation of generated knowledge into behavior.

Data developed and collected to satisfy regulatory requirements offer a rich resource and a driving force for improvements in our capacity for clinical effectiveness research. Mark B. McClellan of the Brookings Institution reports on two examples of how we might begin to take better advantage of these resources to enhance the healthcare system’s capacity to routinely generate and use data to learn what works in practice. The recently passed Food and Drug Administration Amendment Act of 2007 envisions the development of a postmarket surveillance system that actively monitors for suspected safety problems with medical products. However, it also has the potential to lead to the development of infrastructure built into the healthcare system that can be used to address questions of effectiveness and use of products in different types of patients and populations. To take advantage of the opportunity presented, attention is needed to developing standards and consistent methods for defining adverse events and pooling relevant summary data from large-scale analyses, as well as contending with issues that impeded data sharing.

Medicare’s CED policy has also helped to develop the data needed to better inform coverage and clinical decisions by supporting the conduct

of trials or the development of registries that collect and house sophisticated sets of clinical data about the use and impact of several medical technologies on patient outcomes. McClellan pointed out that clarification by Congress of CMS’s authority to use these methods is needed to support efforts currently underway and to encourage similar efforts taking place in the private sector.

These efforts seek to build the capacity to develop better evidence into the healthcare system, an approach that will become increasingly important to contend with the vast scope and scale of current and future knowledge gaps. The majority of these gaps are in areas for which it has been particularly challenging to develop evidence—such as assessing the effects of the many subtle and built-in differences in medical practice for patients with chronic disease. Because such information is critically important to improving outcomes, and will not be derived from traditional RCTs, priority efforts are needed to enhance the healthcare system’s capacity to generate data as a routine part of care and to use these data to learn what works in practice. Additional support is needed for the infrastructure, data aggregation, and analysis, and for improving the relevant statistical methods.

Though large amounts of data exist and have the potential to inform clinical and comparative effectiveness assessment, substantial barriers prevent optimal use of these data. Many innovative opportunities are possible from these publicly supported and generated data, such as the ability to inform clinical practice and policy. However, the restrictive interpretation of HIPAA and related privacy concerns, the growth of Medicare HMOs, and the fragmentation and commercialization of private-sector clinical databases all limit effectiveness research and threaten effectiveness findings. J. Sanford Schwartz referred to this as the paradox of available but inaccessible data and called for more attention to reducing the barriers to data use until they do not impede research effectiveness.

Enhanced coordination in the development of publicly generated data both within and across agencies can mitigate overlap and redundancy; the government should expand the RCT registry to include all comparative effectiveness research to further the range of issues addressed and information available. Access to data generated by public investment, including those by publicly funded investigators, should be expanded through the development of effective technical and support mechanisms. To move past the barriers presented by HIPAA, Medicare HMOs, and private-sector databases, Schwartz urged establishing practical, less burdensome policies for secondary data that protect patient confidentiality, expand Medicare claims files to incorporate new types and sources of data, and develop more

cost-effective access to private-sector, secondary clinical data for publicly funded studies.

Given that evidence-based medicine requires integration of clinical expertise and research and depends on an infrastructure that includes human capital and organizational platforms, the NIH’s Alan M. Krensky said the NIH is committed to supporting a stable, sustainable scientific workforce. Continuity in the pipeline and the increasing age at which new investigators obtain independent funding are the major threats to a stable workforce. To address these concerns, the NIH is developing new programs that target first-time R01-equivalent awardees with programs such as the Pathway to Independence and NIH Director’s New Innovator Awards, with more than 1,600 new R01 investigators funded in 2007. NIH-based organizational platforms are intra- and interinstitutional. CTSAs fund academic health centers to create homes for clinical and translational science, from informatics to trial design, regulatory support, education, and community involvement. The NIH is in the midst of building a national consortium of CTSAs that will serve as a platform for transforming how clinical and translational research is conducted. The Immune Tolerance Network (ITN), funded by the National Institute of Allergy and Infectious Diseases, the National Institute of Diabetes and Digestive and Kidney Diseases, and the Juvenile Diabetes Research Foundation, is an international collaboration focused on critical path research from translation to clinical development. The ITN conducts scientific review, clinical trials planning and implementation, tolerance assays, data analysis, and identification of biomarkers, while also providing scientific support in informatics, trial management, and communications. Centralization, standardization, and the development of industry partnerships allow extensive data mining and specimen collection. Most recently, the Immune Tolerance Institute, a nonprofit, was created at the intersection of academia and industry to quickly transform scientific discoveries into marketable therapeutics. Policies aimed at building a sustainable research infrastructure are central to support evidence-based medicine.

Conducting meaningful clinical effectiveness research requires collecting, sharing, and analyzing large quantities of health information from many individuals, potentially for long periods of time. To be successful, this research will need the support and active participation of patients. The relationship between researcher and research participant, as defined by

current practice, is ill suited to successfully leverage such active participation. As reported by Kathy Hudson of Johns Hopkins University, however, public engagement efforts in biomedical research, while still in their infancy, suggest some key challenges and opportunities for cultivating active public participation in clinical effectiveness research.

The biomedical community—and the science and technology community more generally—traditionally have viewed the linear progression from public education to public understanding to public support as an accurate model through which to cultivate a public enthusiastically supportive of and involved in research. As the flaws in this philosophy have become more apparent, research-performing institutions increasingly are turning to public engagement and public consultation approaches to enlist public support. Unlike unidirectional and hierarchal communications that characterize past efforts, public engagement involves symmetric flow of information using transparent processes and often results in demonstrable shifts in attitudes among participants (though not always in the direction one might expect or prefer). The outcome is different, as well: Rather than aspiring for or insisting on the public’s deeper understanding of science, a primary goal of public engagement is the scientists’ deeper understanding of the public’s preferences and values.

Most issues here require the attention of the research community in order to drive change, with some of the most pressing concerns in areas such as methods improvement, data quality and accessibility, incentive alignment, and infrastructure. Much work is already underway to enhance and accelerate clinical effectiveness research, but efforts are needed to ensure stronger coordination, efficiencies, and economies of scale within the research community. Participants in the final panel, composed of sector thought leaders, were asked to consider how the research community might be best organized to develop and promote the needed change and to offer suggestions on immediate opportunities for progress not contingent on expanded funding or legislative action.

Panelists characterized the current research paradigm, infrastructure, funding approaches, and policies—some more than 50 years old—as in need of overhauling and emendation. Discussion highlighted the need for principles to guide reform, including a clarification of the mission of research as centered on patient outcomes, identification of priority areas for collective focus, a research paradigm that emphasizes best practices in methodologies, and a greater emphasis on supporting innovation. Apart from a need for stronger coordination, collaboration, and the setting of priorities for questions to address and the studies to be undertaken,

there is a need to develop systems that inherently integrate the needs and interests of patients and healthcare providers. The lifecycle approach to evidence development in which trials and studies are staged or sequenced to better monitor the effectiveness of an intervention as it moves into the postmarket environment was also suggested as an approach that could support the development of up-to-date best evidence. Finally, participants suggested immediate opportunities to build on existing infrastructure that could support the continual assessment approach to evidence development, including broader support of clinical registries and networked resources such as CTSAs, as well as the FDA’s efforts to develop a sustainable system for safety surveillance.

Among the range of issues engaged in the workshop’s discussion were a number that could serve as candidates for the sort of multistakeholder consideration and engagement represented by the Roundtable on Value & Science-Driven Health Care, its members, and their colleagues.

• Methodologies. How do various research approaches best align to different study circumstances—e.g., nature of the condition, the type of intervention, the existing body of evidence? Should Roundtable participants develop a taxonomy to help to identify the priority research advances needed to strengthen and streamline current methodologies, and to consider approaches for their advancement and adoption?

• Priorities. What are the most compelling priorities for comparative effectiveness studies and how might providers and patients be engaged in helping to identify them and to set the stage for research strategies and funding partnerships?

• Coordination. Given the oft-stated need for stronger coordination in the identification, priority setting, design, and implementation of clinical effectiveness research, what might Roundtable members do to facilitate evolution of the capacity?

• Clustering. The NCI is exploring the clustering of clinical studies to make the process of study consideration and launching quicker and more efficient. Should this be explored as a model for others?

• Registry collaboration. Since registries offer the most immediate prospects for broader “real-time” learning, can Roundtable participants work with interested organizations on periodic convening

of those involved in maintaining clinical registries, exploring additional opportunities for combined efforts and shared learning?

• Phased intervention with evaluation. How can progress be accelerated in the adoption by public and private payers of approaches to allow phased implementation and reimbursement for promising interventions for which effectiveness and relative advantage has not been firmly established? What sort of neutral venue would work best for a multistakeholder effort through existing research networks (e.g., CTSAs, HMO Research Network [HMORNs])?

• Patient preferences and perspectives. What approaches might help to refine practical instruments to determine patient preferences—such as NIH’s PROMIS (Patient-Reported Outcomes Measurement Information System)—and apply them as central elements of outcome measurement?

• Public–private collaboration. What administrative vehicles might enhance opportunities for academic medicine, industry, and government to engage cooperatively in clinical effectiveness research? Would development of common contract language be helpful in facilitating public–private partnerships?

• Clinician engagement. Should a venue be established for periodic convening of primary care and specialty physician groups to explore clinical effectiveness research priorities, progress in practice-based research, opportunities to engage in registry-related research, and improved approaches to clinical guideline development and application?

• Academic health center engagement. With academic institutions setting the pattern for the predominant approach to clinical research, drawing prevailing patterns closer to broader practice bases will require increasing the engagement with community-based facilities and private practices for practice-based research. How might Roundtable stakeholders partner with the Association of American Medical Colleges and Association of Academic Health Centers to foster the necessary changes?

• Incentives for practice-based research. Might an employer–payer working group from the Roundtable be useful in exploring economic incentives to accelerate progress in using clinical data for new insights by rewarding providers and related groups that are working to improve knowledge generation and application throughout the care process?

• Condition-specific high-priority effectiveness research targets. Might the Roundtable develop a working group to characterize the gap between current results and what should be expected, based on

• Secondary use of clinical data. Successful use of clinical data as a reliable resource for clinical effectiveness evidence development requires the development of standards and approaches that assure the quality of the work. How might Roundtable members encourage or foster work of this sort?

• Privacy and security. What can be done within the existing structures and institutions to clarify definitions and to reduce the tendencies for unnecessarily restrictive interpretations on clinical data access, in particular related to secondary use of data?

• Collaborative data mining. Are there ways that Roundtable member initiatives might facilitate the progress of EHR data-mining networks working on strategies, statistical expertise, and training needs to improve and accelerate post-market surveillance and clinical research?

• Research-related EHR standards. How might EHR standard-setting groups be best engaged to ensure that standards developed are research-friendly, developed with the research utility in mind, and have the flexibility to adapt as research tools expand?

• Transparency and access. What vehicles, approaches, and stewardship structures might best improve the receptivity of the clinical data marketplace to enhanced data sharing, including making federally sponsored clinical data more widely available for secondary analysis (data from federally supported research, as well as Medicare-related data)?

• Research results. Since part of the challenge in public misunderstanding of research results is a product of “hyping” by the research community, how might the Roundtable productively explore the options for “self-regulatory guidelines” on announcing and working with media on research results?

• Patient involvement in the evidence process. If progress in patient outcomes depends on deeper citizen understanding and engagement as full participants in the learning healthcare system—both

as partners with caregivers in their own care, and as supporters of the use of protected clinical data to enhance learning—what steps can accelerate and enhance patient involvement?

As interested parties consider these issues, it is important to remember that the focus of the research discussed at the workshop is, ultimately, for and about the patient. The goals of the work are fundamentally oriented to bringing the right care to the right person at the right time at the right price. The fundamental questions to answer for any healthcare intervention are straightforward: Can it work? Will it work—for this patient, in this setting? Is it worth it? Do the benefits outweigh any harms? Do the benefits justify the costs? Do the possible changes offer important advantages over existing alternatives?

Finally, despite the custom of referring to “our healthcare system,” the research community in practice functions as a diverse set of elements that often seem to connect productively only by happenstance. Because shortfalls in coordination and communication impinge on the funding, effectiveness, and efficiency of the clinical research process—not to mention its progress as a key element of the learning healthcare system—the notion of working productively together is vital for both patients and the healthcare community. Better coordination, collaboration, public–private partnerships, and priority setting are compelling priorities, and the attention and awareness generated in the course of this meeting are important to the Roundtable’s focus on redesigning the clinical effectiveness research paradigm.

AcademyHealth. 2005. Placement, coordination, and funding of health services research within the federal government. In Academyhealth Report. Washington, DC.

———. 2008. Health Services Research (HSR) Methods. [cited June 18, 2008]. http://www.hsrmethods.org/ (accessed June 21, 2010).

DeVoto, E., and B. S. Kramer. 2006. Evidence-based approach to oncology. In Oncology: An Evidence-Based Approach, edited by A. E. Chang, P. A. Ganz, D. F. Hayes, T. Kinsella, H. I. Pass, J. H. Schiller, R. M. Stone, and V. Strecher. New York: Springer.

Hartung, D. M., and D. Touchette. Overview of clinical research design. American Journal of Health-System Pharmacy 66:398-408.

Kravitz, R. L., N. Duan, and J. Braslow. 2004. Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages. Milbank Quarterly 82(4):661-687.

Liang, L. 2007 (March/April). The gap between evidence and practice. Health Affairs 26(2): w119-w121.

Lohr, K. N. 2007. Emerging methods in comparative effectiveness and safety: Symposium overview and summary. Medical Care 45(10):55-58.

Moses, H., 3rd, E. R. Dorsey, D. H. Matheson, and S. O. Thier. 2005. Financial anatomy of biomedical research. Journal of the American Medical Association 294(11):1333-1342.

Rush, A. J. 2008. Developing the evidence for evidence-based practice. Canadian Medical Association Journal 178:1313-1315.

Sackett, D. L., R. Brian Haynes, G. H. Guyatt, and P. Tugwell. 2006. Dealing with the media. Journal of Clinical Epidemiology 59(9):907-913.

Schneeweiss, S. 2007. Developments in post-marketing comparative effectiveness research. Clinical Pharmacology and Therapeutics 82(2):143-156.

This page intentionally left blank.