Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 87
2
The Work Required
INTRODUCTION
Comparative effectiveness research (CER) is composed of a broad
range of activities. Aimed at both individual patient health and overall
health system improvement, CER assesses the risks and benefits of compet-
ing interventions for a specific disease or condition as well as the system-
level opportunities to improve health outcomes. To meet the ultimate goal
of providing information that is useful to guide the healthcare decisions
of patients, providers, and policy makers, the work required includes con-
ducting primary research (e.g., clinical trials, epidemiologic studies, simu-
lation modeling); developing and maintaining data resources in order to
conduct primary research, such as registries or databases for data mining
and analysis, or to enhance the conduct of other types of clinical research;
and synthesizing and translating a body of existing research via systematic
reviews and guideline development methods. To ensure the best return on
investment in these efforts, work is also needed to advance the develop-
ment of new or refined study methodologies that improve the efficiency
and relevance of research as well as reduce its costs. Similarly, to guide the
overall clinical research enterprise in the efficient production of information
of true value to healthcare decision makers requires the identification of
priority research questions that need to be addressed, the coordination of
the various aspects of evidence development and translation work, and the
provision of technical assistance, such as study design and validation.
The papers that follow provide an overview of the nature of the work
required, noting lessons learned about the known benefits of the country’s
OCR for page 88
LEARNING WHAT WORKS
capacity and experience, and illustrating opportunities to improve care
through capacity building. Emerging from these papers is the notion that
although a number of diverse, innovative, and talented organizations are
engaged in various aspects of this work, additional efforts are needed.
Gains in efficiency are possible with improved coordination, prioritization,
and attention to the range of methods that can be employed in CER.
Two papers provide a sense of the potential scope and scale of the
necessary CER. Erin Holve and Patricia Pittman from AcademyHealth
estimate that approximately 600 comparative effectiveness studies were
ongoing in 2008, including head-to-head trials, pragmatic trials, observa-
tional studies, evidence syntheses, and modeling. Costs for these studies
range broadly, but cluster according to study design. Challenges to develop
the workforce needed for CER suggest the need for greater attention to
infrastructure for training and funding researchers. Providing a sense of the
overall need for comparative effectiveness studies, Douglas B. Kamerow
from RTI International discusses the work of a stakeholder group to
develop a prioritization process for CER topics and some possible criteria
for prioritizing evidence needs. This process yielded 16 candidate research
topics for a national inventory of priority CER questions. Possible pitfalls
of such an evaluation and ranking process are discussed.
Three papers provide an overview of the work needed to support,
develop, and synthesize research. Jesse A. Berlin and Paul E. Stang from
Johnson & Johnson survey data resources for research, and discuss how
appropriate use of data and creative uses of data collection mechanisms
are crucial to help inform healthcare decision making. Given the described
strengths and limitations of available data, current systems are primarily
resources for the generation and strengthening of hypotheses. As the field
transitions to electronic health records (EHRs) however, the value of these
data could dramatically increase as targeted studies and data capture capa-
bilities are built into existing medical care databases. Richard A. Justman,
from the United Health Group, discusses the challenges of evidence synthe-
sis and translation as highlighted in a recent Institute of Medicine (IOM)
report (2008). Limitations of evidence synthesis and translation have led to
gaps, duplications, and contradictions; and, key findings and recommenda-
tions from a recent IOM report provide guidance on infrastructure needs
and options for systematic review and guideline development. Eugene H.
Blackstone, Douglas B. Lenat, and Hemant Ishwaran from the Cleveland
Clinic discuss five foundational methodologies that need to be refined or
further developed to move from the current siloed, evidence-based medicine
(EBM) to semantically integrated, information-based medicine and on to
predictive personalized medicine—including reengineered randomized con-
trolled trials (RCTs), approximate RCTs, semantically exploring disparate
OCR for page 89
9
THE WORK REQUIRED
clinical data, computer learning methods, and patient-specific strategic
decision support.
Finally, Jean R. Slutsky from the Agency for Healthcare Research and
Quality (AHRQ) provides an overview of organizations conducting CER
activities and reflects on the importance of coordination and technical
assistance capacities to bridge these activities. Particular attention is needed
as to which functions might be best supported by centralized versus local,
decentralized approaches
THE COST AND VOLUME OF COMPARATIVE
EFFECTIVENESS RESEARCH
Erin Holve, Ph.D., M.P.H., Director; Patricia Pittman, Ph.D.,
Executive Vice President, AcademyHealth
Overview
In the ongoing discussion about CER, there has been limited under-
standing of the current capacity for conducting CER in the United States.
This report intends to help fill this gap by providing an environmental scan
of the volume and the range of costs of recent CER. This work was funded
by the California HealthCare Foundation. Current production of CER is
not well understood, perhaps due to the relatively new use of the term, or
perhaps as a result of fragmented funding streams.
Comparative Effectiveness Research Environmental Scan
This study sought to determine whether there is a significant body of
CER under way so that policy makers interested in improving outcomes
can plan appropriate initiatives to bolster CER in the United States. This
study does not catalog the universe of CER because existing data sources
are limited by the way that research databases are developed. However, it
is a first attempt to assess the approximate volume and cost of CER.
The study focused on four major objectives:
Identify a typology of CER design.
1.
Characterize the volume of research studies that address compara-
2.
tive effectiveness questions.
Provide a range of cost estimates for conducting comparative effec-
3.
tiveness studies by type of study design.
Gather information on training to support the capacity to produce
4.
CER.
OCR for page 90
90 LEARNING WHAT WORKS
These efforts relied on three modes of data collection. The first phase
included the development of a framework of study designs and topics. The
second consisted of a structured search of research projects listed in two
databases: www.clinicaltrials.gov and Health Services Research Projects
in Progress (HSRProj),1 a database of health services research projects in
progress. The third consisted of in-person and telephone interviews with
25 research organizations identified as leaders in comparative effectiveness
studies. The number, type, and costs of studies were noted, although only
studies cited by funders were included in the estimates of volume because
of the possibility of double-counting studies cited by both researchers and
funders. Interviews were used to triangulate information on costs and the
relative importance of different designs.
An important study limitation was that it was not possible to cross-
reference the databases and the interviews. As a result, although these
sources are comparable, they should not be aggregated.
In an initial focus group with experts on CER, a definition of CER was
developed to guide the project. Though many definitions of CER have been
developed,2 this project relies on the following:
• CER is a comparison of the effectiveness of the risks and benefits of
two or more healthcare services or treatments used to treat a spe-
cific disease or condition (e.g., pharmaceuticals, medical devices,
medical procedures, other treatment modalities) in approximate
real-world settings
• The comparative effectiveness of organizational and system-level
strategies to improve health outcomes is excluded from this defini-
tion, as is research that is clearly “efficacy” research. This means
that studies that compare an intervention to placebo or to usual
care were excluded from our counts.
The expert panel also developed a framework of research designs to
make it possible to categorize findings systematically. For the purposes of
this study there was general agreement that there are three primary research
categories3 applicable to CER:
1 HSRProj may be accessed at www.nlm.nih.gov/hsrproj/ (accessed September 22, 2010).
2 In a recent report from the Congressional Budget Office, the authors state that compara-
tive effectiveness is “simply a rigorous evaluation of the impact of different treatment options
that are available for treating a given medical condition for a particular set of patients” (CBO,
2007). An earlier report by the Congressional Research Service makes an additional distinction
that comparative effectiveness is “one form of health technology assessment” (CRS, 2007).
3 During the interviews we attempted to identify research by more specific types, asking
questions about pragmatic trials, registry and modeling studies, and systematic reviews.
OCR for page 91
9
THE WORK REQUIRED
1. head-to-head trials (including pragmatic trials);
2. observational studies (including registry studies, prospective cohort
studies, and database studies); and
3. syntheses and modeling (including systematic reviews).4
Clinicaltrials.gov is the national registry of data on clinical trials, as
mandated by the Food and Drug Administration (FDA) reporting process
required for drug regulation. Clinicaltrials.gov includes more than 53,000
study records and theoretically provides a complete set of information on
all clinical trials of drugs, biologics, and devices subject to FDA regula-
tions (Zarin and Tse, 2008). While the vast majority of trials included in
clinicaltrials.gov are controlled experimental studies, there are some obser-
vational studies as well.
HSRProj is a database of research projects related to healthcare access,
cost, and quality as well as the performance of healthcare systems. Some
clinical research may be included in HSRProj if it is focused on effective-
ness. HSRProj includes a variety of public and private organizations but
only a limited number of projects funded by private or industry sources.
A search was conducted in www.clinicaltrials.gov for phase 3 and phase
4 observational and interventional studies. Phase 4 studies were narrowed
by searching only for studies containing the term effectiveness. Studies that
were explicitly identified as efficacy studies or those that did not include at
least two active comparators were excluded. The HSRProj database was
also searched for studies on effectiveness. Because HSRProj does not dif-
ferentiate between study design phases, a search was also conducted by the
types of studies identified in the framework. The studies identified through
the process of searching both databases were then searched by hand in
order to identify projects that met the definition of CER.
The interview phase of the project included in-person and telephone
interviews with research funders and researchers who conduct CER. An
initial sample of individuals funding and conducting CER were contacted
in response to recommendations by the focus group panel, and these initial
4 Observational research studies include a variety of research designs but are principally
defined by the absence of experimentation or random assignment (Shadish et al., 2002). In
the context of CER, cohort studies and registry studies are generally thought of as the most
common study types. Prospective cohort studies follow a defined group of individuals over
time, often before and after an exposure of interest, to assess their experience or outcomes
(Last, 1983), while retrospective cohort studies frequently use existing databases (e.g., medical
claims, vital health records, survey records) to evaluate the experience of a group at a point
or period in time.
Registry studies are often thought of as a particular type of cohort study based on patient
registry data. Patient registries are organized systems using observational study methods to
collect patient data in a uniform way. These data are then used to evaluate specific outcomes
for a population of interest (Gliklich and Dreyer, 2007).
OCR for page 92
92 LEARNING WHAT WORKS
contacts in turn recommended other respondents—a “snowball” sample. In
total, 35 individuals representing 25 research funders or research organiza-
tions participated in the project.5
Findings
For the project, 689 comparative effectiveness studies were identified
in clinicaltrials.gov and HSRProj. Of these the vast majority are “interven-
tional trials” listed on clinicaltrials.gov; specifically, they are phase 3 trials
that compare two or more treatments “head to head,” have real-world ele-
ments in their study design, and do not explicitly include efficacy in their
description of the study design. Only 19 studies are phase 4 post-marketing
studies that compare multiple treatments. Seventy-three CER projects were
identified in HSRProj. The process of manually searching project titles
confirmed that most studies in this database were observational research,
although a handful of studies were specifically identifiable as registry stud-
ies, evidence synthesis, or systematic reviews.
The interviews with funders identified 617 comparative effectiveness
studies, of which approximately half were observational studies (prospec-
tive cohort studies, registry studies, and database studies). Research synthe-
ses (reviews and modeling studies) and experimental head-to-head studies
also represent a significant proportion of research activities.
Neither clinicaltrials.gov nor HSRProj publish funding amounts, so
interviews with funders and researchers are the sole source of data on cost.
As would be expected across the range of study designs covered, there is an
extremely broad array of cost for CER studies. However, despite the range,
cost estimates provided in the interviews did tend to cluster, particularly by
study type. While the cost of conducting head-to-head randomized trials
was extremely broad, including studies as costly as $125 million, smaller
trials tended to range from $2.5 million to $5 million and larger studies
from $15 million to $20 million.
Likewise, the range of cost for observational studies was extremely
broad but tended to cluster (Table 2-1). While large prospective cohort
studies cost $2 million to $4 million, large registry studies cost between
$800,000 and $6 million, with most examples falling at the higher end of
this range, although a few were substantially less. Retrospective database
studies tended to be less expensive and cost on the order of $100,000
5 Though an initial group of potential respondents was identified as funders of CER, it was
often necessary to speak with multiple individuals to find the appropriate person or group re-
sponsible for CER within the organization’s portfolio. For this reason the response rate among
individuals is lower than might be expected for a series of key informant interviews. Thirteen
organizations were identified that did not suggest they received funding for CER from other
sources. This subset is used as the sample of organizations that fund or self-fund CER.
OCR for page 93
9
THE WORK REQUIRED
TABLE 2-1 Costs of Various Comparative Effectiveness Studies
Type of Study Cost
Head to head
Randomized controlled trials:
Smaller $2.5m–$5m
Larger $15m–$20m
Observational
Registry studies $2m–$4m
Large prospective cohort studies $800k–$6m
Small retrospective database studies $100k–$250k
Synthesis
Simulation/modeling studies $100k–$200k
Systematic reviews $200k–$350k
to $250,000. Systematic reviews and modeling studies tended to be less
expensive and have a far narrower range of cost, in part because these
studies were based on existing data, with many falling between $100,000
to $200,000. It is important to note, however, that this may not include
the cost of procuring data. Systematic reviews cluster around a range of
$200,000 to $350,000.
There are, of course, additional activities and costs of involving stake-
holders in research agenda setting as well as prioritizing, coordinating, and
disseminating research on CER that are not included here.6 These impor-
tant investments will need to be considered in the process of budgeting for
CER.7
6
Examples of activities designed to prioritize and coordinate research activities include the
National Cancer Institute’s CER Cancer Control Planet (http://cancercontrolplanet.cancer.
gov/), which serves as a community resource to help public health professionals design, imple-
ment, and evaluate CER-control efforts (NCI, 2007). Within the the Agency for Healthcare
Research and Quality, the prioritization and research coordination efforts for comparative
effectiveness studies are undertaken as part of the Effective Health Care Program. Translation
and dissemination of CER findings is handled by the John M. Eisenberg Clinical Decisions
and Communications Science Center, which aims to translate research findings to a variety of
stakeholder audiences. No budget information is readily available for the Eisenberg Center
activities.
7 Examples of stakeholder involvement programs include two programs at the FDA focused
on involving patient stakeholders, the Patient Representative Program and the comparative
OCR for page 94
9 LEARNING WHAT WORKS
Finally, the interviews also shed light on two subjects discussed at this
workshop (IOM, 2008): (1) the need for additional training in the meth-
ods and conduct of CER; and (2) the need to bring researchers together to
discuss the relative contributions of RCTs and observational studies in the
context of CER.
While interviewees generally commented that they have some capac-
ity to respond to an increase in the demand for CER, some noted that
they have had difficulty finding adequately trained researchers to conduct
such research. For the moment, training programs are limited primarily
to research trainees working with AHRQ’s evidence-based practice cen-
ters and the Developing Evidence to Inform Decisions about Effectiveness
(DEcIDE) network. Respondents also mentioned two postdoctoral train-
ing programs designed to teach researchers how to conduct effectiveness
research. The Duke Clinical Research Institute (DCRI) offers a program
for fellows and junior faculty. Fellows studying clinical research may take
additional coursework and receive a masters of trial services degree in clini-
cal research as part of the Duke Clinical Research Training Program. The
Clinical Research Enhancement through Supplemental Training (CREST)
program at Boston University is the second program mentioned. CREST
trains researchers in aspects of clinical research design, including clinical
epidemiology, health services research (HSR), biobehavioral research, and
translational research. Both the DCRI and CREST programs have a strong
emphasis on clinical research using both randomized experimental study
designs and observational designs.
Other funding for training includes the National Institutes of Health
(NIH) K30 awards to support career development of clinical investigators
and to develop new modes of training in theories and research methods.
The program’s goal is to produce researchers capable of conducting patient-
oriented research on epidemiologic and behavioral studies and on outcomes
or health services research (NIH, 2006). As of January 2006, the Office
of Extramural Affairs lists 51 curriculum awards funded through the K30
mechanism. In November 2007 AHRQ released a Special Emphasis Notice
effectiveness research Drug Development Patient Consultant Program (Avalere Health, 2008;
FDA, 2009). Other examples of stakeholder involvement programs include the National Insti-
tute for Occupational Safety and Health–National Occupational Research Agenda program,
the American Thoracic Society Public Advisory Roundtable, and the National Institutes of
Health director’s Council of Public Representatives (COPR). These efforts can represent a
sizeable investment in order to assure stakeholder involvement among the potentially diverse
group of end users. For example, the COPR is estimated to cost approximately $350,000 per
year (Avalere Health, 2008). From an international perspective, the UK’s National Institute
for Health and Clinical Excellence (NICE) allocates approximately 4 percent of their annual
budget (approximately $775,000) in NICE’s Citizen’s Council and for their “patient involve-
ment unit” (NICE, 2004).
OCR for page 95
9
THE WORK REQUIRED
for Career Development (K) Grants focused on CER (AHRQ, 2007). Four
career awards are slated to support the development, generation, and trans-
lation of scientific evidence by enhancing the understanding and develop-
ment of methods used to conduct CER.
The challenge of filling the “pipeline” of researchers working on com-
parative effectiveness is exacerbated by what many of the interviewees
viewed as a fundamental philosophical difference between researchers who
were academically trained in observational research and those who are
trained on the job to conduct clinical trials. Several respondents noted that
these differences likely arise because the majority of researchers are trained
in either observational study methods or randomized trials, but rarely both.
In addition, as noted by Hersh and colleagues,8 there are many unknowns
related to assessing the workforce needs for CER, including the unresolved
definitional issues and scope of comparative effectiveness. As noted by
Hersh, the proportion of CER that is focused on randomized trials, obser-
vational research, and syntheses has strong implications for the number of
researchers (and the type of training) that will be needed. For this reason
it is important to track research production and funding for CER in order
to anticipate future needs.
Some respondents noted that differences in training manifest themselves
in disagreements about the benefits of various observational study designs.
Nevertheless, most individuals interviewed as part of this study felt that
RCTs, observational studies (including registry studies, prospective cohort
studies, and quasi-experiments), and syntheses (modeling studies and sys-
tematic reviews) are complementary strategies to generate useful informa-
tion to improve the evidence base for health care. Furthermore, many
participants agreed that, as CER evolves, it will be critical to develop a
balanced research portfolio that builds on the strengths of each study type.
To facilitate this balance, some of the interviews, as well as many comments
at the IOM’s Roundtable on Value & Science-Driven Health Care (IOM,
2008) suggest that training opportunities to bridge gaps in language and
methods used by researchers may be helpful in creating a balanced portfolio
of CER.
Conclusion
Findings from this study indicate that there is a greater volume of ongo-
ing CER than may initially have been supposed. The cost of conducting
this research varies greatly, although it tends to cluster by type of study.
8 Hersh, B., T. Carey, T. Ricketts, M. Helfand, N. Floyd, R. Shiffman, and D. Hickam. A
framework for the workforce required for comparative effectiveness research. See Chapter 4
of this publication.
OCR for page 96
9 LEARNING WHAT WORKS
The range of studies that are currently being conducted and the cost varia-
tion by study type have implications for the mix of activities that may be
undertaken by an entity focused on CER. Furthermore, as identified in the
interviews, assuring sufficient research capacity to conduct CER is likely
to require an investment in multidisciplinary training, with an emphasis
on bridging the gap between awareness of the strengths and limitations of
randomized trials, observational study designs, and syntheses.
INTERVENTION STUDIES THAT NEED TO BE CONDUCTED
Douglas B. Kamerow, M.D., M.P.H.
Chief Scientist, Health Services and Policy Research, RTI International,
and Professor of Clinical Family Medicine, Georgetown University
Overview
CER compares the impact of different treatments for medical condi-
tions in a rigorous, practical manner. At the request of the IOM’s Round-
table on Value & Science-Driven Health Care, IOM staff convened in 2008
a multisectoral working group to create a national priority assessment
inventory. Their charge was to set criteria for choosing appropriate CER
topics and then to nominate and review example topics for needed research.
An abridged summary of the report is presented in Appendix B. Appendixes
C and D, respectively, are the recommended priority CER studies proposed
in 2009 by the IOM Committee on Comparative Effectiveness Research
Prioritization and the Federal Coordinating Council for Comparative Effec-
tiveness Research.
Introduction
CER has been defined as “rigorous evaluation of the impact of differ-
ent options that are available for treating a given medical condition for
a particular set of patients,” (CBO, 2007) and “the direct comparison of
existing healthcare interventions to determine which work best for which
patients and which pose the greatest benefits and harms” (Slutsky and
Clancy, 2009). Broadly construed, this type of research can involve com-
parisons of different drug therapies or devices used to treat a particular
condition as well as comparisons of different modes of treatment, such as
pharmaceuticals versus surgery. It can also be used to compare different
systems or locations of care and varied approaches to care, such as dif-
ferent intervals of follow-up or changes in medication dosing levels for a
particular condition. CER also may be used to investigate diagnostic and
OCR for page 97
9
THE WORK REQUIRED
preventive interventions. All of these studies may include an evaluation of
costs as well as an assessment of clinical effectiveness.
Comparative assessment research is especially valuable now, in an era
of unprecedented healthcare spending and large, unexplained variations in
care for patients with similar conditions. The IOM’s Roundtable on Value
& Science-Driven Health Care set a goal for the year 2020 that 90 percent
of clinical decisions will be supported by accurate, timely, and up-to-date
clinical information that reflects the best available evidence. Currently,
there is insufficient evidence about which treatments are most appropri-
ate for certain groups of patients and whether or not those treatments
merit their sometimes significant costs. The healthcare community requires
improved evidence generation that will address how different interventions
compare to one another when applied to target patient populations. CER
provides the opportunity to ground clinical care in a foundation of sound
evidence.
The current system usually ensures that when a new drug or device is
made available, there is evidence to show its effectiveness compared to a
placebo in ideal research conditions—that is, its efficacy. But there is often
an insufficient body of evidence demonstrating its relative effectiveness
compared to existing or alternative treatment options, especially in real-
world settings. This limited scope of information increases the likelihood
that clinical decisions are not based on evidence but rather on local practice
style, institutional tradition, or physician preference. Although the numbers
vary, some estimate that less than half—perhaps well less than half—of all
clinical decisions are supported by sufficient evidence (IOM, 2007). This
lack of evidence also leads to substantial geographic variations in care,
further supporting the idea that patients may be subjected to treatments
that are unnecessarily invasive—or not aggressive enough—for a variety
of conditions. These variations in care partly explain healthcare spend-
ing differences across geographic regions that cannot be fully accounted
for by price differences or illness rates. Geographic variations in treat-
ment approach are often greater when there is less agreement within the
medical community about the appropriate treatment. Variation in treatment
approach for a variety of conditions is of significant concern because it has
not been demonstrated that areas with higher levels of spending—where
presumably patients are treated with more aggressive or expensive options
or with simply more treatment—have significantly better health outcomes
than areas with lower levels of spending.
The Institute of Medicine and Comparative Effectiveness
The IOM’s Roundtable on Value & Science-Driven Health Care rec-
ognized the importance of furthering CER to ensure that all clinical deci-
OCR for page 142
2 LEARNING WHAT WORKS
Summary
Moving beyond today’s Mt. Everest level of difficulty, RCTs need
to become more nimble and simple to better reflect the real world and
to have their financing restructured. Heterogeneity in practice facilitates
approximate randomized trials via propensity score methods that are inex-
pensive and widely accessible but which require patient-level clinical data
stored as discrete values for variables. Emerging semantic technology can be
exploited to integrate currently disparate, siloed medical data—responding
to investigators’ complex queries and patients’ imprecise ones—and in
the near future holds the promise to automate discovery of unsuspected
relationships and unintended adverse or surprisingly beneficial outcomes.
A next generation of analytic tools for revealing patterns in clinical data
should build on successful methods developed in the discipline of machine
learning. Both new knowledge learned and resulting algorithms should
be transformed into strategic decision support tools. These are but a few
concrete examples of methods that need to be developed to provide an
infrastructure to determine the right treatment for the right patient at the
right time.
Resources Needed
What resources are needed to develop this infrastructure?
Reengineering Randomized Controlled Trials
The cost of an NIH-sponsored simple trial appears to be in the range
of $2 million, but multi-institutional, multinational large trials driven by
clinical end points can consume 10 times that figure. If one uses $100 mil-
lion as a metric, this means 5 to 50 such trials of therapy can be supported.
Considering all the therapies of medicine for which the evidence base is
weak, it is clear that demanding gold-standard RCTs for everything is unaf-
fordable. The cost of RCTs that are highly focused, ethically unambiguous,
and feasible could be brought down to a quarter, perhaps even a tenth, of
this figure based on practical experience. This will require maximum use
of electronic patient records, consisting of values for variables, and quite
specifically longitudinal surveillance data to study the long-term side effects
of therapies.
Approximate Randomized Controlled Trials
The NIH and National Science Foundation (NSF) should join forces
and solicit 3-year methodology grants of approximately $250,000 per year,
OCR for page 143
THE WORK REQUIRED
10 per year. For this $7.5 million investment, a strong understanding of
how best to use nonrandomized data would emerge. With this would come
production of publicly available statistical software.
If rich discrete clinical data were available for analysis, a typical study
using these methods for nonrandomized comparison would cost approxi-
mately $75,000. The cost would double if extensive integration of data was
necessary, possibly over healthcare networks. For $100 million, it would be
possible to conduct more than 1,000 such approximate randomized trials.
This could have a major impact on acquiring what might be called “silver-
level” evidence for practice.
Semantically Integrating, Querying, and
Exploring Disparate Clinical Data
Based on several years of work, it seems that developing a comprehen-
sive ontology of medicine—a new framework for analysis across disparate
medical domains—will cost about 1 hour of time per term for an analyst,
programmer, and clinical expert. One need not start from scratch, but can
exploit SNOMED, UMLS, and other term lists and ontologies to start the
process. Assuming that 100,000 terms would need to be defined in this
fashion, that the wages would be $300 per hour, and that 25 ontologists
would be needed, this work could be completed in 2 years at a cost of
$36 million. This would include the software that must be programmed to
implement a global effort in rallying medical experts to this task.
Computer Learning Methods
Knowledge discovery in medicine involves both methodologic devel-
opment and applications. These should go hand in hand in this new field
because it would accelerate the development of methods as they encounter
problems requiring further methodologic work. The NSF has begun an
initiative called Cyber-Enabled Discovery and Innovation (Jackson, 2007).
This began with a $52 million first-year budget and is intended to ramp
up $50 million per year and finish within 5 years for a total of $750 mil-
lion. It would be useful to add $10 million per year for direct application
to biomedicine, for a total sustained level for these activities within 5 years
of $50 million.
Patient-Specific Strategic Decision Support
Costs in this area are largely for developing software, including the
interfaces to EMR systems. This could be done for approximately $10
million. One could envision every study of clinical effectiveness having a
OCR for page 144
LEARNING WHAT WORKS
patient-specific prediction component built into it. Again, based on experi-
ence doing this, approximately $25,000 per study would be required to
adapt and test the software and couple it with EMRs for decision support.
It is also likely that at some point, the FDA may become involved with tools
such as this and would introduce regulations that are more costly to meet
than those of performing the studies.
COORDINATION AND TECHNICAL ASSISTANCE
THAT NEED TO BE SUPPORTED
Jean R. Slutsky, Director, Center for Outcomes and Evidence,
Agency for Healthcare Quality and Research
Overview
CER as a concept and reality has grown rapidly in the past 5 years.
While it builds on an appreciation for the role of technology assessment,
comparative study designs, and the increased role of health information
technology to gather evidence and distribute it to the point of care, the
capacity and infrastructure for this research has received less targeted
attention. Understanding the landscape of organizations and health sys-
tems undertaking CER is challenging but essential. Without knowing what
capacities and infrastructure currently exist, rational strategic planning for
the future cannot be done. It is also important to address which functions
can be most effective if they are centralized, which are most effective if
they are local or decentralized, and how different activities relate to each
other in a productive way. This paper will explore the practical realities of
what exists now, what is needed for the future, and how the needs of the
country’s diverse healthcare system for CER can best be met.
The Agency for Healthcare Research and Quality Perspective
AHRQ plays a significant role in CER. Under a mandate included in
Section 1013 of the Medicare Prescription Drug, Improvement, and Mod-
ernization Act of 2003, AHRQ is the lead agency for CER in the United
States. AHRQ conducts health technology assessment at the request of
CMS and analyzes data and suggests options for coverage with evidence
development (CED) and post-CED data collection. AHRQ also provides
translation of CER findings, promotes and funds comparative effective-
ness methods research, and funds training grants focused on comparative
effectiveness. AHRQ has an annual budget of over $300 million ($372
million for 2009), and received funds specifically for work on CER ($30
OCR for page 145
THE WORK REQUIRED
million and $50 million in 2008 and 2009, respectively).14 AHRQ has built
a flexible, dynamic infrastructure for CER that includes 41 research cen-
ters nationwide with more than 160 researchers. The program includes K
awards for career development such as the Mentored Research or Clinical
Scientist Development awards, and Independent Scientist awards. AHRQ
has also funded methods research that heretofore had not been funded
except on an ad hoc basis. AHRQ has also put concentrated funding into
the translation of CER findings.
Given the pressing need for evidence, it is important to keep in mind
the high costs of precision. It costs money to conduct precise studies that
answer detailed questions, which speaks to the importance of not only pri-
ority setting but also understanding that this is an important and difficult
task.
In the United States, the landscape for CER is, for the most part, very
well intentioned. All parties engaged in this work want to do the right
thing and see what works best for patients in the United States and else-
where. Nonetheless, current efforts are too ad hoc in nature. There are
no adequate organizing principles, except for those outlined in Section
1013, where language in the legislation focuses on setting priorities, hav-
ing transparent processes, involving stakeholders, and having a translation
component. The effect is that in the United States there is essentially only
a very limited capacity to conduct CER and to translate that research into
meaningful and useful applications. The United States is not accustomed,
nor organized effectively, to conduct this type of research. In part, this is
because the system tends not to grow researchers who have the capacity
to move beyond what might be described as a parochial mind set regard-
ing the types of research study designs, a mind set which has limited the
capacity to readily generate hypotheses and study designs appropriate for
CER. Generally, researchers do not involve stakeholders to the extent that
is required for research aimed at generating information to guide end users
such as patients and physicians. A key shift needed in the current approach
to research is to involve patients and other key stakeholders, such as indus-
try and health plans, in the formulation of questions for investigation and
in study design.
As mentioned above, AHRQ is currently conducting CER under leg-
islative mandate. Other federal agencies also conducting CER include the
NIH, CMS (CED), and the VA; some have done so for decades. In the
private sector, health plans and industry are also engaged in CER. In addi-
tion, CER-focused public–private partnerships are starting to form, as are
private–private partnerships. Unlike other forms of research, CER will most
certainly require the kinds of partnerships that are now emerging.
14 See http://www.ahrq.gov/about/budgtix.htm (accessed September 8, 2010).
OCR for page 146
LEARNING WHAT WORKS
There are a number of common pitfalls in CER. One of the most sig-
nificant failures is that comparative effectiveness studies are not designed
in ways that capture meaningful end points as well as longitudinal and rel-
evant outcomes—end points that would be meaningful not only to patients
but also to decision makers. There are also significant issues about the
applicability of the CER studies that are conducted, and the work we do
for CMS is reflective of the need for research conducted in patients repre-
sentative of the Medicare population. The elderly tend not to be studied in
rigorous trials to the extent that children are, although AHRQ has begun to
address this discrepancy through, for example, work for the Medicaid and
State Children’s Health Insurance programs. There is also a failure to clini-
cally address relevant heterogeneity and biological heterogeneity as well.
Finally, there are also responsibility issues, as evidenced in discussions of
“who pays and who stays” when the need for an important study has been
noted. There is currently discussion of such issues on Capitol Hill.
Today’s reality in CER, therefore, can be summarized as follows. There
is general sentiment that CER can be a positive thing if it is done fairly, is
well designed, and is transparent. This is important because of the potential
impact of CER on many different sectors—not just patients, but also indus-
try and health plans. If CER is not conducted in a way that stakeholders
can understand—and, importantly, in a way by which they have input into
the process—it could happen that CER does not have the impact, in terms
of improving health outcomes, that everyone hopes it will have. As AHRQ
discovered in developing the Section 1013 healthcare program, involving
stakeholders early, listening to them, and involving them throughout the
process through to the end and implementation of the findings, is critically
important.
Another issue with CER today is that there is no agreement on the
best methods for setting priorities. Everyone tries to set priorities finds that
reaching consensus is hugely difficult. In part that is because the process
of setting such priorities tends to revert to personal or narrowly defined
considerations. If a consensus is to be reached, rather than thinking about
individual priorities, it will be important to instead learn to focus on
national priorities. Experience shows, however, that such priority shifting
is difficult.
Another aspect of the reality of CER today that has been disappointing
is that there has been less emphasis on designing good studies than on just
the concept of CER. A great deal of time is spent talking about where CER
should live, what it should look like, and how it should be funded, but there
has tended not to be adequate discussion about how to conduct CER most
effectively from a methodological and implementation standpoint. Further
discussion is needed on both sides of this issue—discussion not only about
which box CER lives in, if you will, but about what’s inside the box.
OCR for page 147
THE WORK REQUIRED
Once a decision to perform a CER study is made, finding a payer or
a funder for the research component is difficult. Obviously, with a budget
of $30 million AHRQ cannot be the sole funder of many of these stud-
ies. Regardless, the reality is that the challenges related to funding lead
to less rigorous and less innovative study designs. Somehow the issue of
how different players collaborate to fund these studies must be addressed.
Otherwise, the rigorous study designs needed to move this field forward
will never be developed.
Successes, however, make the fundamental concept of CER worthwhile.
Among the successes, for example, are cases where CER uncovered findings
that exceeded expectations. Successes also include what might be considered
negative findings, where research results proved to be not good news for
certain subpopulations but still provided findings that were not previously
known. Whether findings from CER are viewed as positive or negative, the
overarching consideration is that they inform how care is provided.
Conclusions
In closing, it will be important to be mindful of several critical factors
while developing the infrastructure necessary to advance CER. First, more
coordination is needed in setting priorities. Much has been learned from
individual efforts that have taken place both within government and outside
government. What is needed now is to capitalize on these lessons learned
and to begin moving forward together in a more coordinated way to reach
consensus in the setting of priorities for CER. A more systematic approach
to the conduct of CER is also warranted, if only because CER tends to
receive a smaller slice of research funding and so it will be important to be
systematic and strategic in spending limited funds effectively. Coordination
is imperative.
A stronger emphasis on training, methods, and translation is also
needed. These three factors are separate, but they are not inseparable.
Enhanced education is necessary to train the next generation of researchers
on the methodologies of new research designs and on methods of translat-
ing research findings in ways that are actionable, understandable, and not
leading with blunt-edge decisions. At the same time, there needs to be more
robust training targeted to help next-generation investigators work effec-
tively with all relevant stakeholders. More funding is needed, specifically
for well-designed studies that meet priorities, that are not underpowered
and that address meaningful health outcomes. Further, more public–private
partnerships are needed to move CER forward from an implementation
standpoint and to resolve some of the funding problems that heretofore
have hindered CER. Finally, more training is needed on the use of findings
to avoid inappropriate or unintended consequences. Too often, the defini-
OCR for page 148
LEARNING WHAT WORKS
tion of success is whether the results of a given study were published in a
peer-reviewed journal. That focus needs to be shifted to the true heart of
the matter, which is how the findings can best be used in practice and how
they are relevant to decision makers.
As to the future of CER, public–private funding and participation is a
critical necessity for CER to go forward. More effort is needed to develop
designs and protocols that more efficiently and effectively answer CER ques-
tions. This would encompass not merely conducting new methodological
research but also working with stakeholders and users of research as well as
people affected by the research. Public–private funding and participation is
a necessity. Finally, there are a number of important issues that will take a
global approach that need to be addressed. Training on research design and
translation must become an accepted use of healthcare dollars. Wide atten-
tion to vastly improved priority setting, at macro and micro levels, is also
necessary. Transparency across participation is important, so no one gets
unequal access and everyone is at the table. Improved technical assistance
for conducting and implementing CER will also be critical to success.
REFERENCES
AAAI (Association for the Advancement of Artificial Intelligence). 2008. Fall symposium on
automated scientific discovery. November 7-9, 2008. Arlington, VA.
Abiteboul, S., D. Quass, J. McHugh, J. Widom, and J. L. Wiener. 1997. The Lorel query lan-
guage for semistructured data. International Journal of Digital Libraries 1:68-88.
AHRQ (Agency for Healthcare Research and Quality). 2007. Special emphasis notice: AHRQ
announces interest in career development (K) grants focused on comparative effectiveness
research. http://grants.nih.gov/grants/guide/notice-files/NOT-HS-08-003.html (accessed
August 5, 2010).
Avalere Health. 2008. Patient and clinician participation in research agenda setting: Lessons
for future application. Washington, DC: Avalere Health.
Balch, C. M. 2006. Randomized clinical trials in surgery: Why do we need them? Journal of
Thoracic and Cardiovascular Surgery 132(2):241-242.
Barnett, G. O., R. A. Jenders, and H. C. Chueh. 1993. The computer-based clinical record—
Where do we stand? Annals of Internal Medicine 119(10):1046-1048.
Bartlett, P. L., P. J. Bickel, P. Buhlmann, and Y. Freund. 2004. Discussions of boosting papers,
and rejoinders. Annals of Statistics 32:85-134.
Beck, J., P. J. Esser, and M. B. Herschel. 2004. Why clinical trials fail before they even get
started: The “frontloading” process. Quality Assurance Journal 8(1):21-32.
Berners-Lee, T., J. Hendler, and O. Lassila. 2001. The semantic web: A new form of web
content that is meaningful to computers will unleash a revolution of new possibilities.
Scientific American May 17:1-18.
Blackstone, E. H. 2002. Comparing apples and oranges. Journal of Thoracic and Cardiovas-
cular Surgery 123(1):8-15.
———. 2005. Could it happen again? The Bjork-Shiley convexo-concave heart valve story.
Circulation 111(21):2717-2719.
———. 2006. Consort and beyond. Journal of Thoracic and Cardiovascular Surgery 132(2):
229-232.
OCR for page 149
9
THE WORK REQUIRED
———. 2007. From trees to wood or wood to trees? Journal of Surgical Research 147:
59-60.
Braunwald, E. 1983. Effects of coronary-artery bypass grafting on survival. Implications
of the randomized coronary-artery surgery study. New England Journal of Medicine
309(19):1181-1184.
Breiman, L. 1996. Bagging predictors. Machine Learning 24:123-140.
———. 2001. Statistical modeling: The two cultures. Statistical Science 16:199-231.
Buchanan, B. G., and E. H. Shortliffe, eds. 1984. Rule based expert systems: The mycin experi-
ments of the Stanford heuristic programming project. Reading, MA: Addison-Wesley.
Buneman, P., M. Fernandez, and D. Suciu. 2000. UnQL: A query language and algebra
for semistructured data based on structural recursion. Very Large Database Journal
9:76-110.
CBO (Congressional Budget Office). 2007. Research on the comparative effectiveness of
medical treatments. Publication No. 2975. Washington, DC: CBO. http://www.cbo.
gov/ftpdocs/88xx/doc8891/12-18-ComparativeEffectiveness.pdf (accessed September 3,
2010).
Cepeda, M. S., R. Boston, J. T. Farrar, and B. L. Strom. 2003. Comparison of logistic regres-
sion versus propensity score when the number of events is low and there are multiple
confounders. American Journal of Epidemiology 158(3):280-287.
Congressional Research Service (CRS). 2007. Comparative clinical effectiveness and cost-
effectiveness research: Background, history, and overview. Washington, DC: CRS.
Diaconis, P., and B. Efron. 1983. Computer-intensive methods in statistics. Scientific American
June 1:116-130.
Donner, A., and N. Klar. 2000. Design and analysis of cluster randomization trials in health
research. New York: Oxford University Press.
Efron, B. 1979. Bootstrap methods: Another look at the jackknife. Annals of Statistics
7:1-26.
———. 1982. The jackknife, the bootstrap and other resampling plans. Monograph. Phila-
delphia, PA: Society for Industrial and Applied Mathematics.
Efron, B., and R. Tibshirani. 1986. Bootstrap methods for standard errors, confidence inter-
vals, and other measures of statistical accuracy. Statistical Science 1:54-77.
Erkinjuntti, T., T. Ostbye, R. Steenhuis, and V. Hachinski. 1997. The effect of different
diagnostic criteria on the prevalence of dementia. New England Journal of Medicine
337(23):1667-1674.
Etheredge, L. 2007. A rapid-learning health system. Health Affairs 26:w107-w118.
FDA (Food and Drug Administration). 2009. Patient participation in FDA regulatory issues.
http://www.fda.gov/ForConsumers/ByAudience/ForPatientAdvocates/PatientInvolvement/
default.htm (accessed August 5, 2010).
Fleming, T. R., and D. L. DeMets. 1996. Surrogate end points in clinical trials: Are we being
misled? Annals of Internal Medicine 125(7):605-613.
Freund, Y., and R. E. Schapire. 1996. Experiments with a new boosting algorithm. Proceedings
of the International Conference on Machine Learning July:148-156.
Friedman, J. H. 2001. Greedy function approximation: A gradient boosting machine. Annals
of Statistics 29:1189-1232.
———. 2002. Stochastic gradient boosting. Computational Statistics & Data Analysis 38:
367-378.
Gliklich, R. E., and N. A. Dreyer, eds. 2007. Registries for evaluating patient outcomes: A
user’s guide. Rockville, MD: Agency for Healthcare Research and Quality.
Grunkemeier, G. L., R. Jin, and A. Starr. 2006. Prosthetic heart valves: Objective performance
criteria versus randomized clinical trial. Annals of Thoracic Surgery 82(3):776-780.
OCR for page 150
0 LEARNING WHAT WORKS
Gum, P. A., M. Thamilarasan, J. Watanabe, E. H. Blackstone, and M. S. Lauer. 2001. Aspirin
use and all-cause mortality among patients being evaluated for known or suspected coro-
nary artery disease: A propensity analysis. Journal of the American Medical Association
286(10):1187-1194.
Heywood, J. 1546. A dialogue conteynyng the nomber in effect of all the prouerbes in the
englische tongue.
Hunt, S. A., W. T. Abraham, M. H. Chin, A. M. Feldman, G. S. Francis, T. G. Ganiats, M.
Jessup, M. A. Konstam, D. M. Mancini, K. Michl, J. A. Oates, P. S. Rahko, M. A.
Silver, L. W. Stevenson, C. W. Yancy, E. M. Antman, S. C. Smith, Jr., C. D. Adams,
J. L. Anderson, D. P. Faxon, V. Fuster, J. L. Halperin, L. F. Hiratzka, A. K. Jacobs, R.
Nishimura, J. P. Ornato, R. L. Page, B. Riegel, American College of Cardiology, American
Heart Association Task Force on Practice Guidelines, American College of Chest Physi-
cians, International Society for Heart and Lung Transplantation, and Heart Rhythm
Society. 2005. ACC/AHA 2005 guideline update for the diagnosis and management of
chronic heart failure in the adult. American Heart Association 112:145-235.
IDSA (Infectious Diseases Society of America). 2006. Frequently asked questions about Lyme
disease. http://www.idsociety.org/lymediseasefacts.htm (accessed August 5, 2010).
ILADS (International Lyme and Associated Diseases Society) Working Group. 2004. Evidence-
based guidelines for the management of Lyme disease. Expert Review of Anti-Infective
Therapy 2(1):S1-S13. http://www.ilads.org/files/ILADS_Guidelines.pdf (accessed August
5, 2010).
Ilkhanoff, L., J. D. Lewis, S. Hennessy, J. A. Berlin, and S. E. Kimmel. 2005. Potential
limitations of electronic database studies of prescription non-aspirin non-steroidal anti-
inflammatory drugs (NANSAIDS) and risk of myocardial infarction (MI). Pharmacoepi-
demiological Drug Safety 14(8):513-522.
Ioannidis, J. P. 2005. Why most published research findings are false. PLoS Medicine 2(8):
e124.
IOM (Institute of Medicine). 1991 The computer-based patient record: An essential technology
for health care. Washington, DC: National Academy Press.
———. 2007. Learning what works best: The nations need for evidence on comparative
effectiveness in health care. http://www.iom.edu/ebm-effectiveness (accessed August 5,
2010).
———. 2008. Knowing what works in health care: A roadmap for the nation. Washington,
DC: The National Academies Press.
Ishwaran, H. 2007. Variable importance in binary regression trees and forests. Electronic
Journal of Statistics 1:519-537.
Ishwaran, H., and J. S. Rao. 2003. Detecting differentially expressed genes in microar-
rays using Bayesian model selection. Journal of the American Statistical Association
98:438-455.
Ishwaran, H., J. S. Rao, and U. B. Kogalur. 2006. Bamarray: Java software for Bayesian
analysis of variance for microarray data. BMC Bioinformatics 7(1):59.
Jackson, A. 2007. Cyber-enabled discovery and innovation. Notices of the American Math-
ematical Society 54:752-754.
Kamerow, D. 2009. Comparative effectiveness studies inventory project. Washington, DC:
A commissioned activity for the IOM Roundtable on Value & Science-Driven Health
Care.
Kirklin, J. W. 1979. A letter to Helen (Presidential address). Journal of Thoracic and Cardio-
vascular Surgery 78(5):643-654.
Kirklin, J. W., and S. S. Vicinanza. 1999. Metadata and computer-based patient records. An-
nals of Thoracic Surgery 68(3 Suppl.):S23-S24.
Last, J., ed. 1983. A dictionary of epidemiology. New York: Oxford University Press.
OCR for page 151
THE WORK REQUIRED
Lenat, D. B. 2008. The voice of the turtle: Whatever happened to AI? Artificial Intelligence
Magazine 29(2):11-22.
Lenat, D. B., and R. V. Guha. 1990. Building large knowledge-based systems: Representation
and inference in the Cyc project. Reading, MA: Addison-Wesley.
Levy, W. C., D. Mozaffarian, D. T. Linker, S. C. Sutradhar, S. D. Anker, A. B. Cropp, I. Anand,
A. Maggioni, P. Burton, M. D. Sullivan, B. Pitt, P. A. Poole-Wilson, D. L. Mann, and
M. Packer. 2006. The Seattle heart failure model: Prediction of survival in heart failure.
Circulation 113(11):1424-1433.
Merriam-Webster Dictionary. 2004. Springfield, MA: Merriam-Webster.
Mosis, G., B. Koes, J. Dieleman, B. Stricker, J. van der Lei, and M. C. Sturkenboom. 2005a.
Randomised studies in general practice: How to integrate the electronic patient record.
Informatics in Primary Care 13(3):209-213.
Mosis, G., A. E. Vlug, M. Mosseveld, J. P. Dieleman, B. C. Stricker, J. van der Lei, and M. C.
Sturkenboom. 2005b. A technical infrastructure to conduct randomized database stud- stud-
ies facilitated by a general practice research database. Journal of the American Medical
Informatics Association 12(6):602-607.
Mosis, G., J. P. Dieleman, B. Stricker, J. van der Lei, and M. C. Sturkenboom. 2006. A
randomized database study in general practice yielded quality data but patient re-
cruitment in routine consultation was not practical. Journal of Clinical Epidemiology
59(5):497-502.
NCCN (National Comprehensive Cancer Networks). 2008. About the NCCN clinical practice
guidelines in oncology. Available at http://www.nccn.org/professionals/physician_gls/
about.asp (accessed September 8, 2010).
———. 2009. NCCN drugs & biologics compendium. http://www.nccn.org/professionals/
drug_compendium/content/contents.asp (accessed September 8, 2010).
NCI (National Cancer Institute). 2007. Cancer care outcomes research and surveillance con-
sortium. http://outcomes.cancer.gov/cancors (accessed May 30, 2009).
NHLBI (National Heart, Lung, and Blood Institute). 1981. National Heart, Lung, and Blood
Institute coronary artery surgery study. A multicenter comparison of the effects of ran-
domized medical and surgical treatment of mildly symptomatic patients with coronary
artery disease, and a registry of consecutive patients undergoing coronary angiography.
Circulation 63(6 Pt. 2):I1-I81.
NICE (National Institute for Health and Clinical Excellence). 2004. Summary budget state-
ment. www.nice.org.uk/niceMedia/pdf/boardmeeting/brdsep04item12.pdf (accessed July
31, 2004).
NIH (National Institutes of Health). 2006. K30 clinical research curriculum award. http://
grants.nih.gov/training/k30.htm?sort=inst#list (accessed January 17, 2006).
Nissen, S. E. 2006. Adverse cardiovascular effects of rofecoxib. New England Journal of
Medicine 355(2):203-205.
Pickett, J. P., ed. 2000. The American heritage dictionary of the English language, 4th ed.
Boston, MA: Houghton Mifflin.
Reason, J. 1999. Human error. Cambridge, MA: Cambridge University Press.
Rosenbaum, P. R., and D. B. Rubin. 1983. The central role of the propensity score in obser-
vational studies for causal effects. Biometrika 70:41-55.
Rubin, D. B. 2007. The design versus the analysis of observational studies for causal effects:
Parallels with the design of randomized trials. Statistics in Medicine 26(1):20-36.
Sabik, J. F., A. M. Gillinov, E. H. Blackstone, C. Vacha, P. L. Houghtaling, J. Navia, N. G.
Smedira, P. M. McCarthy, D. M. Cosgrove, and B. W. Lytle. 2002. Does off-pump coro-
nary surgery reduce morbidity and mortality? Journal of Thoracic and Cardiovascular
Surgery 124(4):698-707.
OCR for page 152
2 LEARNING WHAT WORKS
Sacristan, J. A., J. Soto, I. Galende, and T. R. Hylan. 1997. A review of methodologies for
assessing drug effectiveness and a new proposal: Randomized database studies. Clinical
Therapeutics 19(6):1424-1517.
Schulz, S., B. Suntisrivaraporn, F. Baader, and M. Boeker. 2009. SNOMED reaching its
adolescence: Ontologists’ and logicians’ health check. International Journal of Medical
Informatics 78(Suppl. 1):S86-S94.
Senge, Peter M. 1990. The fifth discipline: The art and practice of the learning organization.
New York: Doubleday.
Shadish, W., T. Cook, and D. Campbell. 2002. Experimental and quasi-experimental designs
for generalized causal inference. Boston, MA: Houghton Mifflin.
Slutsky, J., and C. Clancy. 2009. AHRQ’s effective health care program: Why comparative
effectiveness matters. American Journal of Medical Quality 24:67-70.
Stewart, W. F., N. R. Shah, M. J. Selna, R. A. Paulus, and J. M. Walker. 2007. Bridging the
inferential gap: The electronic health record and clinical evidence. Health Affairs 26(2):
w181-w191.
Strom, B. L. 2005. Overview of automated data systems in pharmacoepidemiology. In Pharma-
coepidemiology, 4th ed., edited by B. L. Strom. Chichester, UK: John Wiley and Sons.
Sturmer, T., M. Joshi, R. J. Glynn, J. Avorn, K. J. Rothman, and S. Schneeweiss. 2006. A
review of the application of propensity score methods yielded increasing use, advantages
in specific settings, but not substantially different estimates compared with conventional
multivariable methods. Journal of Clinical Epidemiology 59(5):437-447.
Thorn, K. E., A. K. Bangalore, and A. C. Browne. 2007. The UMLS knowledge source
server: An experience in Web 2.0 technologies. AMIA Annual Symposium Proceedings
2007:721-725.
Thygesen, K., J. S. Alpert, and H. D. White. 2007. Universal definition of myocardial infarc-
tion. Journal of the American College of Cardiology 50(22):2173-2195.
Weeks, W., and J. Bagian. 2000. Developing a culture of safety in the Veterans Health Admin-
istration. Effective Clinical Practice 3:270-276.
Zarin, D. A., and T. Tse. 2008. Medicine: Moving toward transparency of clinical trials. Sci-
ence 319(5868):1340-1342.
Zipes, D. P., A. J. Camm, M. Borggrefe, A. E. Buxton, B. Chaitman, M. Fromer, G. Gregoratos,
G. Klein, A. J. Moss, R. J. Myerburg, S. G. Priori, M. A. Quinones, D. M. Roden,
M. J. Silka, C. Tracy, S. C. Smith, Jr., A. K. Jacobs, C. D. Adams, E. M. Antman, J. L.
Anderson, S. A. Hunt, J. L. Halperin, R. Nishimura, J. P. Ornato, R. L. Page, B. Riegel,
J. J. Blanc, A. Budaj, V. Dean, J. W. Deckers, C. Despres, K. Dickstein, J. Lekakis, K.
McGregor, M. Metra, J. Morais, A. Osterspey, J. L. Tamargo, and J. L. Zamorano. 2006.
ACC/AHA/ESC 2006 guidelines for management of patients with ventricular arrhythmias
and the prevention of sudden cardiac death: A report of the American College of Cardiol-
ogy/American Heart Association task force and the European Society of Cardiology com-
mittee for practice guidelines (Writing committee to develop guidelines for management
of patients with ventricular arrhythmias and the prevention of sudden cardiac death).
Journal of the American College of Cardiology 48(5):e247-e346.
