Click for next page ( 88


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 87
2 The Work Required INTRODUCTION Comparative effectiveness research (CER) is composed of a broad range of activities. Aimed at both individual patient health and overall health system improvement, CER assesses the risks and benefits of compet- ing interventions for a specific disease or condition as well as the system- level opportunities to improve health outcomes. To meet the ultimate goal of providing information that is useful to guide the healthcare decisions of patients, providers, and policy makers, the work required includes con- ducting primary research (e.g., clinical trials, epidemiologic studies, simu- lation modeling); developing and maintaining data resources in order to conduct primary research, such as registries or databases for data mining and analysis, or to enhance the conduct of other types of clinical research; and synthesizing and translating a body of existing research via systematic reviews and guideline development methods. To ensure the best return on investment in these efforts, work is also needed to advance the develop- ment of new or refined study methodologies that improve the efficiency and relevance of research as well as reduce its costs. Similarly, to guide the overall clinical research enterprise in the efficient production of information of true value to healthcare decision makers requires the identification of priority research questions that need to be addressed, the coordination of the various aspects of evidence development and translation work, and the provision of technical assistance, such as study design and validation. The papers that follow provide an overview of the nature of the work required, noting lessons learned about the known benefits of the country’s 

OCR for page 87
 LEARNING WHAT WORKS capacity and experience, and illustrating opportunities to improve care through capacity building. Emerging from these papers is the notion that although a number of diverse, innovative, and talented organizations are engaged in various aspects of this work, additional efforts are needed. Gains in efficiency are possible with improved coordination, prioritization, and attention to the range of methods that can be employed in CER. Two papers provide a sense of the potential scope and scale of the necessary CER. Erin Holve and Patricia Pittman from AcademyHealth estimate that approximately 600 comparative effectiveness studies were ongoing in 2008, including head-to-head trials, pragmatic trials, observa- tional studies, evidence syntheses, and modeling. Costs for these studies range broadly, but cluster according to study design. Challenges to develop the workforce needed for CER suggest the need for greater attention to infrastructure for training and funding researchers. Providing a sense of the overall need for comparative effectiveness studies, Douglas B. Kamerow from RTI International discusses the work of a stakeholder group to develop a prioritization process for CER topics and some possible criteria for prioritizing evidence needs. This process yielded 16 candidate research topics for a national inventory of priority CER questions. Possible pitfalls of such an evaluation and ranking process are discussed. Three papers provide an overview of the work needed to support, develop, and synthesize research. Jesse A. Berlin and Paul E. Stang from Johnson & Johnson survey data resources for research, and discuss how appropriate use of data and creative uses of data collection mechanisms are crucial to help inform healthcare decision making. Given the described strengths and limitations of available data, current systems are primarily resources for the generation and strengthening of hypotheses. As the field transitions to electronic health records (EHRs) however, the value of these data could dramatically increase as targeted studies and data capture capa- bilities are built into existing medical care databases. Richard A. Justman, from the United Health Group, discusses the challenges of evidence synthe- sis and translation as highlighted in a recent Institute of Medicine (IOM) report (2008). Limitations of evidence synthesis and translation have led to gaps, duplications, and contradictions; and, key findings and recommenda- tions from a recent IOM report provide guidance on infrastructure needs and options for systematic review and guideline development. Eugene H. Blackstone, Douglas B. Lenat, and Hemant Ishwaran from the Cleveland Clinic discuss five foundational methodologies that need to be refined or further developed to move from the current siloed, evidence-based medicine (EBM) to semantically integrated, information-based medicine and on to predictive personalized medicine—including reengineered randomized con- trolled trials (RCTs), approximate RCTs, semantically exploring disparate

OCR for page 87
9 THE WORK REQUIRED clinical data, computer learning methods, and patient-specific strategic decision support. Finally, Jean R. Slutsky from the Agency for Healthcare Research and Quality (AHRQ) provides an overview of organizations conducting CER activities and reflects on the importance of coordination and technical assistance capacities to bridge these activities. Particular attention is needed as to which functions might be best supported by centralized versus local, decentralized approaches THE COST AND VOLUME OF COMPARATIVE EFFECTIVENESS RESEARCH Erin Holve, Ph.D., M.P.H., Director; Patricia Pittman, Ph.D., Executive Vice President, AcademyHealth Overview In the ongoing discussion about CER, there has been limited under- standing of the current capacity for conducting CER in the United States. This report intends to help fill this gap by providing an environmental scan of the volume and the range of costs of recent CER. This work was funded by the California HealthCare Foundation. Current production of CER is not well understood, perhaps due to the relatively new use of the term, or perhaps as a result of fragmented funding streams. Comparative Effectiveness Research Environmental Scan This study sought to determine whether there is a significant body of CER under way so that policy makers interested in improving outcomes can plan appropriate initiatives to bolster CER in the United States. This study does not catalog the universe of CER because existing data sources are limited by the way that research databases are developed. However, it is a first attempt to assess the approximate volume and cost of CER. The study focused on four major objectives: Identify a typology of CER design. 1. Characterize the volume of research studies that address compara- 2. tive effectiveness questions. Provide a range of cost estimates for conducting comparative effec- 3. tiveness studies by type of study design. Gather information on training to support the capacity to produce 4. CER.

OCR for page 87
90 LEARNING WHAT WORKS These efforts relied on three modes of data collection. The first phase included the development of a framework of study designs and topics. The second consisted of a structured search of research projects listed in two databases: www.clinicaltrials.gov and Health Services Research Projects in Progress (HSRProj),1 a database of health services research projects in progress. The third consisted of in-person and telephone interviews with 25 research organizations identified as leaders in comparative effectiveness studies. The number, type, and costs of studies were noted, although only studies cited by funders were included in the estimates of volume because of the possibility of double-counting studies cited by both researchers and funders. Interviews were used to triangulate information on costs and the relative importance of different designs. An important study limitation was that it was not possible to cross- reference the databases and the interviews. As a result, although these sources are comparable, they should not be aggregated. In an initial focus group with experts on CER, a definition of CER was developed to guide the project. Though many definitions of CER have been developed,2 this project relies on the following: • CER is a comparison of the effectiveness of the risks and benefits of two or more healthcare services or treatments used to treat a spe- cific disease or condition (e.g., pharmaceuticals, medical devices, medical procedures, other treatment modalities) in approximate real-world settings • The comparative effectiveness of organizational and system-level strategies to improve health outcomes is excluded from this defini- tion, as is research that is clearly “efficacy” research. This means that studies that compare an intervention to placebo or to usual care were excluded from our counts. The expert panel also developed a framework of research designs to make it possible to categorize findings systematically. For the purposes of this study there was general agreement that there are three primary research categories3 applicable to CER: 1 HSRProj may be accessed at www.nlm.nih.gov/hsrproj/ (accessed September 22, 2010). 2 In a recent report from the Congressional Budget Office, the authors state that compara- tive effectiveness is “simply a rigorous evaluation of the impact of different treatment options that are available for treating a given medical condition for a particular set of patients” (CBO, 2007). An earlier report by the Congressional Research Service makes an additional distinction that comparative effectiveness is “one form of health technology assessment” (CRS, 2007). 3 During the interviews we attempted to identify research by more specific types, asking questions about pragmatic trials, registry and modeling studies, and systematic reviews.

OCR for page 87
9 THE WORK REQUIRED 1. head-to-head trials (including pragmatic trials); 2. observational studies (including registry studies, prospective cohort studies, and database studies); and 3. syntheses and modeling (including systematic reviews).4 Clinicaltrials.gov is the national registry of data on clinical trials, as mandated by the Food and Drug Administration (FDA) reporting process required for drug regulation. Clinicaltrials.gov includes more than 53,000 study records and theoretically provides a complete set of information on all clinical trials of drugs, biologics, and devices subject to FDA regula- tions (Zarin and Tse, 2008). While the vast majority of trials included in clinicaltrials.gov are controlled experimental studies, there are some obser- vational studies as well. HSRProj is a database of research projects related to healthcare access, cost, and quality as well as the performance of healthcare systems. Some clinical research may be included in HSRProj if it is focused on effective- ness. HSRProj includes a variety of public and private organizations but only a limited number of projects funded by private or industry sources. A search was conducted in www.clinicaltrials.gov for phase 3 and phase 4 observational and interventional studies. Phase 4 studies were narrowed by searching only for studies containing the term effectiveness. Studies that were explicitly identified as efficacy studies or those that did not include at least two active comparators were excluded. The HSRProj database was also searched for studies on effectiveness. Because HSRProj does not dif- ferentiate between study design phases, a search was also conducted by the types of studies identified in the framework. The studies identified through the process of searching both databases were then searched by hand in order to identify projects that met the definition of CER. The interview phase of the project included in-person and telephone interviews with research funders and researchers who conduct CER. An initial sample of individuals funding and conducting CER were contacted in response to recommendations by the focus group panel, and these initial 4 Observational research studies include a variety of research designs but are principally defined by the absence of experimentation or random assignment (Shadish et al., 2002). In the context of CER, cohort studies and registry studies are generally thought of as the most common study types. Prospective cohort studies follow a defined group of individuals over time, often before and after an exposure of interest, to assess their experience or outcomes (Last, 1983), while retrospective cohort studies frequently use existing databases (e.g., medical claims, vital health records, survey records) to evaluate the experience of a group at a point or period in time. Registry studies are often thought of as a particular type of cohort study based on patient registry data. Patient registries are organized systems using observational study methods to collect patient data in a uniform way. These data are then used to evaluate specific outcomes for a population of interest (Gliklich and Dreyer, 2007).

OCR for page 87
92 LEARNING WHAT WORKS contacts in turn recommended other respondents—a “snowball” sample. In total, 35 individuals representing 25 research funders or research organiza- tions participated in the project.5 Findings For the project, 689 comparative effectiveness studies were identified in clinicaltrials.gov and HSRProj. Of these the vast majority are “interven- tional trials” listed on clinicaltrials.gov; specifically, they are phase 3 trials that compare two or more treatments “head to head,” have real-world ele- ments in their study design, and do not explicitly include efficacy in their description of the study design. Only 19 studies are phase 4 post-marketing studies that compare multiple treatments. Seventy-three CER projects were identified in HSRProj. The process of manually searching project titles confirmed that most studies in this database were observational research, although a handful of studies were specifically identifiable as registry stud- ies, evidence synthesis, or systematic reviews. The interviews with funders identified 617 comparative effectiveness studies, of which approximately half were observational studies (prospec- tive cohort studies, registry studies, and database studies). Research synthe- ses (reviews and modeling studies) and experimental head-to-head studies also represent a significant proportion of research activities. Neither clinicaltrials.gov nor HSRProj publish funding amounts, so interviews with funders and researchers are the sole source of data on cost. As would be expected across the range of study designs covered, there is an extremely broad array of cost for CER studies. However, despite the range, cost estimates provided in the interviews did tend to cluster, particularly by study type. While the cost of conducting head-to-head randomized trials was extremely broad, including studies as costly as $125 million, smaller trials tended to range from $2.5 million to $5 million and larger studies from $15 million to $20 million. Likewise, the range of cost for observational studies was extremely broad but tended to cluster (Table 2-1). While large prospective cohort studies cost $2 million to $4 million, large registry studies cost between $800,000 and $6 million, with most examples falling at the higher end of this range, although a few were substantially less. Retrospective database studies tended to be less expensive and cost on the order of $100,000 5 Though an initial group of potential respondents was identified as funders of CER, it was often necessary to speak with multiple individuals to find the appropriate person or group re- sponsible for CER within the organization’s portfolio. For this reason the response rate among individuals is lower than might be expected for a series of key informant interviews. Thirteen organizations were identified that did not suggest they received funding for CER from other sources. This subset is used as the sample of organizations that fund or self-fund CER.

OCR for page 87
9 THE WORK REQUIRED TABLE 2-1 Costs of Various Comparative Effectiveness Studies Type of Study Cost Head to head Randomized controlled trials: Smaller $2.5m–$5m Larger $15m–$20m Observational Registry studies $2m–$4m Large prospective cohort studies $800k–$6m Small retrospective database studies $100k–$250k Synthesis Simulation/modeling studies $100k–$200k Systematic reviews $200k–$350k to $250,000. Systematic reviews and modeling studies tended to be less expensive and have a far narrower range of cost, in part because these studies were based on existing data, with many falling between $100,000 to $200,000. It is important to note, however, that this may not include the cost of procuring data. Systematic reviews cluster around a range of $200,000 to $350,000. There are, of course, additional activities and costs of involving stake- holders in research agenda setting as well as prioritizing, coordinating, and disseminating research on CER that are not included here.6 These impor- tant investments will need to be considered in the process of budgeting for CER.7 6 Examples of activities designed to prioritize and coordinate research activities include the National Cancer Institute’s CER Cancer Control Planet (http://cancercontrolplanet.cancer. gov/), which serves as a community resource to help public health professionals design, imple- ment, and evaluate CER-control efforts (NCI, 2007). Within the the Agency for Healthcare Research and Quality, the prioritization and research coordination efforts for comparative effectiveness studies are undertaken as part of the Effective Health Care Program. Translation and dissemination of CER findings is handled by the John M. Eisenberg Clinical Decisions and Communications Science Center, which aims to translate research findings to a variety of stakeholder audiences. No budget information is readily available for the Eisenberg Center activities. 7 Examples of stakeholder involvement programs include two programs at the FDA focused on involving patient stakeholders, the Patient Representative Program and the comparative

OCR for page 87
9 LEARNING WHAT WORKS Finally, the interviews also shed light on two subjects discussed at this workshop (IOM, 2008): (1) the need for additional training in the meth- ods and conduct of CER; and (2) the need to bring researchers together to discuss the relative contributions of RCTs and observational studies in the context of CER. While interviewees generally commented that they have some capac- ity to respond to an increase in the demand for CER, some noted that they have had difficulty finding adequately trained researchers to conduct such research. For the moment, training programs are limited primarily to research trainees working with AHRQ’s evidence-based practice cen- ters and the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) network. Respondents also mentioned two postdoctoral train- ing programs designed to teach researchers how to conduct effectiveness research. The Duke Clinical Research Institute (DCRI) offers a program for fellows and junior faculty. Fellows studying clinical research may take additional coursework and receive a masters of trial services degree in clini- cal research as part of the Duke Clinical Research Training Program. The Clinical Research Enhancement through Supplemental Training (CREST) program at Boston University is the second program mentioned. CREST trains researchers in aspects of clinical research design, including clinical epidemiology, health services research (HSR), biobehavioral research, and translational research. Both the DCRI and CREST programs have a strong emphasis on clinical research using both randomized experimental study designs and observational designs. Other funding for training includes the National Institutes of Health (NIH) K30 awards to support career development of clinical investigators and to develop new modes of training in theories and research methods. The program’s goal is to produce researchers capable of conducting patient- oriented research on epidemiologic and behavioral studies and on outcomes or health services research (NIH, 2006). As of January 2006, the Office of Extramural Affairs lists 51 curriculum awards funded through the K30 mechanism. In November 2007 AHRQ released a Special Emphasis Notice effectiveness research Drug Development Patient Consultant Program (Avalere Health, 2008; FDA, 2009). Other examples of stakeholder involvement programs include the National Insti- tute for Occupational Safety and Health–National Occupational Research Agenda program, the American Thoracic Society Public Advisory Roundtable, and the National Institutes of Health director’s Council of Public Representatives (COPR). These efforts can represent a sizeable investment in order to assure stakeholder involvement among the potentially diverse group of end users. For example, the COPR is estimated to cost approximately $350,000 per year (Avalere Health, 2008). From an international perspective, the UK’s National Institute for Health and Clinical Excellence (NICE) allocates approximately 4 percent of their annual budget (approximately $775,000) in NICE’s Citizen’s Council and for their “patient involve- ment unit” (NICE, 2004).

OCR for page 87
9 THE WORK REQUIRED for Career Development (K) Grants focused on CER (AHRQ, 2007). Four career awards are slated to support the development, generation, and trans- lation of scientific evidence by enhancing the understanding and develop- ment of methods used to conduct CER. The challenge of filling the “pipeline” of researchers working on com- parative effectiveness is exacerbated by what many of the interviewees viewed as a fundamental philosophical difference between researchers who were academically trained in observational research and those who are trained on the job to conduct clinical trials. Several respondents noted that these differences likely arise because the majority of researchers are trained in either observational study methods or randomized trials, but rarely both. In addition, as noted by Hersh and colleagues,8 there are many unknowns related to assessing the workforce needs for CER, including the unresolved definitional issues and scope of comparative effectiveness. As noted by Hersh, the proportion of CER that is focused on randomized trials, obser- vational research, and syntheses has strong implications for the number of researchers (and the type of training) that will be needed. For this reason it is important to track research production and funding for CER in order to anticipate future needs. Some respondents noted that differences in training manifest themselves in disagreements about the benefits of various observational study designs. Nevertheless, most individuals interviewed as part of this study felt that RCTs, observational studies (including registry studies, prospective cohort studies, and quasi-experiments), and syntheses (modeling studies and sys- tematic reviews) are complementary strategies to generate useful informa- tion to improve the evidence base for health care. Furthermore, many participants agreed that, as CER evolves, it will be critical to develop a balanced research portfolio that builds on the strengths of each study type. To facilitate this balance, some of the interviews, as well as many comments at the IOM’s Roundtable on Value & Science-Driven Health Care (IOM, 2008) suggest that training opportunities to bridge gaps in language and methods used by researchers may be helpful in creating a balanced portfolio of CER. Conclusion Findings from this study indicate that there is a greater volume of ongo- ing CER than may initially have been supposed. The cost of conducting this research varies greatly, although it tends to cluster by type of study. 8 Hersh, B., T. Carey, T. Ricketts, M. Helfand, N. Floyd, R. Shiffman, and D. Hickam. A framework for the workforce required for comparative effectiveness research. See Chapter 4 of this publication.

OCR for page 87
9 LEARNING WHAT WORKS The range of studies that are currently being conducted and the cost varia- tion by study type have implications for the mix of activities that may be undertaken by an entity focused on CER. Furthermore, as identified in the interviews, assuring sufficient research capacity to conduct CER is likely to require an investment in multidisciplinary training, with an emphasis on bridging the gap between awareness of the strengths and limitations of randomized trials, observational study designs, and syntheses. INTERVENTION STUDIES THAT NEED TO BE CONDUCTED Douglas B. Kamerow, M.D., M.P.H. Chief Scientist, Health Services and Policy Research, RTI International, and Professor of Clinical Family Medicine, Georgetown University Overview CER compares the impact of different treatments for medical condi- tions in a rigorous, practical manner. At the request of the IOM’s Round- table on Value & Science-Driven Health Care, IOM staff convened in 2008 a multisectoral working group to create a national priority assessment inventory. Their charge was to set criteria for choosing appropriate CER topics and then to nominate and review example topics for needed research. An abridged summary of the report is presented in Appendix B. Appendixes C and D, respectively, are the recommended priority CER studies proposed in 2009 by the IOM Committee on Comparative Effectiveness Research Prioritization and the Federal Coordinating Council for Comparative Effec- tiveness Research. Introduction CER has been defined as “rigorous evaluation of the impact of differ- ent options that are available for treating a given medical condition for a particular set of patients,” (CBO, 2007) and “the direct comparison of existing healthcare interventions to determine which work best for which patients and which pose the greatest benefits and harms” (Slutsky and Clancy, 2009). Broadly construed, this type of research can involve com- parisons of different drug therapies or devices used to treat a particular condition as well as comparisons of different modes of treatment, such as pharmaceuticals versus surgery. It can also be used to compare different systems or locations of care and varied approaches to care, such as dif- ferent intervals of follow-up or changes in medication dosing levels for a particular condition. CER also may be used to investigate diagnostic and

OCR for page 87
9 THE WORK REQUIRED preventive interventions. All of these studies may include an evaluation of costs as well as an assessment of clinical effectiveness. Comparative assessment research is especially valuable now, in an era of unprecedented healthcare spending and large, unexplained variations in care for patients with similar conditions. The IOM’s Roundtable on Value & Science-Driven Health Care set a goal for the year 2020 that 90 percent of clinical decisions will be supported by accurate, timely, and up-to-date clinical information that reflects the best available evidence. Currently, there is insufficient evidence about which treatments are most appropri- ate for certain groups of patients and whether or not those treatments merit their sometimes significant costs. The healthcare community requires improved evidence generation that will address how different interventions compare to one another when applied to target patient populations. CER provides the opportunity to ground clinical care in a foundation of sound evidence. The current system usually ensures that when a new drug or device is made available, there is evidence to show its effectiveness compared to a placebo in ideal research conditions—that is, its efficacy. But there is often an insufficient body of evidence demonstrating its relative effectiveness compared to existing or alternative treatment options, especially in real- world settings. This limited scope of information increases the likelihood that clinical decisions are not based on evidence but rather on local practice style, institutional tradition, or physician preference. Although the numbers vary, some estimate that less than half—perhaps well less than half—of all clinical decisions are supported by sufficient evidence (IOM, 2007). This lack of evidence also leads to substantial geographic variations in care, further supporting the idea that patients may be subjected to treatments that are unnecessarily invasive—or not aggressive enough—for a variety of conditions. These variations in care partly explain healthcare spend- ing differences across geographic regions that cannot be fully accounted for by price differences or illness rates. Geographic variations in treat- ment approach are often greater when there is less agreement within the medical community about the appropriate treatment. Variation in treatment approach for a variety of conditions is of significant concern because it has not been demonstrated that areas with higher levels of spending—where presumably patients are treated with more aggressive or expensive options or with simply more treatment—have significantly better health outcomes than areas with lower levels of spending. The Institute of Medicine and Comparative Effectiveness The IOM’s Roundtable on Value & Science-Driven Health Care rec- ognized the importance of furthering CER to ensure that all clinical deci-

OCR for page 87
2 LEARNING WHAT WORKS Summary Moving beyond today’s Mt. Everest level of difficulty, RCTs need to become more nimble and simple to better reflect the real world and to have their financing restructured. Heterogeneity in practice facilitates approximate randomized trials via propensity score methods that are inex- pensive and widely accessible but which require patient-level clinical data stored as discrete values for variables. Emerging semantic technology can be exploited to integrate currently disparate, siloed medical data—responding to investigators’ complex queries and patients’ imprecise ones—and in the near future holds the promise to automate discovery of unsuspected relationships and unintended adverse or surprisingly beneficial outcomes. A next generation of analytic tools for revealing patterns in clinical data should build on successful methods developed in the discipline of machine learning. Both new knowledge learned and resulting algorithms should be transformed into strategic decision support tools. These are but a few concrete examples of methods that need to be developed to provide an infrastructure to determine the right treatment for the right patient at the right time. Resources Needed What resources are needed to develop this infrastructure? Reengineering Randomized Controlled Trials The cost of an NIH-sponsored simple trial appears to be in the range of $2 million, but multi-institutional, multinational large trials driven by clinical end points can consume 10 times that figure. If one uses $100 mil- lion as a metric, this means 5 to 50 such trials of therapy can be supported. Considering all the therapies of medicine for which the evidence base is weak, it is clear that demanding gold-standard RCTs for everything is unaf- fordable. The cost of RCTs that are highly focused, ethically unambiguous, and feasible could be brought down to a quarter, perhaps even a tenth, of this figure based on practical experience. This will require maximum use of electronic patient records, consisting of values for variables, and quite specifically longitudinal surveillance data to study the long-term side effects of therapies. Approximate Randomized Controlled Trials The NIH and National Science Foundation (NSF) should join forces and solicit 3-year methodology grants of approximately $250,000 per year,

OCR for page 87
 THE WORK REQUIRED 10 per year. For this $7.5 million investment, a strong understanding of how best to use nonrandomized data would emerge. With this would come production of publicly available statistical software. If rich discrete clinical data were available for analysis, a typical study using these methods for nonrandomized comparison would cost approxi- mately $75,000. The cost would double if extensive integration of data was necessary, possibly over healthcare networks. For $100 million, it would be possible to conduct more than 1,000 such approximate randomized trials. This could have a major impact on acquiring what might be called “silver- level” evidence for practice. Semantically Integrating, Querying, and Exploring Disparate Clinical Data Based on several years of work, it seems that developing a comprehen- sive ontology of medicine—a new framework for analysis across disparate medical domains—will cost about 1 hour of time per term for an analyst, programmer, and clinical expert. One need not start from scratch, but can exploit SNOMED, UMLS, and other term lists and ontologies to start the process. Assuming that 100,000 terms would need to be defined in this fashion, that the wages would be $300 per hour, and that 25 ontologists would be needed, this work could be completed in 2 years at a cost of $36 million. This would include the software that must be programmed to implement a global effort in rallying medical experts to this task. Computer Learning Methods Knowledge discovery in medicine involves both methodologic devel- opment and applications. These should go hand in hand in this new field because it would accelerate the development of methods as they encounter problems requiring further methodologic work. The NSF has begun an initiative called Cyber-Enabled Discovery and Innovation (Jackson, 2007). This began with a $52 million first-year budget and is intended to ramp up $50 million per year and finish within 5 years for a total of $750 mil- lion. It would be useful to add $10 million per year for direct application to biomedicine, for a total sustained level for these activities within 5 years of $50 million. Patient-Specific Strategic Decision Support Costs in this area are largely for developing software, including the interfaces to EMR systems. This could be done for approximately $10 million. One could envision every study of clinical effectiveness having a

OCR for page 87
 LEARNING WHAT WORKS patient-specific prediction component built into it. Again, based on experi- ence doing this, approximately $25,000 per study would be required to adapt and test the software and couple it with EMRs for decision support. It is also likely that at some point, the FDA may become involved with tools such as this and would introduce regulations that are more costly to meet than those of performing the studies. COORDINATION AND TECHNICAL ASSISTANCE THAT NEED TO BE SUPPORTED Jean R. Slutsky, Director, Center for Outcomes and Evidence, Agency for Healthcare Quality and Research Overview CER as a concept and reality has grown rapidly in the past 5 years. While it builds on an appreciation for the role of technology assessment, comparative study designs, and the increased role of health information technology to gather evidence and distribute it to the point of care, the capacity and infrastructure for this research has received less targeted attention. Understanding the landscape of organizations and health sys- tems undertaking CER is challenging but essential. Without knowing what capacities and infrastructure currently exist, rational strategic planning for the future cannot be done. It is also important to address which functions can be most effective if they are centralized, which are most effective if they are local or decentralized, and how different activities relate to each other in a productive way. This paper will explore the practical realities of what exists now, what is needed for the future, and how the needs of the country’s diverse healthcare system for CER can best be met. The Agency for Healthcare Research and Quality Perspective AHRQ plays a significant role in CER. Under a mandate included in Section 1013 of the Medicare Prescription Drug, Improvement, and Mod- ernization Act of 2003, AHRQ is the lead agency for CER in the United States. AHRQ conducts health technology assessment at the request of CMS and analyzes data and suggests options for coverage with evidence development (CED) and post-CED data collection. AHRQ also provides translation of CER findings, promotes and funds comparative effective- ness methods research, and funds training grants focused on comparative effectiveness. AHRQ has an annual budget of over $300 million ($372 million for 2009), and received funds specifically for work on CER ($30

OCR for page 87
 THE WORK REQUIRED million and $50 million in 2008 and 2009, respectively).14 AHRQ has built a flexible, dynamic infrastructure for CER that includes 41 research cen- ters nationwide with more than 160 researchers. The program includes K awards for career development such as the Mentored Research or Clinical Scientist Development awards, and Independent Scientist awards. AHRQ has also funded methods research that heretofore had not been funded except on an ad hoc basis. AHRQ has also put concentrated funding into the translation of CER findings. Given the pressing need for evidence, it is important to keep in mind the high costs of precision. It costs money to conduct precise studies that answer detailed questions, which speaks to the importance of not only pri- ority setting but also understanding that this is an important and difficult task. In the United States, the landscape for CER is, for the most part, very well intentioned. All parties engaged in this work want to do the right thing and see what works best for patients in the United States and else- where. Nonetheless, current efforts are too ad hoc in nature. There are no adequate organizing principles, except for those outlined in Section 1013, where language in the legislation focuses on setting priorities, hav- ing transparent processes, involving stakeholders, and having a translation component. The effect is that in the United States there is essentially only a very limited capacity to conduct CER and to translate that research into meaningful and useful applications. The United States is not accustomed, nor organized effectively, to conduct this type of research. In part, this is because the system tends not to grow researchers who have the capacity to move beyond what might be described as a parochial mind set regard- ing the types of research study designs, a mind set which has limited the capacity to readily generate hypotheses and study designs appropriate for CER. Generally, researchers do not involve stakeholders to the extent that is required for research aimed at generating information to guide end users such as patients and physicians. A key shift needed in the current approach to research is to involve patients and other key stakeholders, such as indus- try and health plans, in the formulation of questions for investigation and in study design. As mentioned above, AHRQ is currently conducting CER under leg- islative mandate. Other federal agencies also conducting CER include the NIH, CMS (CED), and the VA; some have done so for decades. In the private sector, health plans and industry are also engaged in CER. In addi- tion, CER-focused public–private partnerships are starting to form, as are private–private partnerships. Unlike other forms of research, CER will most certainly require the kinds of partnerships that are now emerging. 14 See http://www.ahrq.gov/about/budgtix.htm (accessed September 8, 2010).

OCR for page 87
 LEARNING WHAT WORKS There are a number of common pitfalls in CER. One of the most sig- nificant failures is that comparative effectiveness studies are not designed in ways that capture meaningful end points as well as longitudinal and rel- evant outcomes—end points that would be meaningful not only to patients but also to decision makers. There are also significant issues about the applicability of the CER studies that are conducted, and the work we do for CMS is reflective of the need for research conducted in patients repre- sentative of the Medicare population. The elderly tend not to be studied in rigorous trials to the extent that children are, although AHRQ has begun to address this discrepancy through, for example, work for the Medicaid and State Children’s Health Insurance programs. There is also a failure to clini- cally address relevant heterogeneity and biological heterogeneity as well. Finally, there are also responsibility issues, as evidenced in discussions of “who pays and who stays” when the need for an important study has been noted. There is currently discussion of such issues on Capitol Hill. Today’s reality in CER, therefore, can be summarized as follows. There is general sentiment that CER can be a positive thing if it is done fairly, is well designed, and is transparent. This is important because of the potential impact of CER on many different sectors—not just patients, but also indus- try and health plans. If CER is not conducted in a way that stakeholders can understand—and, importantly, in a way by which they have input into the process—it could happen that CER does not have the impact, in terms of improving health outcomes, that everyone hopes it will have. As AHRQ discovered in developing the Section 1013 healthcare program, involving stakeholders early, listening to them, and involving them throughout the process through to the end and implementation of the findings, is critically important. Another issue with CER today is that there is no agreement on the best methods for setting priorities. Everyone tries to set priorities finds that reaching consensus is hugely difficult. In part that is because the process of setting such priorities tends to revert to personal or narrowly defined considerations. If a consensus is to be reached, rather than thinking about individual priorities, it will be important to instead learn to focus on national priorities. Experience shows, however, that such priority shifting is difficult. Another aspect of the reality of CER today that has been disappointing is that there has been less emphasis on designing good studies than on just the concept of CER. A great deal of time is spent talking about where CER should live, what it should look like, and how it should be funded, but there has tended not to be adequate discussion about how to conduct CER most effectively from a methodological and implementation standpoint. Further discussion is needed on both sides of this issue—discussion not only about which box CER lives in, if you will, but about what’s inside the box.

OCR for page 87
 THE WORK REQUIRED Once a decision to perform a CER study is made, finding a payer or a funder for the research component is difficult. Obviously, with a budget of $30 million AHRQ cannot be the sole funder of many of these stud- ies. Regardless, the reality is that the challenges related to funding lead to less rigorous and less innovative study designs. Somehow the issue of how different players collaborate to fund these studies must be addressed. Otherwise, the rigorous study designs needed to move this field forward will never be developed. Successes, however, make the fundamental concept of CER worthwhile. Among the successes, for example, are cases where CER uncovered findings that exceeded expectations. Successes also include what might be considered negative findings, where research results proved to be not good news for certain subpopulations but still provided findings that were not previously known. Whether findings from CER are viewed as positive or negative, the overarching consideration is that they inform how care is provided. Conclusions In closing, it will be important to be mindful of several critical factors while developing the infrastructure necessary to advance CER. First, more coordination is needed in setting priorities. Much has been learned from individual efforts that have taken place both within government and outside government. What is needed now is to capitalize on these lessons learned and to begin moving forward together in a more coordinated way to reach consensus in the setting of priorities for CER. A more systematic approach to the conduct of CER is also warranted, if only because CER tends to receive a smaller slice of research funding and so it will be important to be systematic and strategic in spending limited funds effectively. Coordination is imperative. A stronger emphasis on training, methods, and translation is also needed. These three factors are separate, but they are not inseparable. Enhanced education is necessary to train the next generation of researchers on the methodologies of new research designs and on methods of translat- ing research findings in ways that are actionable, understandable, and not leading with blunt-edge decisions. At the same time, there needs to be more robust training targeted to help next-generation investigators work effec- tively with all relevant stakeholders. More funding is needed, specifically for well-designed studies that meet priorities, that are not underpowered and that address meaningful health outcomes. Further, more public–private partnerships are needed to move CER forward from an implementation standpoint and to resolve some of the funding problems that heretofore have hindered CER. Finally, more training is needed on the use of findings to avoid inappropriate or unintended consequences. Too often, the defini-

OCR for page 87
 LEARNING WHAT WORKS tion of success is whether the results of a given study were published in a peer-reviewed journal. That focus needs to be shifted to the true heart of the matter, which is how the findings can best be used in practice and how they are relevant to decision makers. As to the future of CER, public–private funding and participation is a critical necessity for CER to go forward. More effort is needed to develop designs and protocols that more efficiently and effectively answer CER ques- tions. This would encompass not merely conducting new methodological research but also working with stakeholders and users of research as well as people affected by the research. Public–private funding and participation is a necessity. Finally, there are a number of important issues that will take a global approach that need to be addressed. Training on research design and translation must become an accepted use of healthcare dollars. Wide atten- tion to vastly improved priority setting, at macro and micro levels, is also necessary. Transparency across participation is important, so no one gets unequal access and everyone is at the table. Improved technical assistance for conducting and implementing CER will also be critical to success. REFERENCES AAAI (Association for the Advancement of Artificial Intelligence). 2008. Fall symposium on automated scientific discovery. November 7-9, 2008. Arlington, VA. Abiteboul, S., D. Quass, J. McHugh, J. Widom, and J. L. Wiener. 1997. The Lorel query lan- guage for semistructured data. International Journal of Digital Libraries 1:68-88. AHRQ (Agency for Healthcare Research and Quality). 2007. Special emphasis notice: AHRQ announces interest in career development (K) grants focused on comparative effectiveness research. http://grants.nih.gov/grants/guide/notice-files/NOT-HS-08-003.html (accessed August 5, 2010). Avalere Health. 2008. Patient and clinician participation in research agenda setting: Lessons for future application. Washington, DC: Avalere Health. Balch, C. M. 2006. Randomized clinical trials in surgery: Why do we need them? Journal of Thoracic and Cardiovascular Surgery 132(2):241-242. Barnett, G. O., R. A. Jenders, and H. C. Chueh. 1993. The computer-based clinical record— Where do we stand? Annals of Internal Medicine 119(10):1046-1048. Bartlett, P. L., P. J. Bickel, P. Buhlmann, and Y. Freund. 2004. Discussions of boosting papers, and rejoinders. Annals of Statistics 32:85-134. Beck, J., P. J. Esser, and M. B. Herschel. 2004. Why clinical trials fail before they even get started: The “frontloading” process. Quality Assurance Journal 8(1):21-32. Berners-Lee, T., J. Hendler, and O. Lassila. 2001. The semantic web: A new form of web content that is meaningful to computers will unleash a revolution of new possibilities. Scientific American May 17:1-18. Blackstone, E. H. 2002. Comparing apples and oranges. Journal of Thoracic and Cardiovas- cular Surgery 123(1):8-15. ———. 2005. Could it happen again? The Bjork-Shiley convexo-concave heart valve story. Circulation 111(21):2717-2719. ———. 2006. Consort and beyond. Journal of Thoracic and Cardiovascular Surgery 132(2): 229-232.

OCR for page 87
9 THE WORK REQUIRED ———. 2007. From trees to wood or wood to trees? Journal of Surgical Research 147: 59-60. Braunwald, E. 1983. Effects of coronary-artery bypass grafting on survival. Implications of the randomized coronary-artery surgery study. New England Journal of Medicine 309(19):1181-1184. Breiman, L. 1996. Bagging predictors. Machine Learning 24:123-140. ———. 2001. Statistical modeling: The two cultures. Statistical Science 16:199-231. Buchanan, B. G., and E. H. Shortliffe, eds. 1984. Rule based expert systems: The mycin experi- ments of the Stanford heuristic programming project. Reading, MA: Addison-Wesley. Buneman, P., M. Fernandez, and D. Suciu. 2000. UnQL: A query language and algebra for semistructured data based on structural recursion. Very Large Database Journal 9:76-110. CBO (Congressional Budget Office). 2007. Research on the comparative effectiveness of medical treatments. Publication No. 2975. Washington, DC: CBO. http://www.cbo. gov/ftpdocs/88xx/doc8891/12-18-ComparativeEffectiveness.pdf (accessed September 3, 2010). Cepeda, M. S., R. Boston, J. T. Farrar, and B. L. Strom. 2003. Comparison of logistic regres- sion versus propensity score when the number of events is low and there are multiple confounders. American Journal of Epidemiology 158(3):280-287. Congressional Research Service (CRS). 2007. Comparative clinical effectiveness and cost- effectiveness research: Background, history, and overview. Washington, DC: CRS. Diaconis, P., and B. Efron. 1983. Computer-intensive methods in statistics. Scientific American June 1:116-130. Donner, A., and N. Klar. 2000. Design and analysis of cluster randomization trials in health research. New York: Oxford University Press. Efron, B. 1979. Bootstrap methods: Another look at the jackknife. Annals of Statistics 7:1-26. ———. 1982. The jackknife, the bootstrap and other resampling plans. Monograph. Phila- delphia, PA: Society for Industrial and Applied Mathematics. Efron, B., and R. Tibshirani. 1986. Bootstrap methods for standard errors, confidence inter- vals, and other measures of statistical accuracy. Statistical Science 1:54-77. Erkinjuntti, T., T. Ostbye, R. Steenhuis, and V. Hachinski. 1997. The effect of different diagnostic criteria on the prevalence of dementia. New England Journal of Medicine 337(23):1667-1674. Etheredge, L. 2007. A rapid-learning health system. Health Affairs 26:w107-w118. FDA (Food and Drug Administration). 2009. Patient participation in FDA regulatory issues. http://www.fda.gov/ForConsumers/ByAudience/ForPatientAdvocates/PatientInvolvement/ default.htm (accessed August 5, 2010). Fleming, T. R., and D. L. DeMets. 1996. Surrogate end points in clinical trials: Are we being misled? Annals of Internal Medicine 125(7):605-613. Freund, Y., and R. E. Schapire. 1996. Experiments with a new boosting algorithm. Proceedings of the International Conference on Machine Learning July:148-156. Friedman, J. H. 2001. Greedy function approximation: A gradient boosting machine. Annals of Statistics 29:1189-1232. ———. 2002. Stochastic gradient boosting. Computational Statistics & Data Analysis 38: 367-378. Gliklich, R. E., and N. A. Dreyer, eds. 2007. Registries for evaluating patient outcomes: A user’s guide. Rockville, MD: Agency for Healthcare Research and Quality. Grunkemeier, G. L., R. Jin, and A. Starr. 2006. Prosthetic heart valves: Objective performance criteria versus randomized clinical trial. Annals of Thoracic Surgery 82(3):776-780.

OCR for page 87
0 LEARNING WHAT WORKS Gum, P. A., M. Thamilarasan, J. Watanabe, E. H. Blackstone, and M. S. Lauer. 2001. Aspirin use and all-cause mortality among patients being evaluated for known or suspected coro- nary artery disease: A propensity analysis. Journal of the American Medical Association 286(10):1187-1194. Heywood, J. 1546. A dialogue conteynyng the nomber in effect of all the prouerbes in the englische tongue. Hunt, S. A., W. T. Abraham, M. H. Chin, A. M. Feldman, G. S. Francis, T. G. Ganiats, M. Jessup, M. A. Konstam, D. M. Mancini, K. Michl, J. A. Oates, P. S. Rahko, M. A. Silver, L. W. Stevenson, C. W. Yancy, E. M. Antman, S. C. Smith, Jr., C. D. Adams, J. L. Anderson, D. P. Faxon, V. Fuster, J. L. Halperin, L. F. Hiratzka, A. K. Jacobs, R. Nishimura, J. P. Ornato, R. L. Page, B. Riegel, American College of Cardiology, American Heart Association Task Force on Practice Guidelines, American College of Chest Physi- cians, International Society for Heart and Lung Transplantation, and Heart Rhythm Society. 2005. ACC/AHA 2005 guideline update for the diagnosis and management of chronic heart failure in the adult. American Heart Association 112:145-235. IDSA (Infectious Diseases Society of America). 2006. Frequently asked questions about Lyme disease. http://www.idsociety.org/lymediseasefacts.htm (accessed August 5, 2010). ILADS (International Lyme and Associated Diseases Society) Working Group. 2004. Evidence- based guidelines for the management of Lyme disease. Expert Review of Anti-Infective Therapy 2(1):S1-S13. http://www.ilads.org/files/ILADS_Guidelines.pdf (accessed August 5, 2010). Ilkhanoff, L., J. D. Lewis, S. Hennessy, J. A. Berlin, and S. E. Kimmel. 2005. Potential limitations of electronic database studies of prescription non-aspirin non-steroidal anti- inflammatory drugs (NANSAIDS) and risk of myocardial infarction (MI). Pharmacoepi- demiological Drug Safety 14(8):513-522. Ioannidis, J. P. 2005. Why most published research findings are false. PLoS Medicine 2(8): e124. IOM (Institute of Medicine). 1991 The computer-based patient record: An essential technology for health care. Washington, DC: National Academy Press. ———. 2007. Learning what works best: The nations need for evidence on comparative effectiveness in health care. http://www.iom.edu/ebm-effectiveness (accessed August 5, 2010). ———. 2008. Knowing what works in health care: A roadmap for the nation. Washington, DC: The National Academies Press. Ishwaran, H. 2007. Variable importance in binary regression trees and forests. Electronic Journal of Statistics 1:519-537. Ishwaran, H., and J. S. Rao. 2003. Detecting differentially expressed genes in microar- rays using Bayesian model selection. Journal of the American Statistical Association 98:438-455. Ishwaran, H., J. S. Rao, and U. B. Kogalur. 2006. Bamarray: Java software for Bayesian analysis of variance for microarray data. BMC Bioinformatics 7(1):59. Jackson, A. 2007. Cyber-enabled discovery and innovation. Notices of the American Math- ematical Society 54:752-754. Kamerow, D. 2009. Comparative effectiveness studies inventory project. Washington, DC: A commissioned activity for the IOM Roundtable on Value & Science-Driven Health Care. Kirklin, J. W. 1979. A letter to Helen (Presidential address). Journal of Thoracic and Cardio- vascular Surgery 78(5):643-654. Kirklin, J. W., and S. S. Vicinanza. 1999. Metadata and computer-based patient records. An- nals of Thoracic Surgery 68(3 Suppl.):S23-S24. Last, J., ed. 1983. A dictionary of epidemiology. New York: Oxford University Press.

OCR for page 87
 THE WORK REQUIRED Lenat, D. B. 2008. The voice of the turtle: Whatever happened to AI? Artificial Intelligence Magazine 29(2):11-22. Lenat, D. B., and R. V. Guha. 1990. Building large knowledge-based systems: Representation and inference in the Cyc project. Reading, MA: Addison-Wesley. Levy, W. C., D. Mozaffarian, D. T. Linker, S. C. Sutradhar, S. D. Anker, A. B. Cropp, I. Anand, A. Maggioni, P. Burton, M. D. Sullivan, B. Pitt, P. A. Poole-Wilson, D. L. Mann, and M. Packer. 2006. The Seattle heart failure model: Prediction of survival in heart failure. Circulation 113(11):1424-1433. Merriam-Webster Dictionary. 2004. Springfield, MA: Merriam-Webster. Mosis, G., B. Koes, J. Dieleman, B. Stricker, J. van der Lei, and M. C. Sturkenboom. 2005a. Randomised studies in general practice: How to integrate the electronic patient record. Informatics in Primary Care 13(3):209-213. Mosis, G., A. E. Vlug, M. Mosseveld, J. P. Dieleman, B. C. Stricker, J. van der Lei, and M. C. Sturkenboom. 2005b. A technical infrastructure to conduct randomized database stud- stud- ies facilitated by a general practice research database. Journal of the American Medical Informatics Association 12(6):602-607. Mosis, G., J. P. Dieleman, B. Stricker, J. van der Lei, and M. C. Sturkenboom. 2006. A randomized database study in general practice yielded quality data but patient re- cruitment in routine consultation was not practical. Journal of Clinical Epidemiology 59(5):497-502. NCCN (National Comprehensive Cancer Networks). 2008. About the NCCN clinical practice guidelines in oncology. Available at http://www.nccn.org/professionals/physician_gls/ about.asp (accessed September 8, 2010). ———. 2009. NCCN drugs & biologics compendium. http://www.nccn.org/professionals/ drug_compendium/content/contents.asp (accessed September 8, 2010). NCI (National Cancer Institute). 2007. Cancer care outcomes research and surveillance con- sortium. http://outcomes.cancer.gov/cancors (accessed May 30, 2009). NHLBI (National Heart, Lung, and Blood Institute). 1981. National Heart, Lung, and Blood Institute coronary artery surgery study. A multicenter comparison of the effects of ran- domized medical and surgical treatment of mildly symptomatic patients with coronary artery disease, and a registry of consecutive patients undergoing coronary angiography. Circulation 63(6 Pt. 2):I1-I81. NICE (National Institute for Health and Clinical Excellence). 2004. Summary budget state- ment. www.nice.org.uk/niceMedia/pdf/boardmeeting/brdsep04item12.pdf (accessed July 31, 2004). NIH (National Institutes of Health). 2006. K30 clinical research curriculum award. http:// grants.nih.gov/training/k30.htm?sort=inst#list (accessed January 17, 2006). Nissen, S. E. 2006. Adverse cardiovascular effects of rofecoxib. New England Journal of Medicine 355(2):203-205. Pickett, J. P., ed. 2000. The American heritage dictionary of the English language, 4th ed. Boston, MA: Houghton Mifflin. Reason, J. 1999. Human error. Cambridge, MA: Cambridge University Press. Rosenbaum, P. R., and D. B. Rubin. 1983. The central role of the propensity score in obser- vational studies for causal effects. Biometrika 70:41-55. Rubin, D. B. 2007. The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine 26(1):20-36. Sabik, J. F., A. M. Gillinov, E. H. Blackstone, C. Vacha, P. L. Houghtaling, J. Navia, N. G. Smedira, P. M. McCarthy, D. M. Cosgrove, and B. W. Lytle. 2002. Does off-pump coro- nary surgery reduce morbidity and mortality? Journal of Thoracic and Cardiovascular Surgery 124(4):698-707.

OCR for page 87
2 LEARNING WHAT WORKS Sacristan, J. A., J. Soto, I. Galende, and T. R. Hylan. 1997. A review of methodologies for assessing drug effectiveness and a new proposal: Randomized database studies. Clinical Therapeutics 19(6):1424-1517. Schulz, S., B. Suntisrivaraporn, F. Baader, and M. Boeker. 2009. SNOMED reaching its adolescence: Ontologists’ and logicians’ health check. International Journal of Medical Informatics 78(Suppl. 1):S86-S94. Senge, Peter M. 1990. The fifth discipline: The art and practice of the learning organization. New York: Doubleday. Shadish, W., T. Cook, and D. Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin. Slutsky, J., and C. Clancy. 2009. AHRQ’s effective health care program: Why comparative effectiveness matters. American Journal of Medical Quality 24:67-70. Stewart, W. F., N. R. Shah, M. J. Selna, R. A. Paulus, and J. M. Walker. 2007. Bridging the inferential gap: The electronic health record and clinical evidence. Health Affairs 26(2): w181-w191. Strom, B. L. 2005. Overview of automated data systems in pharmacoepidemiology. In Pharma- coepidemiology, 4th ed., edited by B. L. Strom. Chichester, UK: John Wiley and Sons. Sturmer, T., M. Joshi, R. J. Glynn, J. Avorn, K. J. Rothman, and S. Schneeweiss. 2006. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. Journal of Clinical Epidemiology 59(5):437-447. Thorn, K. E., A. K. Bangalore, and A. C. Browne. 2007. The UMLS knowledge source server: An experience in Web 2.0 technologies. AMIA Annual Symposium Proceedings 2007:721-725. Thygesen, K., J. S. Alpert, and H. D. White. 2007. Universal definition of myocardial infarc- tion. Journal of the American College of Cardiology 50(22):2173-2195. Weeks, W., and J. Bagian. 2000. Developing a culture of safety in the Veterans Health Admin- istration. Effective Clinical Practice 3:270-276. Zarin, D. A., and T. Tse. 2008. Medicine: Moving toward transparency of clinical trials. Sci- ence 319(5868):1340-1342. Zipes, D. P., A. J. Camm, M. Borggrefe, A. E. Buxton, B. Chaitman, M. Fromer, G. Gregoratos, G. Klein, A. J. Moss, R. J. Myerburg, S. G. Priori, M. A. Quinones, D. M. Roden, M. J. Silka, C. Tracy, S. C. Smith, Jr., A. K. Jacobs, C. D. Adams, E. M. Antman, J. L. Anderson, S. A. Hunt, J. L. Halperin, R. Nishimura, J. P. Ornato, R. L. Page, B. Riegel, J. J. Blanc, A. Budaj, V. Dean, J. W. Deckers, C. Despres, K. Dickstein, J. Lekakis, K. McGregor, M. Metra, J. Morais, A. Osterspey, J. L. Tamargo, and J. L. Zamorano. 2006. ACC/AHA/ESC 2006 guidelines for management of patients with ventricular arrhythmias and the prevention of sudden cardiac death: A report of the American College of Cardiol- ogy/American Heart Association task force and the European Society of Cardiology com- mittee for practice guidelines (Writing committee to develop guidelines for management of patients with ventricular arrhythmias and the prevention of sudden cardiac death). Journal of the American College of Cardiology 48(5):e247-e346.