4
Systematic Reviews: The Central Link Between Evidence and Clinical Decision Making

If, as is sometimes supposed, science consisted in nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight. Two processes are thus at work side by side, the reception of new material and the digestion and assimilation of the old…. The work which deserves, but I am afraid does not always receive, the most credit is that in which discovery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out.

J. W. Strutt Lord Rayleigh

Address to the British Association for the Advancement of Science

(Rayleigh, 1884, p. 1)

More than a decade has passed since it was first shown that patients have been harmed by failure to prepare scientifically defensible reviews of existing research evidence. There are now many examples of the dangers of this continuing scientific sloppiness. Organizations and individuals concerned about improving the effectiveness and safety of health care now look to systematic reviews of research—not individual studies—to inform their judgments.

Iain Chalmers

Academia’s Failure to Support Systematic Reviews

(Chalmers, 2005)

Abstract: This chapter provides the committee’s findings and recommendations for conducting systematic evidence reviews under the aegis of a proposed national clinical effectiveness assessment program (“the Program”). The chapter reviews the origins of systematic review methods and describes the fundamental components of systematic reviews and the shortcomings of current efforts. Under the status quo, the quality of the reviews is variable, methods are poorly documented, and findings are often unreliable. The committee recommends that the Program establish evidence-based, methodological standards for systematic reviews, including standard terminology for characterizing the strength of evidence and a standard reporting format for systematic reviews. Once Program stan-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 81
4 Systematic Reviews: The Central Link Between Evidence and Clinical Decision Making If, as is sometimes supposed, science consisted in nothing but the labori- ous accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight. Two processes are thus at work side by side, the reception of new material and the digestion and assimilation of the old. . . . The work which deseres, but I am afraid does not always receie, the most credit is that in which discoery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out. J. W. Strutt Lord Rayleigh Address to the British Association for the Advancement of Science (Rayleigh, 1884, p. 1) More than a decade has passed since it was first shown that patients hae been harmed by failure to prepare scientifically defensible reiews of exist- ing research eidence. There are now many examples of the dangers of this continuing scientific sloppiness. Organizations and indiiduals concerned about improing the effectieness and safety of health care now look to systematic reiews of research—not indiidual studies—to inform their judgments. Iain Chalmers Academia’s Failure to Support Systematic Reviews (Chalmers, 2005) Abstract: This chapter proides the committee’s findings and recommen- dations for conducting systematic eidence reiews under the aegis of a proposed national clinical effectieness assessment program (“the Pro- gram”). The chapter reiews the origins of systematic reiew methods and describes the fundamental components of systematic reiews and the shortcomings of current efforts. Under the status quo, the quality of the reiews is ariable, methods are poorly documented, and findings are often unreliable. The committee recommends that the Program establish eidence-based, methodological standards for systematic reiews, includ- ing standard terminology for characterizing the strength of eidence and a standard reporting format for systematic reiews. Once Program stan- 

OCR for page 81
 KNOWING WHAT WORKS IN HEALTH CARE dards are established, the Program should fund only those reiewers who commit to and consistently meet the standards. The committee found that the new science of systematic reiews has made great strides, but more methodological research is needed. Inesting in the science of research synthesis will increase the quality and the alue of the eidence proided in systematic reiews. It is not clear whether there are sufficient numbers of qualified researchers to conduct high-quality reiews. The capacity of the workforce should be assessed and expanded, if needed. Systematic reviews are central to scientific inquiry into what is known and not known about what works in health care (Glasziou and Haynes, 2005; Helfand, 2005; Mulrow and Lohr, 2001; Steinberg and Luce, 2005). In 1884, J. W. Strutt Lord Rayleigh, who later won a Nobel prize in physics, observed that the synthesis and explanation of past discoveries are integral to future progress (Rayleigh, 1884). Yet, more than a century later, Antman and colleagues (1992) and Lau and colleagues (1992) clearly demonstrated that this message was still largely ignored, with the potential for great harm to patients. In a series of meta-analyses examining the treatment of myocar- dial infarction, the researchers concluded that clinicians need better access to syntheses of the results of existing studies to formulate clinical recom- mendations. Today, systematic reviews of the available evidence remain an often undervalued scientific discipline. This chapter has three principal objectives: (1) to describe the funda- mental components of a systematic review, (2) to present the committee’s recommendations for conducting systematic evidence reviews under the aegis of a proposed national clinical effectiveness assessment program (“the Program”), and (3) to highlight the key challenges in producing high- quality systematic reviews. BACKGROUND What Is a Systematic Review? A systematic review is a scientific investigation that focuses on a specific question and uses explicit, preplanned scientific methods to identify, select, assess, and summarize similar but separate studies (Haynes et al., 2006; West et al., 2002). It may or may not include a quantitative synthesis of the results from separate studies (meta-analysis). A meta-analysis quantitatively combines the results of similar studies in an attempt to allow inference from the sample of studies included to the population of interest. This report uses the term “systematic review” to describe reviews that incorporate meta- analyses as well as reviews that present the study data descriptively rather than inferentially.

OCR for page 81
 SYSTEMATIC REVIEWS Individual studies rarely provide definitive answers to clinical effective- ness questions (Cook et al., 1997). If it is conducted properly, a systematic review should make obvious the gap between what is known about the ef- fectiveness of a particular service and what clinicians and patients want to know (Helfand, 2005). As such, systematic reviews are also critical to the development of an agenda for further primary research because they reveal where the evidence is insufficient and new information is needed (Neumann, 2006). Without systematic reviews, researchers may miss promising leads or pursue questions that have already been answered (Mulrow et al., 1997). In addition, systematic reviews provide an essential bridge between the body of research evidence and the development of clinical guidance. Key U.S. Producers and Users of Systematic Reviews This section briefly describes the variety of contexts in which key U.S. organizations produce or use systematic reviews (Table 4-1). The ultimate purposes of systematic reviews vary and include health coverage decisions, practice guidelines, regulatory approval of new pharmaceuticals or medical devices, clinical research or program planning. Within the federal govern- ment, the users include the Agency for Healthcare Research and Quality (AHRQ), the Centers for Medicare & Medicaid Services (CMS), the Medi- care Evidence Development and Coverage Advisory Committee (MedCAC), the Centers for Disease Control and Prevention (CDC), the U.S. Food and Drug Administration (FDA), the Substance Abuse and Mental Health Ad- ministration (SAMHSA), the U.S. Preventive Services Task Force (USPSTF), and the Veterans Health Administration (VHA). AHRQ plays a lead role in producing systematic reviews through its program of Evidence-based Practice Centers (EPCs) as a part of its Effective Health Care Program. EPCs produce systematic reviews for professional medical societies and several federal agencies, including CMS and the Na- tional Institutes of Health (NIH) Consensus Development Conferences, as well as a variety of other public and private requestors, such as the USPSTF and the American Heart Association. The reviews cover a broad range of topics, including the effectiveness and safety of health care interventions, emergency preparedness, research methods, and approaches to improving the quality and delivery of health care.1 The AHRQ Effective Health Care Program produces comparative effectiveness studies on surgical proce- dures, medical devices, and medical therapies in 10 priority areas (Slutsky, 2007). The CDC conducts or sponsors systematic effectiveness reviews to evaluate and make recommendations on population-based and public 1 See Table 3-3 in Chapter 3 for a list of recent EPC studies.

OCR for page 81
 KNOWING WHAT WORKS IN HEALTH CARE TABLE 4-1 Key U.S. Producers and Users of Systematic Reviews Government Agencies CMS Component AHRQ USPSTF SAMHSA FDA VHA MedCAC CDC Activity • Produces reviews ✓ ✓ ✓ • Sponsors or ✓ ✓ ✓ ✓ ✓ purchases reviews Principal use • Development ✓ ✓ ✓ ✓ ✓ of practice guidelines and recommendations • Decisions ✓ ✓ ✓ ✓ regarding health coverage • Regulatory ✓ approval NOTE: BCBSA TEC = Blue Cross and Blue Shield Association Technology Evaluation Center. health interventions and to improve the underlying research methods (CDC, 2007). The Blue Cross and Blue Shield Association (BCBSA) Technology Evaluation Center (TEC) produces systematic reviews that assess medical technologies for decision makers in its member plans but also provides the results of these reviews to the public for free.2 Many other health plans look to private research organizations, such as the ECRI Institute and Hayes, Inc., that produce systematic evidence assessments available by subscription or for purchase (ECRI, 2006a,b; Hayes, Inc., 2007). Because the reviews are proprietary, they are not free to the public and the subscription fees are considerable. At Hayes, Inc., for example, subscriptions range from $10,000 to $300,000, depending on the size of the subscribing organiza- tions and the types of products licensed.3 The Cochrane Collaboration is an international effort that produces systematic reviews of health interventions; 11 percent (nearly 1,700 indi- viduals) of its active contributors are in the United States (Allen and Clarke, 2007). Cochrane reviews are available by subscription to The Cochrane Library, and abstracts are available for free through PubMed or www. cochrane.org. 2 See http://www.bcbs.com/betterknowledge/tec/. 3 Personal communication, W. S. Hayes, Hayes, Inc., August 29, 2007.

OCR for page 81
 SYSTEMATIC REVIEWS Private Research Firms Other Entities ECRI BCBSA Hayes, Cochrane Health Specialty Institute TEC Inc. Collaboration Plans Societies Manufacturers ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Professional medical societies often sponsor or conduct evidence re- views as the first step in developing a practice guideline. These include, for example, the American College of Physicians, several cardiology groups (the American College of Cardiology, the American College of Chest Physicians, and the American Heart Association), the American Academy of Neurol- ogy, and the American Society of Clinical Oncology. Origins of Systematic Review Methods The term “meta-analysis” was first used by social scientists in the 1970s to describe the process of identifying a representative set of studies of a given topic and summarizing their results quantitatively. In a groundbreak- ing 1976 assessment of treatment for depression, Glass (1976) first used the term “meta-analysis” to describe what is now referred to as systematic review. Textbooks describing the concept and methods of systematic re- views (Cooper and Rosenthal, 1980; Glass et al., 1981; Hedges and Olkin, 1985; Light and Pillemer, 1984; Rosenthal, 1978; Sutton et al., 2000), and research articles exploring issues such as publication bias followed during that and the subsequent decade. Subsequently, as quantitative syntheses started to include qualitative summaries and medical scientists adopted the methods, a new terminol- ogy emerged. Richard Peto and colleagues used the term “overview” for

OCR for page 81
6 KNOWING WHAT WORKS IN HEALTH CARE the new combined approach (Early Breast Cancer Trialists’ Collaborative Group, 1988). Chalmers and Altman (1995) appear to have introduced the term “systematic review” in their book Systematic Reiews. They also suggested that the term “meta-analysis” be restricted to the statistical sum- mary of the results of studies identified as a product of the review process (Chalmers and Altman, 1995). Confusion over terminology persists today, perhaps because the methods grew up in the social sciences and only later were embraced by the medical sciences. The statistical methods underlying the quantitative aspects of system- atic review—i.e., meta-analysis—date to the early 20th century, when statis- ticians started developing methods for combining the findings from separate but similar studies. In 1904, using new statistical methods, Karl Pearson (1904) combined research on the impact of inoculation against enteric fe- ver on mortality in five communities. In a 1907 study on the prevalence of typhoid, Goldberger (1907) again used quantitative synthesis. Social scientists were the first to use methods to critically synthesize results to allow statistical inference from a sample a population. As early as 1940, Pratt and colleagues (1940) at Duke University published a critical synthesis of more than 60 years of research on extrasensory perception. Systematic reviews in the health care arena were comparatively slow to catch on, and the growth in their development and use coincided with the general rise of evidence-based medicine (Guyatt, 1991). The early imple- menters of systematic reviews were those who conducted clinical trials and who saw the need to summarize data from multiple effectiveness trials, many of them with very small sample sizes (Yusuf et al., 1985). In the 1970s, Iain Chalmers organized the first major collaborative effort to de- velop a clinical trials evidence base, beginning with the Oxford Database of Perinatal Trials (Chalmers et al., 1986). This subsequently led to two major compilations of systematic reviews of clinical trials, one of pregnancy and childbirth (Chalmers et al., 1989) and one of the newborn period (Sinclair and Bracken, 1992). The growth of bioinformatics, specifically, electronic communication, data storage, and improved indexing and retrieval of pub- lications, allowed this collaborative effort in the perinatal field to expand further. In 1993, the Cochrane Collaboration was formed (Dickersin and Manheimer, 1998) with the aim of synthesizing information from studies of interventions on all health topics. Up to this time, literature reviews were often used to assess the effec- tiveness of health care interventions, but empiric research also began to re- veal problems in their execution. The methods underlying the reviews were often neither objective nor transparent (Mulrow, 1987; Oxman and Guyatt, 1988); and they did not routinely use scientific methods to identify, assess, and synthesize information. The approach to deciding which literature should be included and which findings should be presented was subjective

OCR for page 81
7 SYSTEMATIC REVIEWS and nonsystematic. The reviews may have provided thoughtful, readable discussions of a topic, but the conclusions were generally not credible. The following sections of the chapter describe the fundamentals of conducting a scientifically rigorous systematic review and then provide the committee’s findings on current efforts. FUNDAMENTALS OF A SYSTEMATIC REVIEW Although researchers use a variety of terms to describe the building blocks of a systematic review, the fundamentals are well established (AHRQ EPC Program, 2007; Counsell, 1997; EPC Coordinating Center, 2005; Haynes et al., 2006; Higgins and Green, 2006; Khan and Kleijnen, 2001; Khan et al., 2001a,b; West et al., 2002).4 Five basic steps (listed below) should be followed, and the key decisions that comprise each step of the review should be clearly documented. Step 1: Formulate the research question. Step 2: Construct an analytic (or logic) framework. Step 3: Conduct a comprehensive search for evidence. Step 4: Critically appraise the evidence. Step 5: Synthesize the body of evidence. The following sections briefly describe each of these steps in the process. Step 1: Formulate the Research Question The foundation of a good systematic review is a well-formulated, clearly defined, answerable question. As such, it guides the analytic (or logic) framework for the review, the overall research protocol (i.e., the search for relevant evidence, decisions about which types of evidence should be used, and how best to identify the evidence), and the critical appraisal of the relevant evidence. The objective, in this first step, is to define a precise, unambiguous answerable research question. Richardson and colleagues (1995) coined the mnemonic PICO (popula- tion, intervention, comparison, and outcome of interest) to help ensure that explicit attention is paid to the four key elements of an evidence question.5,6 4 Unless otherwise noted, this section draws from these references. 5 Personal communication, W. S. Richardson, Boonshoft School of Medicine, Wright State University, October 3, 2007. 6 A recent draft version of an AHRQ comparative effectiveness methods manual proposes expanding the PICO format to PICOTS, adding “t” for timing and “s” for settings (AHRQ, 2007a).

OCR for page 81
 KNOWING WHAT WORKS IN HEALTH CARE Table 4-2 shows examples of how the PICO format can guide the building of a research question. The characteristics of the study population, such as age, sex, severity of illness, and presence of comorbidities, usually vary among studies and can be important factors in the effect of an intervention. Health care inter- ventions may have numerous outcomes of interest. The research question should be formulated so that it addresses all outcomes—beneficial and adverse—that matter to patients, clinicians, payers, developers of practice guidelines, and others who may be affected (Schünemann et al., 2006). For example, treatments for prostate cancer may affect mortality; but patients are also interested in learning about potential harmful treatment effects, such as urinary incontinence and impotence. Imaging tests for Alzheimer’s disease may lead to the early diagnosis of the condition, but patients and the patients’ caregivers may be particularly interested in whether an early diagnosis improves cognitive outcomes or quality of life. Many researchers suggest that decision makers be directly involved in formulating the question to ensure that the systematic review is relevant and can inform decision making (Lavis et al., 2005; Schünemann et al., 2006). The questions posed by end users must sometimes be reframed to be answerable by clinical research studies. TABLE 4-2 PICO Format for Formulating an Evidence Question PICO Component Tips for Building Question Example Patient population or “How would I describe this “In patients with heart failure from problem group of patients?” dilated cardiomyopathy who are in • Balance precision with sinus rhythm . . .” breity Intervention (a cause, “Which main intervention “. . . would adding anticoagulation prognostic factor, is of interest?” with warfarin to standard heart treatment, etc.) • Be specific failure therapy . . .” Comparison “What is the main “. . . when compared with intervention alternative to be compared standard therapy alone . . .” (if necessary) with the intervention?” • Be specific Outcomes “What do I hope “. . . lead to lower mortality or the intervention will morbidity from thromboembolism? accomplish?” “What could Is this enough to be worth the this exposure really affect?” increased risk of bleeding?” • Be specific SOURCE: Adapted from the Eidence-based Practice Center Partner’s Guide (EPC Coordinat- ing Center, 2005).

OCR for page 81
9 SYSTEMATIC REVIEWS Step 2: Construct an Analytic Framework Once the research question is established, it should be articulated in an analytic framework that clearly lays out the chain of logic underlying the case for the health intervention of interest. The complexity of the analysis will vary depending on the number of linkages between the intervention and the outcomes of interest. For preventive services, there may be mul- tiple steps between, for example, screening for a disease and reductions in morbidity and mortality. Figure 4-1 shows the generic analytic framework 1 5 Reduced Treatment Screening Association Persons Early Detection of Morbidity Intermediate 3 4 6 at Risk and/or Target Condition Outcome 2 Mortality 7 8 Adverse Effects Adverse Effects of Treatment of Screening FIGURE 4-1 Analytic framework used by the U.S. Preventive Services Task Force. NOTE: Generic analytic framework for screening topics. Numbers refer to key questions as follow: (1) Is there direct evidence that screening reduces morbidity and/or mortality? (2) What is the prevalence of disease in the target groups? Can a high-risk group be reliably identified? (3) Can the screening test accurately detect the target condition? (a) What are the sensitivity and specificity of the test? (b) Is there 4-1 significant variation between examiners in how the test is performed? (c) In actual screening programs, how much earlier are patientstype new identified and treated? (4) Does treatment reduce the incidence of the intermediate outcome? (a) Does treatment work under ideal, clinical trial conditions? (b) How do the efficacy and effectiveness of treatments compare in community settings? (5) Does treatment improve health outcomes for people diagnosed clinically? (a) How similar are people diagnosed clinically to those diagnosed by screening? (b) Are there reasons to expect people diagnosed by screening to have even better health outcomes than those diagnosed clinically? (6) Is there intermediate outcome reliability associated with reduced morbidity and/or mortality? (7) Does screening result in adverse effects? (a) Is the test acceptable to patients? (b) What are the potential harms, and how often do they occur? (8) Does treatment result in adverse effects? SOURCE: Reprinted from the American Journal of Preentie Medicine, 20(3) Harris, R. P., M. Helfand, S. H. Woolf, K. N. Lohr, C. D. Mulrow, S. M. Teutsch, and D. Atkins, Current methods of the US Preventive Services Task Force: A review of the process, 21-35, Copyright 2007, with permission from Elsevier.

OCR for page 81
90 KNOWING WHAT WORKS IN HEALTH CARE that the USPSTF uses to assess screening interventions. It makes explicit the population at risk (left side of the figure), preventive services, diagnostic or therapeutic interventions, and intermediate and health outcomes to be considered (Harris et al., 2001). It also illustrates the chain of logic that the evidence must support to link the service to potential health outcomes: the arrows (linkages), labeled with a service or treatment, represent the ques- tions that the evidence must answer; dotted lines represent associations; and rectangles represent the intermediate outcomes (rounded corners) or the health states (square corners) by which those linkages are measured. The overarching linkage (Arrow 1) above the primary framework rep- resents evidence that directly links screening to changes in health out- comes. For example, a randomized controlled trial (RCT) of screening for Chlamydia established a direct, causal connection between screening and reductions in the incidence of pelvic inflammatory disease (Meyers et al., 2007; Scholes et al., 1996). That is, a single body of evidence established the connection between the preventive service (screening) and the health outcome (reduced morbidity). When direct evidence is lacking or is of insufficient quality to be con- vincing, the USPSTF relies on a chain of linkages to assess the likely effec- tiveness of a service. These linkages correspond to key questions about the screening test accuracy (Arrow 3), the efficacy of treatment (Arrows 4 and 5 for intermediate and health outcomes, respectively), and the association between intermediate measures and health outcomes (Dotted Line 6). A similar analytic framework can be constructed for questions of drug treat- ment, devices, behavior change, procedures, health care delivery, or any type of health intervention used in a population or in individuals. Deciding Which Eidence to Use: Study Selection Criteria What constitutes evidence that a health care service is highly effective? As noted in Chapter 1, scientists view evidence as knowledge that is ex- plicit, systematic, and replicable. However, patients, clinicians, payers, and other decision makers have different perspectives on what constitutes evi- dence of effectiveness. For example, some may view the scientific evidence as demonstrating what works under ideal circumstances but not necessarily under a particular set of real world circumstances. A variety of factors can affect the applicability of a particular RCT to individual clinical decisions or circumstances, including patient factors, such as comorbidities, underly- ing risk, adherence to therapies, disease stage and severity, health insurance coverage, and demographics; intervention factors, such as care setting, level of training, timing and quality of the intervention, and an array of other factors (Atkins, 2007).

OCR for page 81
9 SYSTEMATIC REVIEWS The choice of study designs to be included in a systematic review should be based on the type of research question being asked and should have the goal of minimizing bias (Glasziou et al., 2004; Oxman et al., 2006). Table 4-3 provides examples of research questions and the types of evidence that are the most appropriate for addressing them. RCTs can answer ques- tions about the efficacy of screening, preventive, and therapeutic interven- tions. Although RCTs can best answer questions about the potential harms from interventions, observational study designs, such as cohort studies, case series, or case control studies, may be all that are available or possible for the evaluation of rare or long-term outcomes.7 In fact, because harms from interventions are often rare or occur far in the future, a systematic review of observational research may be the best approach to identifying reliable evidence on potential rare harms (or benefits). Observational studies are generally the most appropriate for answering questions related to prognosis, diagnostic accuracy, incidence, prevalence, and etiology (Chou and Helfand, 2005; Tatsioni et al., 2005). Cohort stud- ies and case series are useful for examining long-term outcomes because RCTs may not monitor patients beyond the primary outcome of interest or for rare outcomes because they generally have small numbers of par- ticipants. Case series are often used, for example, to identify the potential long-term harms of new types of radiotherapy. Similarly, the best evidence on potential harms related to oral contraceptive use (e.g., an increased risk of thromboembolism) may be from nonrandomized cohort studies or case- control studies (Glasziou et al., 2004). Many systematic reviews use a best evidence approach that allows the use of broader inclusion criteria when higher-quality evidence is lacking (Atkins et al., 2005). In these cases, the systematic reviews consider obser- vational studies because, at a minimum, noting the available evidence helps to delineate what is known and what is not known about the effectiveness of the intervention in question. By highlighting the gaps in knowledge, the review establishes the need for better quality evidence and helps to priori- tize research topics. For intervention effectiveness questions for which RCTs form the high- est level of evidence, it is essential to fully document the rationale for including nonrandomized evidence in a review. Current practice does not meet this standard, however. Researchers have found, for example, that 30 of 49 EPC reports that included observational studies did not disclose the rationale for doing so (Norris and Atkins, 2005). 7 See Chapter 1 for the definitions of the types of experimental and observational studies.

OCR for page 81
0 KNOWING WHAT WORKS IN HEALTH CARE BOX 4-5 Unresolved Methodological Issues in Conducting Systematic Reviews Locating and Selecting Studies • How best to identify all relevant published studies • Whether to include and how best to identify non-English-language studies • Whether to include and how best to identify unpublished studies and studies in the gray literature (e.g., abstracts) • Search strategies for identifying observational studies in MEDLINE, EMBASE, and other databases • Search strategies for identifying studies of diagnostic accuracy in MEDLINE, EMBASE, and other databases Assessing Study Quality • Understanding the sources of reporting deficiencies in studies being synthesized • Understanding and identifying potential biases and conflicts of interest • Quality thresholds for study inclusion and the management of individual study quality in the context of a review Collecting Data • Identifying and selecting information to assess treatment harms • Obtaining important unpublished data from relevant studies • Methods used for data abstraction Analyzing and Presenting Results • Use of qualitative data in systematic reviews • Use of economic data in systematic reviews • Methods for combining results of diagnostic test accuracy Statistical Methods (e.g., statistical heterogeneity, fixed versus random effects, and meta-regression) • Inclusion of interstudy variability into displays of results • How best to display findings and their reliability for users • Methods and validity of indirect comparisons Interpreting Results • Understanding why reviews on similar topics may yield different results • Updating systematic reviews • Frequency of updates SOURCE: Cochrane Collaboration (2007); Higgins and Green (2006). and provide excitement about the potential to contribute to health research and to health care practice overall. Moreover, the academic community must recognize the scientific scholarship that is required to conduct high- quality systematic reviews.

OCR for page 81
 SYSTEMATIC REVIEWS OTHER PROGRAM CHALLENGES Keeping Reviews Up-to-Date Systematic reviews are not only difficult and time consuming, they also must be kept up-to-date to ensure patient safety. Having an organiza- tion that exercises oversight on the production of systematic reviews, for example, the Cochrane Collaboration or professional societies that pro- duce clinical practice guidelines, provides an infrastructure and chain of responsibility for the updating of reviews. There has been little research on updating, and the research that does exist indicates that not all organiza- tions have mechanisms for systematically updating their reviews. In 2001, Shekelle and colleagues (2001) examined how quickly the AHRQ guidelines went out of date. At the time of that study, they classified only 3 of the 17 guidelines in circulation at that time as still valid. About half of the guidelines were out of date in 5.8 years from the time of their release, and at 3.6 years, at least 10 percent were out of date. A more recent report examining a sample of 100 high-quality systematic reviews of inter- ventions found that within 5.5 years, half of the reviews had new evidence that would substantively change the conclusions about the effectiveness of interventions, and within 2 years almost 25 percent had such evidence (Shojania et al., 2007). The frequency of updating was associated with the clinical topic area and the initial heterogeneity of the results. Thus, it appears that the failure to update systematic reviews and guidelines within a few years could easily result in patient care that is not evidence based and, worse, care that is not as effective as possible or po- tentially dangerous. New and Emerging Technologies Although this chapter has focused on comprehensive, systematic re- views, the committee recognizes that some decision makers have a legiti- mate need for objective advisories on new and emerging technologies in order to respond to coverage requests when few, if any, high-quality studies or systematic reviews exist. In addition, patients and providers want infor- mation on new health care services as soon as the services become known, often because manufacturers are pressing them to adopt a product or be- cause patients have read direct-to-consumer advertising and want answers from their physicians and other health care providers. Private technology assessment organizations, such as the ECRI Institute and Hayes, Inc., have responded to the market demand for early reviews of new technologies (ECRI, 2006b; Hayes, Inc., 2007). These firms and other private, proprietary organizations offer clients brief reviews based on

OCR for page 81
 KNOWING WHAT WORKS IN HEALTH CARE readily available sources of information. Two examples are provided in Ap- pendix E (as proprietary products, they are not in the public domain). The reviews aggregate what little is known from searches of electronic databases (e.g., MEDLINE, EMBASE, or the Cochrane Central Register of Controlled Trials) and published conference abstracts. Other easily obtained informa- tion, such as reports from FDA advisory committee meetings, may also be included. Typically, the reviews include a brief description of an interven- tion; its relevance to clinical care; a short, preliminary list of the relevant research citations that have been identified; two- to three-paragraph sum- maries of selected research abstracts; and details on the methods used to search the literature. The Program should consider producing brief advisories on new and emerging technologies in addition to full systematic reviews. If so, like the ECRI Institute and Hayes, Inc., products, the advisories produced under the aegis of the Program should clearly emphasize and highlight the limitations of the information. The advisories clearly state their limitations, so that no one will misinterpret them as an adequate substitute for substantive assess- ments of evidence on effectiveness. REFERENCES AHRQ (Agency for Healthcare Research and Quality). 2006. The guide to clinical preentie serices 006: Recommendations of the U.S. Preentie Serices Task Force. AHRQ. Pub. No. 06-0. Rockville, MD: AHRQ. ———. 2007a. Guide for conducting comparatie effectieness reiews (Draft for public comment) http://effectivehealthcare.ahrq.gov/getInvolved/commentFormMethodsGuide. cfm?DocID=1 (accessed October 10, 2007). ———. 2007b. User’s guide to registries ealuating patient outcomes: Summary. AHRQ Pub. No. 07-EHC00-. Rockville, MD: AHRQ. AHRQ EPC Program (Evidence-based Practice Center Program). 2007. Template for submis- sions of topics for AHRQ eidence reports or technology assessments http://www.ahrq. gov/clinic/epcpartner/epcesubtempl.doc (accessed January 17, 2007). Allen, C., and M. Clarke. 2007 (unpublished). International actiity in Cochrane Reiew Groups with a focus on the USA. Cochrane Collaboration. Als-Nielsen, B., W. Chen, C. Gluud, and L. L. Kjaergard. 2003. Association of funding and conclusions in randomized drug trials: A reflection of treatment effect or adverse events? JAMA 290(7):921-928. Annals of Internal Medicine. 2007. Information for authors http://www.annals.org/shared/ author_info.html (accessed July 11, 2007). Antman, E. M., J. Lau, B. Kupelnick, F. Mosteller, and T. C. Chalmers. 1992. A comparison of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 268(2):240-248. Archives of General Psychiatry. 2007. Instructions for authors http://archpsyc.ama-assn.org/ misc/ifora.dtl (accessed July 12, 2007). Atkins, D. 2007. Creating and synthesizing evidence with decision makers in mind: Integrat- ing evidence from clinical trials and other study designs. Medical Care 45(10 Suppl 2): S16-S22.

OCR for page 81
 SYSTEMATIC REVIEWS Atkins, D., K. Fink, and J. Slutsky. 2005. Better information for better health care: The Evidence-based Practice Center program and the Agency for Healthcare Research and Quality. Annals of Internal Medicine 142(12 Part 2):1035-1041. Barton, M. 2007. Using systematic reiews to deelop clinical recommendations (Submitted responses to the IOM HECS committee meeting, January , 007). Washington, DC. BCBSA (Blue Cross and Blue Shield Association). 2007. Blue Cross and Blue Shield Asso- ciation proposes payer-funded institute to ealuate what medical treatments work best http://www.bcbs.com/news/bcbsa/blue-cross-and-blue-shield-association-proposes-payer- funded-institute.html (accessed May 2007). Bekelman, J. E., Y. Li, and C. P. Gross. 2003. Scope and impact of financial conflicts of interest in biomedical research: A systematic review. JAMA 289(4):454-465. Bhandari, M., F. Morrow, A. V. Kulkarni, and P. Tornetta. 2001. Meta-analyses in orthopaedic surgery: A systematic review of their methodologies. Journal of Bone and Joint Surgery 83A:15-24. Bossuyt, P. M., J. B. Reitsma, D. E. Bruns, C. A. Gatsonis, P. P. Glasziou, L. M. Irwig, D. Moher, D. Rennie, H. C. de Vet, and J. G. Lijmer. 2003. The STARD statement for reporting studies of diagnostic accuracy: Explanation and elaboration. Clinical Chemistry 49:7-18. British Medical Journal. 2007. Resources for authors: Article requirements http://resources. bmj.com/bmj/authors/article-submission/article-requirements (accessed July 11, 2007). Buscemi, N., L. Hartling, B. Vandermeer, L. Tjosvold, and T. P. Klassen. 2006. Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology 59(7):697-703. CANCER. 2007. CANCER instructions for authors http://www.interscience.wiley.com/cancer/ (accessed July 11, 2007). CDC (Centers for Disease Control and Prevention). 2007. Community preentie serices: Methods: Effectieness ealuation http://www.thecommunityguide.org/methods/ (ac- cessed October 3, 2007). Chalmers, I. 2003. Trying to do more good than harm in policy and practice: The role of rigor- ous, transparent, up-to-date evaluations. Annals of the American Academy of Political and Social Science 589(1):22-40. ———. 2005. Academia’s failure to support systematic reviews. Lancet 365(9458):469. Chalmers, I., and D. G. Altman. 1995. Systematic reiews. London: BMJ Publications. Chalmers, I., J. Hetherington, and M. Newdick. 1986. The Oxford database of perinatal tri- als: Developing a register of published reports of controlled trials. Controlled Clinical Trials 7(4):306-324. Chalmers, I., M. Enkin, and M. Keirse. 1989. Effectie care in pregnancy and childbirth. Oxford, UK: Oxford University Press. Chan, A. W., A. Hrobjartsson, M. T. Haahr, P. C. Gøtzsche, and D. G. Altman. 2004. Em- perical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291:2457-2465. CHEST. 2006. CHEST instructions to authors and statement of CHEST policies http://www. chestjournal.org/misc/PolicyInstruxA.pdf (accessed July 11, 2007). Chou, R., and M. Helfand. 2005. Challenges in systematic reviews that assess treatment harms. Annals of Internal Medicine 142(12 Part 2):1090-1099. Circulation. 2007. Instructions for authors http://circ.ahajournals.org/misc/ifora.shtml (ac- cessed July 12, 2007). Claxton, K., J. T. Cohen, and P. J. Neumann. 2005. When is evidence sufficient? Health Af- fairs 24(1):93-101. Cochrane Collaboration. 2007. Methods groups newsletter Vol. 11. Oxford, UK.

OCR for page 81
 KNOWING WHAT WORKS IN HEALTH CARE Congressional Budget Office. 2007. Research on the comparative effectiveness of medical treatments: Options for an expanded federal role. Testimony by Director Peter R. Orszag before House Ways and Means Subcommittee on Health http://www.cbo.gov/ftpdocs/ 82xx/doc8209/Comparative_Testimony.pdf (accessed June 12, 2007). Cook, D. J., C. D. Mulrow, and R. B. Haynes. 1997. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine 126(5):376-380. Cooper, H. M., and R. Rosenthal. 1980. A comparison of statistical and traditional procedures for summarizing research. Psychological Bulletin 87:442-449. Counsell, C. 1997. Formulating questions and locating primary studies for inclusion in sys- tematic reviews. Annals of Internal Medicine 127(5):380-387. Delaney, A., S. M. Bagshaw, A. Ferland, B. Manns, and K. B. Laupland. 2005. A systematic evaluation of the quality of meta-analyses in the critical care literature. Critical Care 9: R575-R582. Diabetes. 2007. Diabetes instructions for authors http://care.diabetesjournals.org/misc/ifora. shtml (accessed July 30, 2007). Dickersin, K. 2002. Systematic reviews in epidemiology: Why are we so far behind? Interna- tional Journal of Epidemiology 31(1):6-12. ———. 2005. Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In Publication bias in meta-analysis: Preention, assessment, and adjustments. Edited by Rothstein, H., A. Sutton, and M. Borenstein. London, UK: John Wiley and Sons, Ltd. ———. 2007 (unpublished). Steps in eidence-based healthcare. PowerPoint Presentation. Baltimore, MD. Dickersin, K., and Y.-I. Min. 1993. Publication bias: The problem that won’t go away. In Do- ing more good than harm: The ealuation of health care interentions. Edited by Warren, K. S., and F. Mosteller. New York: New York Academy of Sciences. Pp. 135-148. Dickersin, K., and E. Manheimer. 1998. The Cochrane Collaboration: Evaluation of health care and services using systematic reviews of the results of randomized controlled trials. Clinical Obstetrics and Gynecology 41(2):315-331. Dickersin, K., P. Hewitt, and L. Mutch. 1985. Perusing the literature: Comparison of MEDLINE searching with a perinatal trials database. Controlled Clinical Trials 6(4):306-317. Dickersin, K., R. Scherer, and C. Lefebvre. 1994. Identifying relevant studies for systematic reviews. BMJ 309(6964):1286-1291. Dickersin, K., E. Manheimer, S. Wieland, K. A. Robinson, C. Lefebvre, S. McDonald, and the CENTRAL Development Group. 2002. Development of the Cochrane Collaboration’s CENTRAL register of controlled clinical trials. Ealuation and the Health Professions 25:38-64. Early Breast Cancer Trialists’ Collaborative Group. 1988. Effects of adjuvant tamoxifen and of cytotoxic therapy on mortality in early breast cancer: An overview of 61 randomised trials among 28 896 women. New England Journal of Medicine 319:1681-1692. Easterbrook, P. J., J. A. Berlin, R. Gopalan, and D. R. Matthews. 1991. Publication bias in clinical research. Lancet 337:867-872. Ebrahim, S., and M. Clarke. 2007. STROBE: New standards for reporting observational epi- demiology, a chance to improve. International Journal of Epidemiology 36(5):945-948. ECRI. 2006a (unpublished). 006 ECRI price list. ECRI Health Technology Assessment Information Service. ———. 2006b. About ECRI http://www.ecri.org/About_ECRI/About_ECRI.aspx (accessed January 31, 2007). Eddy, D. M. 2007. Linking electronic medical records to large-scale simulation models: Can we put rapid learning on turbo? Health Affairs 26(2):w125-w136.

OCR for page 81
 SYSTEMATIC REVIEWS Editors. 2005. Reviews: Making sense of an often tangled skein of evidence. Annals of Internal Medicine 142(12 Part 1):1019-1020. Egger, M., T. Zellweger-Zähner, M. Schneider, C. Junker, C. Lengeler, and G. Antes. 1997. Language bias in randomised controlled trials published in English and German. Lancet 350(9074):326-329. Egger, M., G. Davie Smith, and K. O’Rourke. 2001. Rationale, potentials, and promise of systematic reviews. In Systematic Reiews in Health Care: Meta-Analysis in Context. Edited by Egger, M. London, UK: BMJ Publishing Group. Pp. 3-19. Egger, M., P. Juni, C. Bartlett, F. Holenstein, and J. Sterne. 2003. How important are com- prehensive literature searches and the assessment of trial quality in systematic reviews? Health Technology Assessment 7(1):76. EPC Coordinating Center. 2005. Eidence-based practice centers partner’s guide http://www. ahrq.gov/clinic/epcpartner/epcpartner.pdf (accessed January 25, 2007). Francis, J., and J. B. Perlin. 2006. Improving performance through knowledge translation in the Veterans Health Administration. Journal of Continuing Education in the Health Professions 26(1):63-71. Garfield, E. 2006. The history and meaning of the journal impact factor. JAMA 295(1): 90-93. Glass, G. V. 1976. Primary, secondary and meta-analysis. Educational Researcher 5(10):3-8. Glass, G. V., B. McGaw, and M. L. Smith. 1981. Meta-analysis in social research. Newbury Park, CA: Sage Publications. Glasziou, P., and B. Haynes. 2005. The paths from research to improved health outcomes. ACP Journal Club 142(2):A8-A10. Glasziou, P., J. Vandenbroucke, and I. Chalmers. 2004. Assessing the quality of research. BMJ 328(7430):39-41. Glenny, A. M., M. Esposito, P. Coulthard, and H. V. Worthington. 2003. The assessment of systematic reviews in dentistry. European Journal of Oral Sciences 111:85-92. Gluud, L. L. 2006 Bias in clinical intervention research. American Journal of Epidemiology 163(6):493-501. Goldberger, J. 1907. Typhoid bacillus carriers. Edited by Rosenau, M. J., L. L. Lumsden, and J. H. Kastle. Report on the origin and prevalence of typhoid fever in the District of Columbia. Hygienic Laboratory Bulletin No.  167-174. Golder, S., H. M. McIntosh, S. Duffy, and J. Glanville. 2006. Developing efficient search strate- gies to identify reports of adverse effects in Medline and Embase. Health Information and Libraries Journal 23(1):3-12. Gøtzsche, P. C., A. Hrobjartsson, K. Maric, and B. Tendal. 2007. Data extraction errors in meta-analyses that use standardized mean differences. JAMA 298(4):430-437. GRADE Working Group. 2004. Grading quality of evidence and strength of recommenda- tions. BMJ 328(7454):1490. Guyatt, G. H. 1991. Evidence-based medicine. ACP Journal Club 114:A-16. Harris, R. P., M. Helfand, S. H. Woolf, K. N. Lohr, C. D. Mulrow, S. M. Teutsch, and D. Atkins. 2001. Current methods of the U.S. Preventive Services Task Force: A review of the process. American Journal of Preentie Medicine 20(3 Suppl):21-35. Hayden, J. A., P. Cote, and C. Bombardier. 2006. Evaluation of the quality of prognosis studies in systematic reviews. Annals of Internal Medicine 144:427-437. Hayes, Inc. 2007. Welcome to Hayes http://hayesinc.com (accessed May 8, 2007). Haynes, R. B., D. L. Sackett, G. H. Guyatt, and P. Tugwell. 2006. Clinical epidemiology: How to do clinical practice research. 3rd ed. Philadelphia, PA: Lipincott Williams & Wilkins. The Health Industry Forum. 2006. Comparatie effectieness forum: Key themes. Washington, DC: The Health Industry Forum.

OCR for page 81
6 KNOWING WHAT WORKS IN HEALTH CARE Hedges, L. V., and I. Olkin. 1985. Statistical methods for meta-analysis. Orlando, FL: Aca- demic Press. Helfand, M. 2005. Using evidence reports: Progress and challenges in evidence-based decision making. Health Affairs 24(1):123-127. Heres, S., J. Davis, K. Maino, E. Jetzinger, W. Kissling, and S. Leucht. 2006. Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: An exploratory analysis of head-to-head comparison studies of second-generation antipsy- chotics. American Journal of Psychiatry 163(2):185-194. Higgins, J. T., and S. Green. 2006. Cochrane handbook for systematic reiews of interentions ..6 [updated September 006], The Cochrane Library, Issue 4, 2006. Chichester, UK: John Wiley & Sons, Ltd. Hopewell, S., M. Clarke, C. Lefebvre, and R. Scherer. 2007a. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database of Systematic Reiews (2). Hopewell, S., S. McDonald, M. Clarke, and M. Egger. 2007b. Grey literature in meta-analy- ses of randomized trials of health care interventions. Cochrane Database of Systematic Reiews (2). Hypertension. 2007. Instructions to authors http://hyper.ahajournals.org/misc/ifora.shtml (ac- cessed July 30, 2007). ICMJE (International Committee of Medical Journal Editors). 2006. Uniform requirements for manuscripts submitted to biomedical journals http://www.icmje.org (accessed September 5, 2007). Ioannidis, J. P., and J. Lau. 2004. Systematic review of medical evidence. Journal of Law and Policy 12(2):509-535. Ioannidis, J. P., J. W. Evans, P. C. Gøtzsche, R. T. O’Neill, D. Altman, K. Schulz, and D. Moher. 2004. Better reporting of harms in randomized trials: An extension of the CONSORT Statement. Annals of Internal Medicine 141:781-788. IOM (Institute of Medicine). 2007. Learning what works best: The nation’s need for eidence on comparatie effectieness in health care http://www.iom.edu/ebm-effectiveness (ac- cessed April 2007). Jadad, A. R., and H. J. McQuay. 1996. Meta-analyses to evaluate analgesic interventions: A systematic qualitative review of their methodology. Journal of Clinical Epidemiology 49:235-243. Jadad, A. R., M. Moher, G. P. Browman, L. Booker, C. Sigouin, M. Fuentes, and R. Stevens. 2000. Systematic reviews and meta-analyses on treatment of asthma: Critical evaluation. BMJ 320:537-540. JAMA. 2007. Instructions for authors http://jama.ama-assn.org/misc/ifora.dtl (accessed July 12, 2007). Jones, A. P., T. Remmington, P. R. Williamson, D. Ashby, and R. L. Smyth. 2005. High preva- lence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. Journal of Clinical Epidemiology 58:741-742. Jorgensen, A. W., J. Hilden, and P. Gøtzsche. 2006. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: Systematic review. BMJ Online 333:782-786. Journal of Clinical Oncology. 2007. Information for contributers http://jco.ascopubs.org/misc/ ifora.shtml (accessed July 12, 2007). Journal of the National Cancer Institute. 2007. Instructions to authors http://www. oxfordjournals.org/our_journals/jnci/for_authors/index.html (accessed July 30, 2007). Khan, K. S., and J. Kleijnen. 2001. Stage II conducting the review: Phase 4 selection of studies. In CRD Report Number . Edited by Khan, K. S., G. ter Riet, H. Glanville, A. J. Sowden, and J. Kleijnen. York, UK: NHS Centre for Reviews and Dissemination, University of York.

OCR for page 81
7 SYSTEMATIC REVIEWS Khan, K. S., J. Popay, and J. Kleijnen. 2001a. Stage I planning the review: Phase 2 development of a review protocol. In CRD Report Number . Edited by Khan, K. S., G. ter Riet, H. Glanville, A. J. Sowden, and J. Kleijnen. York, UK: NHS Centre for Reviews and Dis- semination, University of York. Khan, K. S., G. ter Riet, J. Popay, J. Nixon, and J. Kleijnen. 2001b. Stage II conducting the review: Phase 5 study quality assessment. In CRD Report Number . Edited by Khan, K. S., G. ter Riet, H. Glanville, A. J. Sowden, and J. Kleijnen. York, UK: NHS Centre for Reviews and Dissemination, University of York. Kunz, R., G. Vist, and A. D. Oxman. 2007. Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Systematic Reiews (2). Lancet. 2007. Information for authors http://www.thelancet.com/authors/lancet/authorinfo/ (accessed July 30, 2007). Lau, J., E. M. Antman, J. Jimenez-Silva, B. Kupelnick, F. Mosteller, and T. C. Chalmers. 1992. Cumulative meta-analysis of therapeutic trials for myocardial infarction. New England Journal of Medicine 327(4):248-254. Lavis, J., H. Davies, A. Oxman, J. Denis, K. Golden-Biddle, and E. Ferlie. 2005. Towards systematic reviews that inform health care management and policy-making. Journal of Health Serices Research and Policy 10(Suppl 1):35-48. Lexchin, J., L. A. Bero, B. Djulbegovic, and O. Clark. 2003. Pharmaceutical industry sponsor- ship and research outcome and quality: Systematic review. BMJ 326:1167-1170. Light, R. J., and D. B. Pillemer. 1984. Summing up. Cambridge, MA: Harvard University Press. Mallen, C., G. Peat, and P. Croft. 2006. Quality assessment of observational studies is not commonplace in systematic reviews. Journal of Clinical Epidemiology 59:765-769. Medicare Payment Advisory Commission. 2007. Chapter 2: Producing comparative effective- ness information. In Report to the Congress: Promoting greater efficiency in Medicare http://www.medpac.gov/documents/Jun07_EntireReport.pdf (accessed June 2007). Meyers, D., H. Halvorson, and S. Luckhaupt. 2007. Eidence synthesis number . Screening for chlamydial infection: A focused eidence update for the U.S. Preentie Serices Task Force. Gaithersburg, MD: Agency for Healthcare Research and Quality. Moher, D., D. J. Cook, S. Eastwood, I. Olkin, D. Rennie, D. F. Stroup, and the QUOROM Group. 1999. Improving the quality of reports of meta-analyses of randomized controlled trials: The QUOROM statement. Lancet 354:1896-1900. Moher, D., A. Jones, L. Lepage, and the CONSORT Group. 2001a. Use of the CONSORT Statement and quality of reports of randomized trials. JAMA 285(15):1992-1995. Moher, D., K. F. Schulz, D. Altman, and the CONSORT Group. 2001b. The CONSORT state- ment: Revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 285(15):1987-1991. Moher, D., J. Tetzlaff, A. C. Tricco, M. Sampson, and D. G. Altman. 2007. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 4(3):447-455. Montori, V., N. Wilczynski, D. Morgan, and R. B. Haynes, for the Hedges Team. 2003. Sys- tematic reviews: A cross-sectional study of location and citation counts. BMC Medicine 1(1):2. Mulrow, C. 1987. The medical review article: State of the science. Annals of Internal Medicine 106:485-488. Mulrow, C., and K. Lohr. 2001. Proof and policy from medical research evidence. Journal of Health Politics, Policy and Law 26(2):249-266. Mulrow, C. D., D. J. Cook, and F. Davidoff. 1997. Systematic reviews: Critical links in the great chain of evidence. Annals of Internal Medicine 126(5):389-391. Neumann, P. J. 2006. Emerging lessons from the Drug Effectiveness Review Project. Health Affairs 25(4):w262-w271.

OCR for page 81
 KNOWING WHAT WORKS IN HEALTH CARE Neumann, P. J., N. Divi, M. T. Beinfeld, B. S. Levine, P. S. Keenan, E. F. Halpern, and G. S. Gazelle. 2005. Medicare’s national coverage decisions, 1999-2003: Quality of evidence and review times. Health Affairs 24(1):243-254. Neumann, P. J., N. Divi, M. T. Beinfeld, and B. S. Levine. 2007 (unpublished). Medicare National Coerage Decision Database. Tufts-New England Medical Center. Sponsored by the Commonwealth Fund. New England Journal of Medicine. 2007. Instructions for submitting a NEW manuscript http://authors.nejm.org/Misc/NewMS.asp (accessed July 12, 2007). NGC (National Guideline Clearinghouse). 2007. Search for cardiology http://www.guideline. gov/search/searchresults.aspx?Type=3&txtSearch=cardiology&num=500 (accessed July 11, 2007). Norris, S. L., and D. Atkins. 2005. Challenges in using nonrandomized studies in system- atic reviews of treatment interventions. Annals of Internal Medicine 142(12 Part 2): 1112-1119. Obstetrics and Gynecology. 2007. Instructions for authors http://www.greenjournal.org/misc/ authors.pdf (accessed July 12, 2007). Oxman, A. D., and G. H. Guyatt. 1988. Guidelines for reading literature reviews. Canadian Medical Association Journal 138:697-703. Oxman, A. D., H. J. Schünemann, and A. Fretheim. 2006. Improving the use of research evi- dence in guideline development: 7. Deciding what evidence to include. Health Research Policy and Systems 4(19). Pearson, K. 1904. Report on certain enteric fever inoculation statistics. BMJ 3:1243-1246. Pediatrics. 2007. Instructions for authors http://mc.manuscriptcentral.com/societyimages/ pediatrics/2004_author_instructions.pdf (accessed July 12, 2007). Peppercorn, J., E. Blood, E. Winer, and A. Partridge. 2007. Association between pharmaceutical involvement and outcomes in breast cancer clinical trials. Cancer 109(7):1239-1246. Perlin, J. B., and J. Kupersmith. 2007. Information technology and the inferential gap. Health Affairs 26(2):w192-w194. Poolman, R., P. Struijs, R. Krips, I. Sierevelt, K. Lutz, and M. Bhandari. 2006. Does a “Level I Evidence” rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC Medical Research Methodology 6(1):44. Pratt, J. G., J. B. Rhine, B. M. Smith, C. E. Stuart, and J. A. Greenwood. 1940. Extra-sensory perception after sixty years: A critical appraisal of the research in extra-sensory percep- tion. New York: Henry Holt. Radiology. 2007. Publication information for authors http://www.rsna.org/publications/rad/ pdf/pia.pdf (accessed July 11, 2007). Rayleigh, L. 1884. Address by the Rt. Hon. Lord Rayleigh. In Report of the fifty-fourth meeting of the British Association for the Adancement of Science. Edited by Murray, J. Montreal. Reviews in Clinical Gerontology. 2007. Instructions for contributors http://assets.cambridge. org/RCG/RCG_ifc.pdf (accessed July 30, 2007). Richardson, W. S., M. C. Wilson, J. Nishikawa, and R. S. A. Hayward. 1995. The well-built clinical question: A key to evidence-based decisions [editorial]. ACP Journal Club 123: A12-A13. Rietveld, R. P., H. C. P. M. van Weert, G. ter Riet, and P. J. E. Bindels. 2003. Diagnostic impact of signs and symptoms in acute infectious conjunctivitis: Systematic literature search. BMJ 327(7418):789. Rosenthal, R. 1978. Combining results of independent studies. Psychological Bulletin 85: 185-193. Santaguida, P., M. Helfand, and P. Raina. 2005. Challenges in systematic reviews that evaluate drug efficacy or effectiveness. Annals of Internal Medicine 142(12 Part 2):1066-1072.

OCR for page 81
9 SYSTEMATIC REVIEWS Scherer, R. W., P. Langenberg, and E. von Elm. 2007. Full publication of results initially pre- sented in abstracts. Cochrane Database of Systematic Reiews (2). Scholes, D., A. Stergachis, F. E. Heidrich, H. Andrilla, K. K. Holmes, and W. E. Stamm. 1996. Prevention of pelvic inflammatory disease by screening for cervical chlamydial infection. New England Journal of Medicine 334(21):1362-1366. Schünemann, H., D. Best, G. Vist, and A. D. Oxman. 2003. Letters, numbers, symbols and words: How to communicate grades of evidence and recommendations. Canadian Medi- cal Association Journal 169(7):677-680. Schünemann, H., A. Oxman, and A. Fretheim. 2006. Improving the use of research evidence in guideline development: 6. Determining which outcomes are important. Health Research Policy and Systems 4(1):18. Shea, B., D. Moher, I. Graham, B. A. Pham, and P. Tugwell. 2002. A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals. Ealua- tion and the Health Professions 25:116-129. Shekelle, P. G., E. Ortiz, S. Rhodes, S. C. Morton, M. P. Eccles, J. M. Grimshaw, and S. H. Woolf. 2001. Validity of the Agency for Healthcare Research and Quality Clinical Practice Guidelines: How quickly do guidelines become outdated? JAMA 286:1461-1467. Shojania, K. G., M. Sampson, M. T. Ansari, J. Ji, S. Doucette, and D. Moher. 2007. How quickly do systematic reviews go out of date? A survival analysis. Annals of Internal Medicine 147:224-233. Sinclair, J., and M. Bracken. 1992. Effectie care of the newborn infant. New York: Oxford University Press. Slutsky, J. 2007. Approaches to priority setting: Identifying topics and selection. Submitted Responses to the HECS Committee Meeting, January , 007. Washington, DC. Song, F., A. J. Eastwood, S. Gilbody, L. Duley, and A. J. Sutton. 2000. Publication and related biases. Health Technology Assessment 4(10). Spine. 2007. Instructions for authors http://edmgr.ovid.com/spine/accounts/ifauth.htm (ac- cessed July 12, 2007). Steinberg, E. P., and B. R. Luce. 2005. Evidence based? Caveat emptor! Health Affairs 24(1):80-92. Stewart, W. F., N. R. Shah, M. J. Selna, R. A. Paulus, and J. M. Walker. 2007. Bridging the inferential gap: The electronic health record and clinical evidence. Health Affairs 26(2): w181-w191. Stroup, D. F., J. A. Berlin, S. C. Morton, I. Olkin, G. D. Williamson, D. Rennie, D. Moher, B. J. Becker, T. A. Sipe, and S. B. Thacker for the Meta-analysis Of Observational Stud- ies in Epidemiology (MOOSE) Group. 2000. Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA 283(15):2008-2012. Sutton, A. J., K. R. Abrams, D. R. Jones, T. A. Sheldon, and F. Song. 2000. Methods for meta- analysis in medical research. London, UK: John Wiley. Tatsioni, A., D. A. Zarin, N. Aronson, D. J. Samson, C. R. Flamm, C. Schmid, and J. Lau. 2005. Challenges in systematic reviews of diagnostic technologies. Annals of Internal Medicine 142(12 Part 2):1048-1055. Treadwell, J. R., S. J. Tregear, J. T. Reston, and C. M. Turkelson. 2006. A system for rating the stability and strength of medical evidence. BMC Medical Research Methodology [electronic resource] 6:52. Tunis, S. 2006. Improing eidence for health care decisions. Presentation to IOM staff, April , 006. Washington, DC. USPSTF (U.S. Preventive Services Task Force). 2007. U.S. Preentie Serices Task Force rat- ings http://www.ahrq.gov/clinic/uspstf07/ratingsv2.htm (accessed July 10, 2007).

OCR for page 81
0 KNOWING WHAT WORKS IN HEALTH CARE von Elm, E., D. G. Altman, M. Egger, S. J. Pocock, P. C. Gøtzsche, J. P. Vandenbroucke, and the Strobe Initiative. 2007. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for reporting observational studies. Annals of Internal Medicine 147(8):573-577. West, S., V. King, T. Carey, K. Lohr, N. McCoy, S. Sutton, and L. Lux. 2002. Systems to rate the strength of scientific eidence. Eidence Report/Technology Assessment No. 7. (Pre- pared by the Research Triangle Institute-Uniersity of North Carolina Eidence-based Practice Center under Contract No. 90-97-00.) AHRQ Publication No. 0-E06. Rockville, MD: Agency for Healthcare Research and Quality. Whiting, P., A. W. Rutjes, J. Dinnes, J. B. Reitsma, P. M. Bossuyt, and J. Kleijnen. 2005. A systematic review finds that diagnostic reviews fail to incorporate quality despite avail- able tools. Journal of Clinical Epidemiology 58:1-12. Wieland, S., and K. Dickersin. 2005. Selective exposure reporting and Medline indexing limited the search sensitivity for observational studies of the adverse effects of oral con- traceptives. Journal of Clinical Epidemiology 58(6):560-567. Wilczynski, N. L., D. Morgan, R. B. Haynes, and the Hedges Team. 2005. An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Medical Informatics and Decision Making 5(20). Wilensky, G. R. 2006. Developing a center for comparative effectiveness information. Health Affairs w572. World Health Organization. 2007. International clinical trials registry platform http://www. who.int/ictrp/en/ (accessed August 9, 2007). Yusuf, S., R. Peto, J. Lewis, R. Collins, and P. Sleight. 1985. Beta blockade during and after myocardial infarction: An overview of the randomized trials. Progress in Cardioascular Diseases 27(5):335-371.