National Academies Press: OpenBook

Finding What Works in Health Care: Standards for Systematic Reviews (2011)

Chapter: 5 Standards for Reporting Systematic Reviews

« Previous: 4 Standards for Synthesizing the Body of Evidence
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

5
Standards for Reporting Systematic Reviews

Abstract: Authors of publicly sponsored systematic reviews (SRs) should produce a detailed, comprehensive final report. The committee recommends three related standards for documenting the SR process, responding to input from peer reviewers and other users and stakeholders, and making the final report publicly available. The standards draw extensively from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. The committee recommends several reporting items in addition to the PRISMA requirements to ensure that the final report (1) describes all of the steps and judgments required by the standards in the previous chapters and (2) focuses on informing patient and clinical decision making.

High-quality systematic review (SR) reports should accurately document all of the steps and judgments in the SR process using clear language that is understandable to users and stakeholders. A report should provide enough detail that a knowledgeable reader could reproduce the SR. The quality of a final report has profound implications for patients and clinicians. Too often the information that researchers report in published SRs does not adequately reflect

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

their study methods (Devereaux et al., 2004).1 If SRs are poorly reported, patients and clinicians have difficulty determining whether an SR is trustworthy enough to be used to guide decision making or the development of clinical practice guidelines (Moher et al., 2007). High-quality SR reports summarize the methodological strengths and weaknesses of the SR and include language designed to help nonexperts interpret and judge the value of the SR (AHRQ, 2010b; CRD, 2010a; Higgins and Green, 2008; Liberati et al. 2009; Moher et al. 2009). However, according to an extensive literature, many published SRs inadequately document important aspects of the SR process (Delaney et al., 2005, 2007; Golder et al., 2008; McAlister et al., 1999; Moher et al., 2007; Mulrow, 1987; Roundtree et al., 2008; Sacks et al., 1987). A seminal study conducted by Mulrow, for example, assessed 50 review articles published in four leading medical journals and found that many reviews failed to report the methods of identifying, selecting, and validating information, and choosing areas for future research (Mulrow, 1987). More recently, Moher and colleagues (2007) evaluated 300 SRs indexed in MEDLINE during November 2004. They concluded that information continues to be poorly reported, with many SRs failing to report key components of SRs, such as assessing for publication bias, aspects of the searching and screening process, and funding sources. Other studies have found that SRs published in journals often inadequately report search strategies, validity assessments of included studies, and authors’ conflicts of interest (Delaney et al., 2005; Golder et al., 2008; Roundtree et al., 2008).

Authors of all publicly sponsored SRs must produce a detailed final report, which is typically longer and more detailed than the version submitted for journal publication. The sponsor typically publishes the final report on its website, where it stands as the definitive documentation of the review. The standards recommended by the committee apply to this definitive comprehensive final report. The committee recommends three standards for producing a comprehensive SR final report (Box 5-1), including standards for documenting the SR process, responding to input from peer reviewers and other users and stakeholders, and making the final reports publicly available. Each standard includes elements of performance that the committee deems essential. The evidence base for developing standards for the final report is sparse. In addition, most evaluations of the quality of published SRs have focused on

1

See Chapter 3 for a review of the literature on reporting bias and dearth of adequate documentation in most SRs of comparative effectiveness.

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

BOX 5-1

Recommended Standards for Reporting Systematic Reviews

Standard 5.1 Prepare the final report using a structured format

Required elements:

5.1.1

Include a report title*

5.1.2

Include an abstract*

5.1.3

Include an executive summary

5.1.4

Include a summary written for the lay public

5.1.5

Include an introduction (rationale and objectives)*

5.1.6

Include a methods section. Describe the following:

  • Research protocol*

  • Eligibility criteria (criteria for including and excluding studies in the sysematic review)*

  • Analytic framework and key questions

  • Databases and other information sources used to identify relevant studies*

  • Search strategy*

  • Study selection process*

  • Data extraction process*

  • Methods for handling missing information*

  • Information to be extracted from included studies*

  • Methods to appraise the quality of individual studies*

  • Summary measures of effect size (e.g., risk ratio, difference in means)*

  • Rationale for pooling (or not pooling) results of included studies

  • Methods of synthesizing the evidence (qualitative and meta-analysis*)

  • Additional analyses, if done, indicating which were prespecified*

5.1.7

Include a results section. Organize the presentation of results around key questions. Describe the following (repeat for each key question):

  • Study selection process*

  • List of excluded studies and reasons for their exclusion*

  • Appraisal of individual studies’ quality*

  • Qualitative synthesis

  • Meta-analysis of results, if performed (explain rationale for doing one)*

  • Additional analyses, if done, indicating which were prespecified*

  • Tables and figures

5.1.8

Include a discussion section. Include the following:

  • Summary of the evidence*

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

 

  • Strengths and limitations of the systematic review*

  • Conclusions for each key questions*

  • Gaps in evidence

  • Future research needs

5.1.9

Include a section describing funding sources* and COI

Standard 5.2 Peer review the draft report

Required elements:

5.2.1

Use a third party to manage the peer review process

5.2.2

Provide a public comment period for the report and publicly report on disposition of comments

Standard 5.3 Publish the final report in a manner that ensures free public access

  

* Indicates items from the PRISMA checklist. (The committee endorses all of the PRISMA checklist items.)

journal articles rather than SR reports. The committee developed the standards by first reviewing existing expert guidance, particularly the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist (Liberati et al., 2009). However, PRISMA is focused on journal articles, not comprehensive final reports to public sponsors. The committee recommended including items that were not on the PRISMA checklist because it believed that the report of an SR should describe all the steps and judgments required by the committee’s standards in Chapters 2 through 4 to improve the transparency of the SR process and to inform patient and clinical decision making. The committee also took into account the legislatively mandated reporting requirements for the Patient-Centered Outcomes Research Institute (PCORI), as specified by the 2010 Patient Protection and Affordable Care Act (ACA). Box 5-2 describes the ACA reporting requirements for research funded by PCORI. See Appendix G for the Agency for Healthcare Research and Quality (AHRQ), Centre for Reviews and Dissemination (CRD), and Cochrane Collaboration guidance on writing an SR final report. Appendix H contains the PRISMA checklist.

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

BOX 5-2

Requirements for Research Funded by the Patient-Centered Outcomes Research Institute

The 2010 Patient Protection and Affordable Care Act created the Patient-Centered Outcomes Research Institute (PCORI), a nonprofit corporation intended to advance comparative effectiveness research. The act stipulates that research funded by PCORI, including systematic reviews, adhere to the following reporting and publication requirements:

  • For each research study, the following information should be posted on PCORI’s website:

    • A research protocol, including measures taken, methods of research and analysis, research results, and other information the institute determines appropriate.

    • The research findings conveyed in a manner that is comprehensible and useful to patients and providers in making healthcare decisions.

    • Considerations specific to certain subpopulations, risk factors, and comorbidities, as appropriate.

    • The limitations of the research and what further research may be needed as appropriate.

    • The identity of the entity and the investigators conducting the research.

    • Conflicts of interest, including the type, nature, and magnitude of the interests.

  • PCORI is required to:

    • Provide a public comment period for systematic reviews to increase public awareness, and to obtain and incorporate public input and feedback on research findings.

    • Ensure there is a process for peer review to assess a study’s scientific integrity and adherence to methodological standards.

    • Disseminate research to physicians, healthcare providers, patients, payers, and policy makers.

SOURCE: The Patient Protection and Affordable Care Act, Public Law 111-148, 111th Cong., Subtitle D, § 6301 (March 23, 2010).

REPORTING GUIDELINES

Over the past decade, several international, multidisciplinary groups have collaborated to develop guidelines for reporting the methods and results of clinical research (reporting guidelines). Reporting guidelines exist for many types of health research (Ioannidis et al., 2004; Liberati et al., 2009; Moher et al., 1999, 2001a,b,

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

2009; Stroup et al., 2000). These guideline initiatives were undertaken out of concern that reports on health research were poorly documenting the methods and results of the research studies (IOM, 2008). Detailed reporting requirements are also seen as a line of defense against reporting bias.2 For SRs to be trustworthy enough to inform healthcare decisions, accurate, thorough, and transparent reporting are essential. The adoption of reporting guidelines furthers this goal. Examples of reporting guidelines include the Consolidated Standards of Reporting Trials (CONSORT) statement for reporting randomized clinical trials (Ioannidis et al., 2004; Moher et al., 2001b), and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for reporting observational studies in epidemiology (von Elm et al., 2007). The major reporting guideline for SRs and meta-analyses is PRISMA (Liberati et al., 2009; Moher et al., 2009), an update to the 1999 Quality of Reporting of Meta-analyses (QUOROM) statement (Moher et al., 1999). In 2006, the Enhancing Quality and Transparency of Health Research (EQUATOR) Network was launched to coordinate initiatives to promote transparent and accurate reporting of health research and to assist in the development of reporting guidelines (EQUATOR Network, 2010). See Box 5-3 for a historical overview of reporting guidelines.

The methodological quality of SRs (i.e., how well the SR is conducted) is distinct from reporting quality (i.e., how well reviewers report their methodology and results) (Shea et al., 2007). Whether reporting guidelines improve the underlying methodological quality of research studies is unknown. However, incomplete documentation of the SR process makes it impossible to evaluate its methodological quality, so that it is impossible to tell whether a step in the SR process was performed correctly but not reported (poor reporting quality), completed inadequately, or not completed at all and therefore not reported (poor methodological quality).

At present, the evidence that reporting guidelines improve the quality of reports of SRs and meta-analyses is weak. The few observational studies that have addressed the issue have serious flaws. For example, Delaney and colleagues (2005) compared the quality of reports of meta-analyses addressing critical care, including topics related to shock, resuscitation, inotropes, and mechanical ventilation, published before and after the release of the QUOROM statement (the precursor to PRISMA). They found that reports of meta-analyses published after QUOROM were of higher quality

2

See Chapter 3 for a discussion on reporting bias.

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

BOX 5-3

A History of Reporting Guidelines for Comparative Effectiveness Research

In 1993 the Standards for Reporting Trials (SORT) group met to address inadequate reporting of randomized controlled trials (RCTs). This group developed the concept of a structured reporting guideline, and proposed a checklist of essential items for reporting RCTs. Five months later the Asilomar Working group met independently to discuss challenges in reporting RCTs and developed a reporting checklist. The Consolidated Standards of Reporting Trials (CONSORT) statement was developed in 1996 and consolidated the recommendation from both groups. The CONSORT statement consists of a checklist of reporting items, such as the background, methods, results, discussion, and conclusion sections, as well as a flow diagram for documenting participants through the trial. Many journals have adopted the CONSORT statement. It has been extended to address a number of specific issues in the reporting of RCTs (e.g., reporting of harms, noninferiority and equivalence RCTs, cluster RCTs).

Following the success of the CONSORT statement, two international groups of review authors, methodologists, clinicians, medical editors, and consumers developed standard formats for reporting systematic reviews (SRs) and meta-analyses: Quality of Reporting of Meta-analyses (QUOROM) and Meta-analysis of Observational Studies in Epidemiology (MOOSE). The statements consist of checklists of items to include in reports and flow diagrams for documenting the search process. However, unlike CONSORT, reporting guidelines for SRs and meta-analyses have not been widely adopted by prominent journals.

In 2009, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement was published to update the QUOROM statement. According to its developers, PRISMA reflects the conceptual and practical advances made in the science of SRs since the development of QUOROM. These conceptual advances include the following: completing an SR is an iterative process; the conduct and reporting of research are distinct processes; the assessment of risk of bias requires both a study-level assessment (e.g., adequacy of allocation concealment) and outcome-level assessment (i.e., reliability and validity of the data for each outcome); and the importance of addressing reporting bias. PRISMA decouples several checklist items that were a single item on the QUOROM checklist and links other items to improve the consistency across the SR report. PRISMA was funded by the Canadian Institutes of Health Research; Universita di Modena e Reggio Emilia, Italy; Cancer Research U.K.; Clinical Evidence BMJ Knowledge; The Cochrane Collaboration; and GlaxoSmithKline, Canada.a It has been endorsed by a number of organizations and journals, including the Centre for Reviews and Dissemination, Cochrane Collaboration, British Medical Journal, and Lancet.b

  

a The following Institute of Medicine committee members were involved in the development of PRISMA: Jesse Berlin, Kay Dickersin, and Jeremy Grimshaw.

  

b See the following website for a full list of organizations endorsing PRISMA: http://www.prisma-statement.org/endorsers.htm (accessed July 14, 2010).

SOURCES: Begg et al. (1996); Ioannidis et al. (2004); IOM (2008); Liberati et al. (2009); Moher et al. (1999, 2001a, 2001b, 2007, 2009); Stroup et al. (2000).

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

than reports published before and were more likely to describe whether a comprehensive literature search was conducted; the criteria for screening the studies; and the methods used to combine the findings of relevant studies (Delaney et al., 2005). Mrkobrada and colleagues (2008) evaluated 90 SRs published in 2005 in the field of nephrology. They found that only a minority of journals (4 out of 48) recommended adherence to SR reporting guidelines. The four journals that endorsed or adopted reporting guidelines published SRs of significantly higher methodological quality than the other journals, and were more likely to report assessing methodological quality of included studies and taking precautions to avoid bias in study selection. Neither of these studies, however, assessed whether the journals endorsing QUOROM published higher quality reviews than the other journals prior to the adoption of QUOROM. In addition, journals that endorse reporting guidelines, such as QUOROM, may merely recommend that authors comply with the reporting items, but may not require authors to show compliance by submitting a checklist stating whether or not they adhered to each item as a condition of accepting the SR for review. As a result, whether the reporting improvements were due to QUOROM or other developments in the field is unclear. No controlled trials have evaluated the effectiveness of PRISMA on improving the reporting of SRs (Liberati et al., 2009).

In light of this history of reporting guidelines for medical journals, the committee decided to develop reporting guidelines specifically for the final report to the sponsor of an SR. The committee intends for its reporting requirements to improve the documentation of SR final report study methodology and results, and to increase the likelihood that SR final reports will provide enough information for patients and clinicians to determine whether an SR is trustworthy enough to be used to guide decision making.

SYSTEMATIC REVIEWS PUBLISHED IN JOURNALS

The committee recognizes that a journal publishing SRs will choose the level of documentation that is most appropriate for its readers. It also recognizes that its reporting requirements for final reports to public sponsors of SRs are quite detailed and comprehensive, and will produce manuscripts that are too long and too detailed for most journals to publish in full. Ideally, all published SRs (both final reports to sponsors and journal publications) should follow one reporting standard. With the advent of electronic-only appendixes to journal articles, journals can now require authors to

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

meet the committee’s full reporting guidelines (i.e., journals can post any reporting items not included in the actual journal publication in an online appendix). Alternatively, journals can publish a link to the website of the full SR report to the public sponsor, explaining what information readers would find only at the sponsor’s website.

RECOMMENDED STANDARD FOR PREPARING THE FINAL REPORT

The committee recommends the following standard for preparing the final report:

Standard 5.1—Prepare the final report using a structured format

Required elements:

5.1.1

Include a report title

5.1.2

Include an abstract

5.1.3

Include an executive summary

5.1.4

Include a summary written for the lay public

5.1.5

Include an introduction (rationale and objectives)

5.1.6

Include a methods section

5.1.7

Include a results section. Organize the presentation of results around key questions

5.1.8

Include a discussion section

5.1.9

Include a section describing funding sources and COI

Rationale

All SR reports to public sponsors should use a structured format to help guide the readers to relevant information, to improve the documentation of the SR process, and to promote consistency in reporting. More than 150 journals have adopted the PRISMA requirements (PRISMA, 2010). Because of this support, the committee used the PRISMA checklist as its starting point for developing its reporting standards. However, PRISMA is focused on journal articles, which are usually subject to length restrictions in the print version of the article, and the committee’s reporting standards are directed at comprehensive, final reports to public sponsors (e.g., AHRQ, PCORI), which typically do not have word limits. Most of the committee’s additions and revisions to PRISMA were necessary to make the standards for the final report consistent with all of the steps and judgments in the SR process required by the standards for

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

performing an SR, as recommended in Chapters 2 through 4 of this report. In addition, the committee added several items to PRISMA because of the committee’s focus on setting standards for public agencies that sponsor SRs of comparative effectiveness research (CER), which place a strong emphasis on generating evidence to inform patient and clinical decision making.

Therefore, the committee’s reporting recommendations build on PRISMA, but incorporate the following revisions: greater specificity in reporting the data collection and study selection process, and eight new checklist items. The checklist items are as follows: (1) an executive summary, (2) a summary written for the lay public, (3) an analytic framework and description of the chain of logic for how the intervention may improve a health outcome, (4) rationale for pooling (or not pooling) results across studies, (5) results of the qualitative synthesis, including findings of differences in responses to the intervention for key subgroups (this requirement reflects a specific characteristic of CER: the search for evidence to help patients and clinicians tailor the decisions to the characteristics and needs of the individual patient), (6) tables and figures summarizing the results, (7) gaps in evidence, and (8) future research needs.

The following sections present the committee’s recommendations for the key components of a final SR report: title, abstract and summaries, introduction, methods, results, discussion, and funding and conflict-of-interest (COI) sections of SR reports (see Box 5-1 for a complete list of all required reporting elements).

Report Title

The title should identify the report as an SR, a meta-analysis, or both (if appropriate). This may improve the indexing and identification of SRs in bibliographic databases (Liberati et al., 2009). The title should also reflect the research questions addressed in the review in order to help the reader understand the scope of the SR. PRISMA provides the following example of a clear title: “Recurrence Rates of Video-assisted Thoracoscopic versus Open Surgery in the Prevention of Recurrent Pneumothoraces: A Systematic Review of Randomized and Nonrandomized Trials” (Barker et al., 2007; Liberati et al., 2009).

Abstract, Executive Summary, and Plain-Language Summary

The SR final report should include a structured abstract organized under a series of headings corresponding to the background,

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

methods, results, and conclusions (Haynes et al., 1990; Mulrow et al., 1988). A structured abstract helps readers to quickly determine the scope, processes, and findings of a review without reading the entire report. Structured abstracts also give the reader more complete information than unstructured abstracts (Froom and Froom, 1993; Hartley, 2000; Hartley et al., 1996; Pocock et al., 1987). In SR final reports, the abstract should address, as applicable: background; objectives; data sources; study eligibility criteria (inclusion/exclusion criteria), participants, and interventions; study appraisal and synthesis methods; results; appraisal of the body of evidence; limitations; conclusions and implications of key findings; and SR registration number3 (Liberati et al., 2009). See Box 5-4 for an example of a structured abstract.

The final report should also include an executive summary. Many users and stakeholders find concise summaries that highlight the main findings and allow for rapid scanning of results very useful (Lavis et al., 2005; Oxman et al., 2006). Because the length of abstracts is often limited they may not provide enough information to satisfy decision makers. The committee’s recommendation to include an executive summary and abstract in final reports is consistent with guidance from AHRQ and CRD (AHRQ, 2009a; CRD, 2009).

SR reports, including their abstracts and executive summaries, are often written in language that is too technical for consumers and patients to use in decision making. This is especially problematic for SRs of CER studies because one of the major goals of CER is to help patients and consumers make healthcare decisions (IOM, 2009). To improve the usability of SRs for patients and consumers, the committee recommends that final reports include summaries written in nontechnical language (the plain-language summary) (see Box 5-5 for an example). The plain-language summary should include background information about the healthcare condition, population, intervention, and main findings. The committee believes the plainlanguage summary should explain the shortcomings of the body of evidence, so the public can form a realistic appreciation of the limitations of the science. Developing plain-language summaries requires specialized knowledge and skills. An important resource in this area is the John M. Eisenberg Clinical Decisions and Communications Science Center at Baylor College of Medicine in Houston, Texas. The Center, with AHRQ funding, translates SRs of CER conducted by the

3

An SR registration number is the unique identification number assigned to a protocol in an electronic registry. See Chapter 2 for a discussion on protocol publication.

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

BOX 5-4

Example of a Structured Abstract: Clinical Utility of Cancer Family History Collection in Primary Care

Objectives: This systematic review aimed to evaluate, within unselected populations, the:

  1. Performance of family history (FHx)-based models in predicting cancer risk.

  2. Overall benefits and harms associated with established cancer prevention interventions.

  3. Impact of FHx-based risk information on the uptake of preventive interventions.

  4. Potential for harms associated with collecting cancer FHx.

Data sources: MEDLINE, EMBASE, CINAHL Cochrane Central, Cochrane Database of Systematic Reviews, and PsycINFO were searched from 1990 to June 2008. Cancer guidelines and recommendations were searched from 2002 forward and systematic reviews from 2003 to June 2008.


Review methods: Standard systematic review methodology was employed. Eligibility criteria included English studies evaluating breast, colorectal, ovarian, or prostate cancers. Study designs were restricted to systematic review, experimental and diagnostic types. Populations were limited to those unselected for cancer risk. Interventions were limited to collection of cancer FHx; primary and/or secondary prevention interventions for breast, colorectal, ovarian, and prostate cancers.


Results:

  • Accuracy of models: Seven eligible studies evaluated systems based on the Gail model, and on the Harvard Cancer Risk Index. No evaluations demonstrated more than modest discriminatory

EPCs into short, easy-to-read guides and tools that can be used by consumers, clinicians, and policy makers (AHRQ, 2010b).

Advice about the best method of presenting the research results for a consumer audience has a substantial body of evidence to support it (Akl et al., in press; Glenton, 2002; Glenton et al., 2006a; Glenton et al., 2006b; Lipkus, 2007; Santesso et al., 2006; Schünemann et al., 2004; Schwartz et al., 2009; Trevena et al., 2006; Wills and Holmes-Rovner, 2003). For example, Glenton (2010) conducted a series of semi-structured interviews with members of the public and

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
  • accuracy at an individual level. No evaluations were identified relevant to ovarian or prostate cancer risk.

  • Efficacy of preventive interventions: From 29 eligible systematic reviews, 7 found no experimental studies evaluating interventions of interest. Of the remaining 22, none addressed ovarian cancer prevention. The reviews were generally based on limited numbers of randomized or controlled clinical trials. There was no evidence either to support or refute the use of selected chemoprevention interventions, there was some evidence of effectiveness for mammography and fecal occult blood testing.

  • Uptake of intervention: Three studies evaluated the impact of FHx-based risk information on uptake of clinical preventive interventions for breast cancer. The evidence is insufficient to draw conclusions on the effect of FHx-based risk information on change in preventive behavior.

  • Potential harms of FHx taking: One uncontrolled trial evaluated the impact of FHx-based breast cancer risk information on psychological outcomes and found no evidence of significant harm.

Conclusions: Our review indicates a very limited evidence base with which to address all four of the research questions:

  1. The few evaluations of cancer risk prediction models do not suggest useful individual predictive accuracy.

  2. The experimental evidence base for primary and secondary cancer prevention is very limited.

  3. There is insufficient evidence to assess the effect of FHx-based risk assessment on preventive behaviors.

  4. There is insufficient evidence to assess whether FHx-based personalized risk assessment directly causes adverse outcomes.

SOURCE: AHRQ (2009b).

found that summarizing SR results using both qualitative statements and numbers in tables improves consumer comprehension (Glenton, 2010). Other research has found that consumer comprehension is improved if authors use frequencies (e.g., 1 out of 100) rather than percentages or probabilities; use a consistent numeric format to summarize research results; and use absolute risk rather than relative risk (Akl et al., in press; Lipkus, 2007; Wills and Holmes-Rovner, 2003). The recommendation to include a plain-language summary follows guidance from AHRQ and Cochrane (AHRQ, 2010a; Higgins

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

BOX 5-5

Example of a Plain-Language Summary: Antenatal Corticosteroids for Accelerating Fetal Lung Maturation for Women at Risk of Preterm Birth

Corticosteroids given to women in early labor help the babies’ lungs to mature and so reduce the number of babies who die or suffer breathing problems at birth.

Babies born very early are at risk of breathing difficulties (respiratory distress syndrome) and other complications at birth. Some babies have developmental delay and some do not survive the initial complications. In animal studies, corticosteroids are shown to help the lungs to mature and so it was suggested these drugs may help babies in preterm labor too. This review of 21 trials shows that a single course of corticosteroid, given to the mother in preterm labor and before the baby is born, helps to develop the baby’s lungs and reduces complications like respiratory distress syndrome. Furthermore, this treatment results in fewer babies dying and fewer common serious neurological and abdominal problems, e.g. cerebroventricular haemorrhage and necrotising enterocolitis, that affect babies born very early. There does not appear to be any negative effects of the corticosteroid on the mother. Long-term outcomes on both baby and mother are also good.


SOURCE: Roberts and Dalziel (2006).

and Green, 2008). Also consistent with the requirement is that PCORI convey the research findings so patients can understand and apply them to their personal circumstances.4

Introduction to the Final Report

The introduction section of an SR final report should describe the research questions as well as the rationale for undertaking the review. The description should address the perspectives of both patients and clinicians, the current state of knowledge, and what the SR aims to add to the body of knowledge. It should indicate whether the review is new or an update of an existing one. If it is an update, the authors should state why the update is needed and describe in general terms how the evidence base has changed since the previous

4

The Patient Protection and Affordable Care Act, Public Law 111-148, 111th Cong., Subtitle D, § 6301(d)(8)(A)(i).

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

review (e.g., three new, large randomized controlled trials have been published in the past 2 years).

Methods Section

Detailed reporting of methods is important because it allows the reader to assess the reliability and validity of the review. Table 5-1 lists and describes the topics that should be included in the methods section.

Results Section

The results section should logically lay out the key findings from the SR and include all the topics described in Table 5-2.

Discussion Section

The discussion should include a summary of the main findings; the strength of evidence; a general interpretation of the results for each key question; the strengths and limitations of the study; and gaps in evidence, including future research needs. The discussion should draw conclusions only if they are clearly supported by the evidence (Docherty and Smith, 1999; Higgins and Green, 2008). At the same time, the discussion should provide an interpretation of the data that are useful to users and stakeholders. The peer review process often improves the quality of discussion sections and can provide an evaluation of whether the authors went beyond the evidence in their interpretation of the results (Goodman et al., 1994).

Future research is particularly important for authors to discuss because most SRs identify significant gaps in the body of evidence (Clarke et al., 2007). The ACA language specifies that reports funded by PCORI should “include limitations of the research and what further research may be needed as appropriate.”5 Policy makers and research funders rely on well-written discussions of future research needs to set research agendas and funding priorities. When information gaps are reported clearly, SRs can bring attention to future research needs. Odierna and Bero, for example, used Drug Effectiveness Review Project SRs to identify the need for better drug studies in non-white and economically disadvantaged populations (Odierna and Bero, 2009).

5

The Patient Protection and Affordable Care Act, Public Law 111-148, 111th Cong., Subtitle D, § 6301(d)(8)(A)(iii) (March 23, 2010).

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

TABLE 5-1 Topics to Include in the Methods Section

Methods Topic

Include

Research protocol

  • Rationale for deviations from the protocol in the conduct of the systematic review (SR)

  • Registration number (if applicable)

Eligibility criteria (for including and excluding studies in the SR)

  • Research designs (trials, observational studies), patients, interventions, comparators, outcomes, length of follow-up

  • Report characteristics (e.g., publication period, language)

  • Rationale for each criterion

Analytic framework and key questions

  • A diagram illustrating the chain of logic describing the mechanism by which the intervention could improve a health outcome

  • Key questions written in a structured format (e.g., PICO[TS])

Databases and other information sources

  • All sources of information about potentially eligible articles (including contact with study authors)

  • Date of last search

Search strategy

  • Electronic search strategy for at least one database, including any limits used and the date of searches (include all search strategies in an electronic appendix)

Study selection

  • Process for screening studies, including the number of individual screeners and their qualifications

  • Process for resolving differences among screeners

Data extraction

  • Process for extracting data from included studies, including the data collection form, number of individual data extractors and their qualifications, and whether more than one person independently extracted data from the same study

  • Process for resolving differences among extractors

Missing information

  • Researchers contacted, information requested, and success of requests

Information to be extracted

  • All variables for which data were sought (e.g., PICO[TS])

  • Any assumptions made about missing and unclear data

Appraisal of individual studies

  • Description of how risk of bias was assessed

  • Description of how the relevance of the studies to the populations, interventions, and outcome measures was assessed

  • Description of how the fidelity of the implementation of interventions was assessed

Summary measures

  • Principal summary measures (e.g., risk ratio, difference in means)

Data pooling across studies

  • Rationale for pooling decision

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Methods Topic

Include

Synthesizing the results

  • Summary of qualitative and quantitative synthesis methods, including how heterogeneity, sensitivity, and statistical uncertainty were addressed

  • Description of the methods for assessing the characteristics of the body of evidence

Additional analyses

  • Description of analyses (e.g., subgroup analyses, meta-regression) not prespecified in the protocol

NOTE: PICO(TS) = population, intervention, comparator, outcome, timing, and setting.

Unfortunately, many SRs are not explicit when recommending future research and not specific enough about recommending types of participants, interventions, or outcomes that need additional examination (Clarke et al., 2007). The EPICOT acronym is a helpful guide for organizing the discussion of future research needs: Evidence, Population, Intervention, Comparison, Outcomes, and Time Stamp (see Table 5-3 for an example recommending that further evidence be collected on the efficacy and adverse effects of intensive blood-pressure lowering in representative populations) (Brown et al., 2006). This tool indicates that recommendations on future research needs should be specific on all of the PICO elements that are required in SR topic formulation (see Chapter 2). The discussion should also report the strength of existing evidence on the topic, using consistent language when discussing different studies (see Chapter 4) and the date of the most recent literature search or recommendation.

Funding and Conflict-of-Interest Section

The final report should describe the sources of funding for the SR; the role of the funder in carrying out the review (including approval of the content); the review authors’, contributing users’, and stakeholders’ biases and COIs; and how any potential conflicts were managed (See Box 5-6 for examples of how to report funding and COI statements).6 The sponsor of an SR can have a significant impact on the SR process and resulting conclusions. SRs funded by industry, for example, are more likely to favor the sponsor’s product

6

See Chapter 2 for an overview of COI and bias in the review team, and a discussion of the role of the sponsor in the SR process.

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

TABLE 5-2 Topics to Include in the Results Section (repeat for each key question)

Results Topic

Include

Study selection

  • Numbers of studies that were screened, assessed for eligibility, and included in the review

  • A flow chart that shows the number of studies that remain after each stage of the selection process

  • Provide a citation for each included study

Excluded studies

  • Excluded studies that experts might expect to see included and reason for their exclusion

Appraisal of individual studies

  • Summarize the threats to validity in each study and, if available, any outcome-level assessment of the effects of bias

  • Summarize the relevance of the studies to the populations, interventions, and outcome measures

  • Summarize the fidelity of the implementation of interventions

Qualitative synthesis

  • Summarize clinical and methodological characteristics of the included studies, such as:

    • Number and characteristics of study participants, including factors that may impact generalizability of results to real-world settings (e.g., comorbidities in studies of older patients or race/ethnicity in conditions where disparities exist)

    • Clinical settings

    • Interventions

    • Primary and secondary outcome measures

    • Follow-up period

  • Observed patterns of threats to validity across studies, strengths, and weaknesses of the evidence, and confidence in the results

  • Description of the overall body of evidence across the following domains:

    • Risk of bias

    • Consistency

    • Precision

    • Directness

    • Reporting bias

    • Dose-response association

    • Plausible confounding that would change the observed effect

    • Strength of association

  • Findings of differences in responses to the intervention for key subgroups (e.g., by age, race, gender, socioeconomic status, and/or clinical findings)

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Results Topic

Include

Meta-analysis (if performed)

  • Justification for why a pooled estimate might be more useful to decision makers than the results of each study individually

  • Examination of how heterogeneity in the treatment’s effects may be due to clinical differences in the study population or methodological differences in the studies’ design

  • Results of each meta-analysis, including a measure of statistical uncertainty and the sensitivity of the conclusions to changes in the protocol, assumptions, and study selection

Additional analyses

  • If done, results of additional analyses (e.g., subgroup analyses, meta-regression), indicating whether the analysis was prespecified or exploratory

Tables and figures

  • An evidence table summarizing the characteristics of included studies

  • Graphic displays of results (e.g., forest plots to summarize quantitative findings, GRADE summary tables)

NOTE: GRADE = Grading of Recommendations Assessment, Development and Evaluation.

than SRs funded through other sources (Lexchin et al., 2003; Yank et al., 2007). Identifying the sources of funding and the role of the sponsor (including whether the sponsor reserved the right to approve the content of the report) in the final report improves the transparency and is critical for the credibility of the report (Liberati et al., 2009).

Currently, many peer-reviewed publications fail to provide complete or consistent information regarding the authors’ biases and COI (Chimonas et al., 2011; McPartland, 2009; Roundtree et al., 2008). A recent study of payments received by physicians from orthopedic device companies identified 41 individuals who each received $1 million or more in 2007. In 2008 and 2009, these individuals published a total of 95 articles relating to orthopedics. Fewer than half the articles disclosed the authors’ relationships with the orthopedic device manufacturers, and an even smaller number provided information on the amount of the physicians’ payments (Chimonas et al., 2011). Requiring authors to disclose any potential outside influences on their judgment, not just industry relationships, improves the

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

TABLE 5-3 EPICOT Format for Formulating Future Research Recommendations

EPICOT Component

Issues to Consider

Example

E Evidence

What is the current evidence?

One systematic review dominated by a large randomized controlled study conducted in hospital setting

P Population

Diagnosis, disease stage, comorbidity, risk factor, sex, age, ethnic group, specific inclusion or exclusion criteria, clinical setting

Primary care patients with confirmed stroke or transient ischemic attack (mean age ≥ 75 years, female–male ratio 1:1, time since last cerebrovascular event ≥ 1 year)

I Intervention

Type, frequency, dose, duration, prognostic factor

Intensive blood pressure lowering

C Comparison

Placebo, routine care, alternative treatment/management

No active treatment or placebo

O Outcomes

Which clinical or patient-related outcomes will the researcher need to measure, improve, influence, or accomplish? Which methods of measurement should be used?

Major vascular events (stroke, myocardial infarction, vascular death); adverse events, risk of discontinuation of treatment because of adverse events

T Time stamp

Date of literature search or recommendation

February 2006

SOURCE: Brown et al. (2006).

transparency and trustworthiness of the review. The ACA contains a similar requirement for authors of research funded by PCORI.7

RECOMMENDED STANDARD FOR REPORT REVIEW

The committee recommends one overarching standard for review by scientific peers, other users and stakeholders, and the public:

Standard 5.2—Peer review the draft report

Required elements:

5.2.1

Use a third party to manage the peer review process

7

The Patient Protection and Affordable Care Act at § 6301(h)(3)(B).

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

BOX 5-6

Reporting Funding and Conflict of Interest: Selected Examples

Source of Funding

“PRISMA was funded by the Canadian Institutes of Health Research; Universita’ di Modena e Reggio Emilia, Italy; Cancer Research UK; Clinical Evidence BMJ Knowledge; the Cochrane Collaboration; and GlaxoSmithKline, Canada. AL is funded, in part, through grants of the Italian Ministry of University (COFIN–PRIN 2002 prot. 2002061749 and COFIN–PRIN 2006 prot. 2006062298). DGA is funded by Cancer Research UK. DM is funded by a University of Ottawa Research Chair.”


Role of Funders

“None of the sponsors had any involvement in the planning, execution, or write-up of the PRISMA documents. Additionally, no funder played a role in drafting the manuscript.”


Potential Conflicts of Interest

The authors have declared that no competing interests exist.”


SOURCE: Moher et al. (2009).

5.2.2

Provide a public comment period for the report and publicly report on disposition of comments

Rationale

SR final reports should be critically reviewed by peer reviewers to ensure accuracy and clarity and to identify any potential methodological flaws (e.g., overlooked studies, methodological errors). The original protocol for the SR (including any amendments) should be made available to the peer reviewers. A small body of empirical evidence suggests that the peer review process improves the quality of published research by making the manuscripts more readable and improving the comprehensiveness of reporting (Goodman et al., 1994; Jefferson et al., 2002; Weller, 2002). In addition, the critical assessment of manuscripts by peer reviewers is an essential part of the scientific process (ICMJE, 2010). Journals rely on the peer review process to establish when a study is suitable for publication and to improve the quality of reporting and compliance with reporting guidelines (ICMJE, 2010). Some version of peer review is recom-

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

mended in the guidance from all the major producers of SRs (CRD, 2009; Higgins and Green, 2008; Slutsky et al., 2010). Peer review of research funded through PCORI will be required, either directly through a process established by PCORI or by an appropriate medical journal or other entity.8

The evidence is unclear on how to select peer reviewers, the qualifications that are important for peer reviewers to possess, and what type of training and instructions improve the peer review process (Callaham and Tercier, 2007; Jefferson et al., 2002; Schroter et al., 2004, 2006). In the context of publicly funded SRs, the committee recommends that peer reviewers include a range of relevant users and stakeholders, such as practicing clinicians, statisticians and other methodologists, and consumers. This process can be used to gather input from perspectives that were not represented on the review team (e.g., individuals with diverse clinical specialties).

The committee also recommends that the public be given an opportunity to comment on SR reports as part of the peer review process. Allowing public comments encourages publicly funded research that is responsive to the public’s interests and concerns and is written in language that is understandable and usable for patient and clinical decision making. Requiring a public comment period is also consistent with the ACA, which directs PCORI to obtain public input on research findings,9 as well as guidance from AHRQ and Cochrane (Higgins and Green, 2008; Whitlock et al., 2010).

The review team should be responsive to the feedback provided by the peer reviewers and the public, and publicly report how it revised the SR in response to the comments. The authors should document the major comments and input received; how the final report was or was not modified accordingly; and the rationale for the course of action. The authors’ response to this feedback can be organized into general topic areas of response, rather than responding to each individual comment. Requiring authors to report on the disposition of comments holds the review authors accountable for responding to the peer reviewers’ comments and improves the public’s confidence in the scientific integrity and credibility of the SR (Whitlock et al., 2010).

A neutral third party should manage and oversee the entire peer review process. The main role of the third party should be to provide an independent judgment about the adequacy of the authors’ responses (Helfand and Balshem, 2010). This recommendation is

8

The Patient Protection and Affordable Care Act at § 6301(d)(7).

9

The Patient Protection and Affordable Care Act at § 6301(h).

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

consistent with the rules governing PCORI that allow, but do not require, the peer review process to be overseen by a medical journal or outside entity.10 It also furthers SR authors’ accountability for responding to reviewers’ feedback and it is consistent with AHRQ guidance (Helfand and Balshem, 2010; Whitlock et al., 2010). The National Academies has an office that manages the review of all Academies studies. A monitor and coordinator, chosen by the report review office from the membership of the Academies, oversee the response to external review. They must approve the response to review before release of the report.

RECOMMENDED STANDARD FOR PUBLISHING THE FINAL REPORT

The committee recommends one standard for publishing the final report:

Standard 5.3—Publish the final report in a manner that ensures free public access

Rationale

The final report should be publicly available. PCORI will be required to post research findings on a website accessible to clinicians, patients, and the general public no later than 90 days after receipt of the research findings and completion of the peer review process.11 This requirement should be extended to all publicly funded SRs of effectiveness research. Publishing final reports is consistent with leading guidance (AHRQ, 2010c; CRD, 2009; Higgins and Green, 2008) and this committee’s criteria of transparency and credibility. Public sponsors should not prevent the SR team from publishing the SR in a peer-reviewed journal and should not interfere with the journal’s peer review process. Ideally, the public sponsor will cooperate with the journal to ensure timely, thorough peer review, so that journal publication and posting on the sponsor’s website can take place simultaneously. In any case, posting an SR final report on a government website should not qualify as a previous publication, in the same way that journals have agreed that publication of an abstract describing clinical trial results in clinicaltrials.gov (which is required by federal law) does not count as prior publication (ICMJE,

10

The Patient Protection and Affordable Care Act at § 6301(d)(7).

11

The Patient Protection and Affordable Care Act at § 6301(d)(8)(A).

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

2009). In addition, public sponsors should encourage the review team to post the research results in international SR registries, such as the one being developed by the CRD (CRD, 2010b).

REFERENCES

AHRQ (Agency for Healthcare Research and Quality). 2009a. AHRQ Evidence-based Practice Centers partners guide. Gaithersburg, MD: AHRQ.

AHRQ. 2009b. Cancer, clinical utility of family history. http://www.ahrq.gov/clinic/tp/famhist2tp.htm (accessed September 20, 2010).

AHRQ. 2010a. Consumers and patients. http://www.ahrq.gov/consumer/compare.html (accessed September 10, 2010).

AHRQ. 2010b. About the Eisenberg Center. http://www.effectivehealthcare.ahrq.gov/index.cfm/who-is-involved-in-the-effective-health-care-program1/about-the-eisenbergcenter/ (accessed December 27, 2010).

AHRQ. 2010c. Search for guides, reviews, and reports. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/ (accessed July 2, 2010).

AHRQ. 2010c. Search for guides, reviews, and reports. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/ (accessed July 2, 2010).

Akl, E., A. D. Oxman, J. Herrin, G. E. Vist, I. Terrenato, F. Sperati, C. Costiniuk, D. Blank, and H. Schunemann. In press. Using alternative statistical formats for presenting risks and risk reduction. Cochrane Database of Systematic Reviews.

Barker, A., E. C. Maratos, L. Edmonds, and M. Lim. 2007. Recurrence rates of video-assisted thoracoscopic versus open surgery in the prevention of recurrent pneumothoraces: A systematic review of randomised and non-randomised trials. Lancet 370(9584):329–335.

Begg, C., M. Cho, S. Eastwood, R. Horton, D. Moher, I. Olkin, R. Pitkin, D. Rennie, K. F. Schulz, D. Simel, and D. F. Stroup. 1996. Improving the quality of reporting of randomized controlled trials: The CONSORT statement. JAMA 276(8):637–639.

Brown, P., K. Brunnhuber, K. Chalkidou, I. Chalmers, M. Clarke, M. Fenton, C. Forbes, J. Glanville, N. J. Hicks, J. Moody, S. Twaddle, H. Timimi, and P. Young. 2006. How to formulate research recommendations. BMJ 333(7572):804–806.

Callaham, M. L., and J. Tercier. 2007. The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Medicine 4(1):e40.

Chimonas, S., Z. Frosch, and D. J. Rothman. 2011. From disclosure to transparency: The use of company payment data. Archives of Internal Medicine.171(1):81–86.

Clarke, L., M. Clarke, and T. Clarke. 2007. How useful are Cochrane reviews in identifying health needs? Journal of Health Services & Research Policy 12(2):101–103.

CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: York Publishing Services, Ltd.

CRD. 2010a. Database of Abstracts of Reviews of Effects (DARE). http://www.crd.york.ac.uk/crdweb/Home.aspx?DB=DARE (accessed December 27, 2010).

CRD. 2010b. Register of ongoing systematic reviews. http://www.york.ac.uk/inst/crd/projects/register.htm (accessed June 17, 2010).

Delaney, A., S. Bagshaw, A. Ferland, B. Manns, K. Laupland, and C. Doig. 2005. A systematic evaluation of the quality of meta-analyses in the critical care literature. Critical Care 9(5):R575–R582.

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Delaney, A., S. M. Bagshaw, A. Ferland, K. Laupland, B. Manns, and C. Doig. 2007. The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: An independent appraisal. Critical Care Medicine 35(2): 589–594.

Devereaux, P. J., P. Choi, S. El-Dika, M. Bhandari, V. M. Montori, H. J. Schünemann, A. X. Garg, J. W. Busse, D. Heels-Ansdell, W. A. Ghali, B. J. Manns, and G. H. Guyatt. 2004. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. Journal of Clinical Epidemiology 57(12):1232–1236.

Docherty, M., and R. Smith. 1999. The case for structuring the discussion of scientific papers. BMJ 318(7193):1224–1225.

EQUATOR Network. 2010. Welcome to the EQUATOR Network website: The resource centre for good reporting of health research studies. http://www.equator-network.org/home/ (accessed June 29, 2010).

Froom, P., and J. Froom. 1993. Deficiencies in structured medical abstracts. Journal of Clinical Epidemiology 46(7):591–594.

Glenton, C. 2002. Developing patient-centered information for back pain patients. Health Expectations 5(4):319–329.

Glenton, C. 2010. Presenting the results of Cochrane systematic reviews to a consumer audience: A qualitative study. Medical Decision Making 30(5):566–577.

Glenton, C., E. Nilsen, and B. Carlsen. 2006a. Lay perceptions of evidence-based information: A qualitative evaluation of a website for back pain sufferers. BMC Health Services Research 6(1):34.

Glenton, C., V. Underland, M. Kho, V. Pennick, and A. D. Oxman. 2006b. Summaries of findings, descriptions of interventions and information about adverse effects would make reviews more informative. Journal of Clinical Epidemiology 59(8):770–778.

Golder, S., Y. Loke, and H. M. McIntosh. 2008. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Journal of Clinical Epidemiology 61(5):440–448.

Goodman, S. N., J. Berlin, S. W. Fletcher, and R. H. Fletcher. 1994. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Annals of Internal Medicine 121(1):11–21.

Hartley, J. 2000. Clarifying the abstracts of systematic literature reviews. Bulletin of the Medical Library Association 88(4):332–337.

Hartley, J., M. Sydes, and A. Blurton. 1996. Obtaining information accurately and quickly: Are structured abstracts more efficient? Journal of Information Science 22(5):349–356.

Haynes, R. B., C. D. Mulrow, E. J. Huth, D. G. Altman, and M. J. Gardner. 1990. More informative abstracts revisited. Annals of Internal Medicine 113(1):69–76.

Helfand, M., and H. Balshem. 2010. AHRQ Series Paper 2: Principles for developing guidance: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63(5):484–490.

Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons.

ICMJE (International Committee of Medical Journal Editors). 2009. Obligation to register clinical trials. http://www.icmje.org/publishing_10register.html (accessed September 9, 2010).

ICMJE. 2010. Uniform requirements for manuscripts submitted to biomedical journals: Ethical considerations in the conduct and reporting of research: Peer review. http://www.icmje.org/ethical_3peer.html (accessed July 12, 2010).

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Ioannidis, J. P., J. W. Evans, P. C. Gøtzsche, R. T. O’Neill, D. Altman, K. Schulz, and D. Moher. 2004. Better reporting of harms in randomized trials: An extension of the CONSORT Statement. Annals of Internal Medicine 141(10):781–788.

IOM (Institute of Medicine). 2008. Knowing what works in health care: A roadmap for the nation. Edited by J. Eden, B. Wheatley, B. McNeil, and H. Sox. Washington, DC: The National Academies Press.

IOM. 2009. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press.

Jefferson, T., P. Alderson, E. Wager, and F. Davidoff. 2002. Effects of editorial peer review: A systematic review. JAMA 287(21):2784–2786.

Lavis, J., H. Davies, A. D. Oxman, J. L. Denis, K. Golden-Biddle, and E. Ferlie. 2005. Towards systematic reviews that inform health care management and policy-making. Journal of Health Services & Research Policy 10(Suppl 1):35–48.

Lexchin, J., L. A. Bero, B. Djubegovic, and O. Clark. 2003. Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. BMJ 326(7400):1167–1170.

Liberati, A., D. G. Altman, J. Tetzlaff, C. Mulrow, P. Gotzsche, J. P. Ioannidis, M. Clarke, P. J. Devereaux, J. Kleijnen, and D. Moher. 2009. The PRISMA Statement for reporting systematic reviews and meta-analysis of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine 151(4):W11–W30.

Lipkus, I. M. 2007. Numeric, verbal, and visual formats of conveying health risks: Suggested best practices and future recommendations. Medical Decision Making 27(5):696–713.

McAlister, F. A., H. D. Clark, C. van Walraven, S. E. Straus, F. Lawson, D. Moher, and C. D. Mulrow. 1999. The medical review article revisited: Has the science improved? Annals of Internal Medicine 131(12):947–951.

McPartland, J. M. 2009. Obesity, the endocannabinoid system, and bias arising from pharmaceutical sponsorship. PLoS One 4(3):e5092.

Moher, D., D. J. Cook, S. Eastwood, I. Olkin, D. Rennie, and D. F. Stroup. 1999. Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Lancet 354(9193):1896–1900.

Moher, D., A. Jones, and L. Lepage. 2001a. Use of the CONSORT statement and quality of reports of randomized trials: A comparative before-and-after evaluation. JAMA 285:1992–1995.

Moher, D., K. F. Schulz, D. Altman, and the CONSORT Group. 2001b. The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 285(15):1987–1991.

Moher, D., J. Tetzlaff, A. C. Tricco, M. Sampson, and D. G. Altman. 2007. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 4(3):447–455.

Moher, D., A. Liberati, J. Tetzlaff, and D. G. Altman. 2009. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine 6(7):1–6.

Mrkobrada, M., H. Thiessen-Philbrook, R. B. Haynes, A. V. Iansavichus, F. Rehman, and A. X. Garg. 2008. Need for quality improvement in renal systematic reviews. Clinical Journal of the American Society of Nephrology 3(4):1102–1114.

Mulrow, C. D. 1987. The medical review article: State of the science. Annals of Internal Medicine 106(3):485–488.

Mulrow, C. D., S. B. Thacker, and J. A. Pugh. 1988. A proposal for more informative abstracts of review articles. Annals of Internal Medicine 108(4):613–615.

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Odierna, D. H., and L. A. Bero. 2009. Systematic reviews reveal unrepresentative evidence for the development of drug formularies for poor and nonwhite populations. Journal of Clinical Epidemiology 62(12):1268–1278.

Oxman, A. D., H. Schunemann, and A. Fretheim. 2006. Improving the use of research evidence in guideline development: Synthesis and presentation of evidence. Health Research Policy and Systems 4:20.

Pocock, S., M. D. Hughes, and R. J. Lee. 1987. Statistical problems in the reporting of clinical trials. A survey of three medical journals. New England Journal of Medicine 317(7):426–432.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). 2010. PRISMA endorsers. http://www.prisma-statement.org/endorsers.htm (accessed September 13, 2010).

Roberts, D., and S. Dalziel. 2006. Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth. In Cochrane Database of Systematic Reviews. http://www.cochrane.org/cochrane-reviews/sample-review#PLS (accessed October 6, 2010).

Roundtree, A. K., M. A. Kallen, M. A. Lopez-Olivo, B. Kimmel, B. Skidmore, Z. Ortiz, V. Cox, and M. E. Suarez-Almazor. 2008. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: A systematic review. Journal of Clinical Epidemiology 62(2):128–137

Sacks, H. S., D. Reitman, V. A. Ancona-Berk, and T. C. Chalmers. 1987. Meta–analysis of randomized controlled trials. New England Journal of Medicine 316(8):450–455.

Santesso, N., L. Maxwell, P. S. Tugwell, G. A. Wells, A. M. Connor, M. Judd, and R. Buchbinder. 2006. Knowledge transfer to clinicians and consumers by the Cochrane Musculoskeletal Group. The Journal of Rheumatology 33(11):2312–2318.

Schroter, S., N. Black, S. Evans, J. Carpenter, F. Godlee, and R. Smith. 2004. Effects of training on quality of peer review: Randomised controlled trial. BMJ 328(7441):657–658.

Schroter, S., L. Tite, A. Hutchings, and N. Black. 2006. Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA 295(3):314–317.

Schünemann, H. J., E. Ståhl, P. Austin, E. Akl, D. Armstrong, and G. H. Guyatt. 2004. A comparison of narrative and table formats for presenting hypothetical health states to patients with gastrointestinal or pulmonary disease. Medical Decision Making 24(1):53–60.

Schwartz, L. M., S. Woloshin, and H. G. Welch. 2009. Using a drug facts box to communicate drug benefits and harms. Annals of Internal Medicine 150(8):516–527.

Shea, B., L. M. Bouter, J. Peterson, M. Boers, N. Andersson, Z. Ortiz, T. Ramsay, A. Bai, V. K. Shukla, and J. M. Grimshaw. 2007. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2(12):e1350.

Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ Series Paper 1: Comparing medical interventions: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63(5):481–483.

Stroup, D. F., J. Berlin, S. Morton, I. Olkin, G. D. Williamson, D. Rennie, D. Moher, B. J. Becker, T. A. Sipe, S. B. Thacker, and the Meta-analysis of Observational Studies in Epidemiology Group. 2000. Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA 283(15):2008–2012.

Trevena, L. J., A. Barratt, P. Butow, and P. Caldwell. 2006. A systematic review on communicating with patients about evidence. Journal of Evaluation in Clinical Practice 12(1):13–23.

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

von Elm, E., D. G. Altman, M. Egger, S. J. Pocock, P. C. Gotzsche, J. P. Vandenbroucke, and the Strobe Initiative. 2007. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. Annals of Internal Medicine 147(8):573–577.

Weller, A. C. 2002. Editorial peer review: Its strengths and weaknesses. Medford, NJ: American Society for Information Science and Technology. http://books.google.com/books?hl=en&lr=&id=lH3DVnkbTFsC&oi=fnd&pg=PR12&dq=ICMJE+PEER+REVIEW&ots=rpXA8rZOna&sig=nCltaFOsxGxyrVSoGCeWWhRNLIE#v=onepage&q=ICMJE%20PEER%20REVIEW&f=false (accessed July 12, 2010).

Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2010. AHRQ Series Paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63(5):491–501.

Wills, C. E., and M. Holmes-Rovner. 2003. Patient comprehension of information for shared treatment decision making: State of the art and future directions. Patient Education and Counseling 50(3):285–290.

Yank, V., D. Rennie, and L. A. Bero. 2007. Financial ties and concordance between results and conclusions in meta-analyses: Retrospective cohort study. BMJ 335(7631):1202–1205.

Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 195
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 196
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 197
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 198
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 199
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 200
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 201
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 202
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 203
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 204
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 205
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 206
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 207
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 208
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 209
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 210
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 211
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 212
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 213
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 214
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 215
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 216
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 217
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 218
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 219
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 220
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 221
Suggested Citation:"5 Standards for Reporting Systematic Reviews." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 222
Next: 6 Improving the Quality of Systematic Reviews: Discussion, Conclusions, and Recommendations »
Finding What Works in Health Care: Standards for Systematic Reviews Get This Book
×
Buy Paperback | $61.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Healthcare decision makers in search of reliable information that compares health interventions increasingly turn to systematic reviews for the best summary of the evidence. Systematic reviews identify, select, assess, and synthesize the findings of similar but separate studies, and can help clarify what is known and not known about the potential benefits and harms of drugs, devices, and other healthcare services. Systematic reviews can be helpful for clinicians who want to integrate research findings into their daily practices, for patients to make well-informed choices about their own care, for professional medical societies and other organizations that develop clinical practice guidelines.

Too often systematic reviews are of uncertain or poor quality. There are no universally accepted standards for developing systematic reviews leading to variability in how conflicts of interest and biases are handled, how evidence is appraised, and the overall scientific rigor of the process.

In Finding What Works in Health Care the Institute of Medicine (IOM) recommends 21 standards for developing high-quality systematic reviews of comparative effectiveness research. The standards address the entire systematic review process from the initial steps of formulating the topic and building the review team to producing a detailed final report that synthesizes what the evidence shows and where knowledge gaps remain.

Finding What Works in Health Care also proposes a framework for improving the quality of the science underpinning systematic reviews. This book will serve as a vital resource for both sponsors and producers of systematic reviews of comparative effectiveness research.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!