National Academies Press: OpenBook

Finding What Works in Health Care: Standards for Systematic Reviews (2011)

Chapter: Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence

« Previous: Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

F
Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

TABLE F-1 Comparison of Chapter 4 Guidance on Conducting Systematic Reviews (SRs) of Comparative Effectiveness Research

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

4.1 Use a prespecified method to evaluate the body of evidence

The AHRQ method for evaluating the body of evidence is conceptually similar to the GRADE system (see below).

The planned approach to evaluating the body of evidence should be decided at the outset of the review, depending on the type of question posed and the type of studies that are likely to be available.

Adopts the GRADE system for evaluating the body of evidence.

4.1.1 For each outcome, systematically assess the following characteristics of the body of evidence:

  • Risk of bias

  • Consistency

  • Precision

  • Directness

  • Reporting bias

Requires the assessment of:

  • Risk of bias.

  • Consistency.

  • Precision.

  • Directness.

  • Applicability.

  • Publication bias (if there is reason to believe that relevant empirical findings have not been published).

Quality assessment is likely to consider the following:

  • Appropriateness of study design.

  • Risk of bias.

  • Choice of outcome measure.

  • Statistical issues.

  • Quality of reporting.

  • Quality of the intervention.

  • Generalizability.

Requires the assessment of:

  • Risk of bias.

  • Consistency.

  • Precision.

  • Directness.

  • Publication bias.

Reviewers should evaluate the applicability of a body of evidence as part of the assessment of directness.

Reviewers should evaluate the applicability of a body of evidence separately from directness.

The importance of each of these aspects of quality will depend on the focus and nature of the review.

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

4.1.2 For bodies of evidence that include observational research, also systematically assess the following characteristics for each outcome:

  • Dose–response association

  • Plausible confounding that would change the observed effect

  • Strength of association

The following characteristics should be assessed if they are relevant to a particular SR. They are applied more often to evidence from observational studies than to evidence from randomized controlled trials.

  • Dose–response association.

  • Plausible confounding that would decrease an observed effect.

  • Strength of association.

The quality assessment should be guided by the types of study designs included in the SR.

For bodies of evidence that include observational research, assess the following characteristics for each outcome:

  • Dose–response association.

  • Plausible confounding that would decrease an observed effect.

  • Strength of association.

4.1.3 For each outcome specified in the protocol, use consistent language to characterize the level of confidence in the estimates of the effect of an intervention

The quality of evidence receives a single grade: high, moderate, low, or insufficient.

Not mentioned.

The quality of evidence receives a single grade: high, moderate, low, or very low.

4.2 Conduct a qualitative synthesis

All SRs should include a narrative synthesis. Provides guidance (see below).

All SRs should include a narrative synthesis. Provides guidance (see below).

A narrative synthesis should be used where meta-analysis is not feasible or not sensible. Provides guidance on some elements (see below).

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

4.2.1 Describe the clinical and methodological characteristics of the included studies, including their size, inclusion or exclusion of important subgroups, timeliness, and other relevant factors

Summarize the available evidence using PICOTS domains in a summary table:

  • Characteristics of enrolled populations. Where possible, describe the proportion with important characteristics (e.g., % over age 65) rather than the range.

  • General characteristics of the intervention.

  • Comparators used.

  • Outcomes most frequently reported.

  • Range of follow-up.

Provide a clear descriptive summary of the included studies, with details about study type, interventions, number of participants, a summary of participant characteristics, outcomes, and outcome measures.

Review authors should, as a minimum, include the following in the characteristics of included studies table: methods, participants, intervention, and outcomes. Where appropriate, use an extra field to provide information about the funding of each study.

4.2.2 Describe the strengths and limitations of individual studies and patterns across studies

Assess and document decisions on “quality” and applicability of individual studies, including criteria for overall quality assessment.

Recording the strengths and weaknesses of included studies provides an indication of whether the results have been unduly influenced by aspects of study design or conduct.

Whether the synthesis is quantitative or qualitative, methodological limitations are described in detail through presentation of risk of bias tables, through written summaries of risk of bias assessments, and by footnotes in summary of findings tables.

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

4.2.3 Describe, in plain terms, how flaws in the design or execution of the study (or groups of studies) could bias the results, explaining the reasoning behind these judgments

EPCs describe criteria for assessing risk of bias of individual studies, which, by definition, describes how the study design and execution may bias the results.

Assess the risk of bias in included studies caused by inadequacies in study design, conduct, or analysis that may have led to the treatment effect being over- or underestimated.

Assess risk of bias in all studies in a review irrespective of the anticipated variability in either the results or the validity of the included studies.

4.2.4 Describe the relationships between the characteristics of the individual studies and their reported findings and patterns across studies

EPCs should explore heterogeneity of findings. They should prespecify subanalyses or characteristics by which they analyze heterogeneity, whether for methodologic heterogeneity or clinical heterogeneity.

Provide an analysis of the relationships within and between studies.

Organizing the studies into groupings or clusters is encouraged (e.g., by intervention type, population groups, setting, etc.).

4.2.5 Discuss the relevance of individual studies to the populations, comparisons, cointerventions, settings, and outcomes or measures of interest

EPCs should describe the limitations of applicability of a body of evidence within the PICOS structure.

Not mentioned.

Not mentioned.

4.3 Decide if, in addition to a qualitative analysis, the systemic review will include a quantitative analysis (meta-analysis)

Meta-analysis is appropriate if combining studies will give a meaningful answer to a well-formulated research question.

The approach to quantitative synthesis should be decided at the outset of the review.

Describe why a meta-analysis is appropriate. The choice of meta-analysis method should be stated, including whether a fixed-effect or a random-effects model is used.

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

 

 

Meta-analysis is not always possible or sensible. The type of synthesis depends on the type of question posed and the type of studies that are available. Initial descriptive phase of synthesis will be helpful in confirming that studies are similar and reliable enough to synthesize and that it is appropriate to pool results.

4.3.1 Explain why a pooled estimate might be useful to decision makers

Authors should explain the reason a combined estimate might be useful to decision makers.

Not mentioned.

Not mentioned.

4.4 If conducting a meta-analysis, then do the following:

Provides guidance on conducting a meta-analysis (see below).

Provides guidance on conducting a meta-analysis (see below).

Provides guidance on conducting a meta-analysis (see below).

4.4.1 Use expert methodologists to develop, execute, and peer review the meta-analyses

Review team must include an individual with statistical expertise. A peer reviewer with statistical expertise should be invited as appropriate.

The review team should ideally include expertise in statistics. The team may wish to seek advice from methodological experts formally through an advisory group, or informally.

Review teams must include, or have access to, expertise in systematic review methodology (including statistical expertise).

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

4.4.2 Address the heterogeneity among study effects

Evaluate the amount of heterogeneity for each meta-analysis. Explore statistical heterogeneity using subgroup analysis or meta-regression or sensitivity analyses.

Variation in results across studies should be investigated informally by visual examination of the forest plot, tested using chi square test or Q statistic, quantified using the I squared statistic. If statistical heterogeneity is observed, then the possible reasons for differences should be explored. The influence of patient-level characteristics or issues related to equity can also be explored through subgroup analyses, meta-regression, or other modeling approaches.

It is important to consider to what extent the results of studies are consistent. A statistical test for heterogeneity is available, but a useful statistic for quantifying inconsistency is I2. It is clearly of interest to determine the causes of heterogeneity among results of studies. However, most Cochrane reviews do not have enough studies to allow the reliable investigation of the reasons for heterogeneity.

4.4.3 Accompany all estimates with measures of statistical uncertainty

Appropriate measures of variance should be included with point estimates from meta-analyses.

Results should be expressed as point estimates together with associated confidence intervals and exact p-values.

Results should always be accompanied by a measure of uncertainty, such as a 95% confidence interval.

4.4.4 Assess the sensitivity of conclusions to changes in the protocol, assumptions, and study selection (sensitivity analysis)

Sensitivity analysis should be conducted to investigate the robustness of the results.

Sensitivity analyses should be used to explore the robustness of the main meta-analysis by repeating the analyses after having made some changes to the data or methods.

Sensitivity analyses should be used to examine whether overall findings are robust to potentially influential decisions.

NOTES: Some information on AHRQ-, CRD-, and Cochrane-recommended methods was provided via personal communication with Stephanie Chang, EPC Program Task Order Officer, AHRQ (October 5, 2010); Lesley Stewart, Director, CRD (October 14, 2010); and Julian Higgins, Senior Statistician, MRC Biostatistics Unit, Institute of Public Health, University of Cambridge (October 4, 2010). The order of the standards does not indicate the sequence in which they are carried out.

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

REFERENCES

Atkins, D., S. Chang, G. Gartlehner, D. I. Buckley, E. P. Whitlock, E. Berliner, and D. Matchar. 2010. Assessing the applicability of studies when comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?productid=603&pageaction=displayproduct (accessed January 19, 2011).

Chou, R., N. Aronson, D. Atkins, A. S. Ismaila, P. Santaguida, D. H. Smith, E. Whitlock, T. J. Wilt, and D. Moher. 2010. AHRQ series paper 4: Assessing harms when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):502–512.

CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: York Publishing Services, Ltd.

Fu, R., G. Gartlehner, M. Grant, T. Shamliyan, A. Sedrakyan, T. J. Wilt, L. Griffith, M. Oremus, P. Raina, A. Ismaila, P. Santaguida, J. Lau, and T. A. Trikalinos. 2010. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=554 (accessed January 19, 2011).

Helfand, M., and H. Balshem. 2010. AHRQ series paper 2: Principles for developing guidance: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):484–490.

Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons.

Norris, S., D. Atkins, W. Bruening, S. Fox, E. Johnson, R. Kane, S. C. Morton, M. Oremus, M. Ospina, G. Randhawa, K. Schoelles, P. Shekelle, and M. Viswanathan. 2010. Selecting observational studies for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=454 (accessed January 19, 2011).

Owens, D. K., K. N. Lohr, D. Atkins, J. R. Treadwell, J. T. Reston, E. B. Bass, S. Chang, and M. Helfand. 2010. AHRQ series paper 5: Grading the strength of a body of evidence when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):513–523.

Relevo, R., and H. Balshem. 2011. Finding evidence for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=605 (accessed January 19, 2011).

Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ series paper 1: Comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):481–483.

White, C. M., S. Ip, M. McPheeters, T. S. Carey, R. Chou, K. N. Lohr, K. Robinson, K. McDonald, and E. Whitlock. 2009. Using existing systematic reviews to replace de novo processes in CERs. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effective-healthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=329 (accessed January 19, 2011).

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2010. AHRQ series paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):491–501.

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

This page intentionally left blank.

Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 281
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 282
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 283
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 284
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 285
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 286
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 287
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 288
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 289
Suggested Citation:"Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 290
Next: Appendix G: Expert Guidance for Chapter 5: Standards for Reporting Systemic Reviews »
Finding What Works in Health Care: Standards for Systematic Reviews Get This Book
×
Buy Paperback | $61.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Healthcare decision makers in search of reliable information that compares health interventions increasingly turn to systematic reviews for the best summary of the evidence. Systematic reviews identify, select, assess, and synthesize the findings of similar but separate studies, and can help clarify what is known and not known about the potential benefits and harms of drugs, devices, and other healthcare services. Systematic reviews can be helpful for clinicians who want to integrate research findings into their daily practices, for patients to make well-informed choices about their own care, for professional medical societies and other organizations that develop clinical practice guidelines.

Too often systematic reviews are of uncertain or poor quality. There are no universally accepted standards for developing systematic reviews leading to variability in how conflicts of interest and biases are handled, how evidence is appraised, and the overall scientific rigor of the process.

In Finding What Works in Health Care the Institute of Medicine (IOM) recommends 21 standards for developing high-quality systematic reviews of comparative effectiveness research. The standards address the entire systematic review process from the initial steps of formulating the topic and building the review team to producing a detailed final report that synthesizes what the evidence shows and where knowledge gaps remain.

Finding What Works in Health Care also proposes a framework for improving the quality of the science underpinning systematic reviews. This book will serve as a vital resource for both sponsors and producers of systematic reviews of comparative effectiveness research.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!