F
Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 281
F Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence 281

OCR for page 281
TABLE F-1 Comparison of Chapter 4 Guidance on Conducting Systematic Reviews (SRs) of Comparative 282 Effectiveness Research Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration 4.1 Use a prespecified The AHRQ method for The planned approach to Adopts the GRADE system for method to evaluate the evaluating the body of evaluating the body of evaluating the body of evidence. body of evidence evidence is conceptually evidence should be decided similar to the GRADE system at the outset of the review, (see below). depending on the type of question posed and the type of studies that are likely to be available. 4.1.1 For each outcome, Requires the assessment of: Quality assessment is likely to Requires the assessment of: systematically assess the •  Risk of bias. consider the following: •  Risk of bias. following characteristics of •  Consistency. •    ppropriateness of study  A •  Consistency. the body of evidence: •  Precision. design. •  Precision. •  Risk of bias •  Directness. •  Risk of bias. •  Directness. •  Consistency •  Applicability. •  Choice of outcome measure. •  Publication bias.  •  Precision  •    ublication bias (if there  P •  Statistical issues. Reviewers should evaluate the •  Directness is reason to believe that •  Quality of reporting. applicability of a body of evidence •  Reporting bias  relevant empirical findings •  Quality of the intervention. as part of the assessment of have not been published). •  Generalizability. directness. Reviewers should evaluate The importance of each of these the applicability of a body aspects of quality will depend of evidence separately from on the focus and nature of the directness. review.

OCR for page 281
4.1.2 For bodies of evidence The following characteristics The quality assessment should For bodies of evidence that include that include observational should be assessed if they are be guided by the types of study observational research, assess the research, also systematically relevant to a particular SR. designs included in the SR. following characteristics for each assess the following They are applied more often outcome: characteristics for each to evidence from observational •  Dose–response association. outcome: studies than to evidence from •    lausible confounding that  P •    ose–response  D randomized controlled trials. would decrease an observed association •  Dose–response association. effect. •    lausible confounding  P •    lausible confounding that  P •  Strength of association. that would change the would decrease an observed observed effect effect. •    trength of association S •  Strength of association. 4.1.3 For each outcome The quality of evidence Not mentioned. The quality of evidence receives a specified in the protocol, receives a single grade: high, single grade: high, moderate, low, use consistent language moderate, low, or insufficient. or very low. to characterize the level of confidence in the estimates of the effect of an intervention 4.2 Conduct a qualitative All SRs should include a All SRs should include a A narrative synthesis should be synthesis narrative synthesis. Provides narrative synthesis. Provides used where meta-analysis is not guidance (see below). guidance (see below). feasible or not sensible. Provides guidance on some elements (see below). 283 continued

OCR for page 281
TABLE F-1 Continued 284 Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration 4.2.1 Describe the clinical Summarize the available Provide a clear descriptive Review authors should, as a and methodological evidence using PICOTS summary of the included minimum, include the following characteristics of the domains in a summary table: studies, with details about in the characteristics of included studies, including •    haracteristics of enrolled  C study type, interventions, included studies table: methods, their size, inclusion or populations. Where possible, number of participants, a participants, intervention, and exclusion of important describe the proportion with summary of participant outcomes. Where appropriate, subgroups, timeliness, and important characteristics characteristics, outcomes, and use an extra field to provide other relevant factors (e.g., % over age 65) rather outcome measures. information about the funding of than the range. each study. •    eneral characteristics of the  G intervention. •  Comparators used. •    utcomes most frequently  O reported. •  Range of follow-up. 4.2.2 Describe the Assess and document decisions Recording the strengths and Whether the synthesis is strengths and limitations on “quality” and applicability weaknesses of included studies quantitative or qualitative, of individual studies and of individual studies, including provides an indication of methodological limitations are patterns across studies criteria for overall quality whether the results have been described in detail through assessment. unduly influenced by aspects presentation of risk of bias tables, of study design or conduct. through written summaries of risk of bias assessments, and by footnotes in summary of findings tables.

OCR for page 281
4.2.3 Describe, in plain EPCs describe criteria for Assess the risk of bias in Assess risk of bias in all studies terms, how flaws in the assessing risk of bias of included studies caused by in a review irrespective of the design or execution of the individual studies, which, by inadequacies in study design, anticipated variability in either study (or groups of studies) definition, describes how the conduct, or analysis that may the results or the validity of the could bias the results, study design and execution have led to the treatment effect included studies. explaining the reasoning may bias the results. being over- or underestimated. behind these judgments 4.2.4 Describe the EPCs should explore Provide an analysis of the Organizing the studies into relationships between heterogeneity of findings. They relationships within and groupings or clusters is the characteristics of the should prespecify subanalyses between studies. encouraged (e.g., by intervention individual studies and or characteristics by which they type, population groups, setting, their reported findings and analyze heterogeneity, whether etc.). patterns across studies for methodologic heterogeneity or clinical heterogeneity. 4.2.5 Discuss the relevance EPCs should describe the Not mentioned. Not mentioned. of individual studies to the limitations of applicability of populations, comparisons, a body of evidence within the cointerventions, settings, PICOS structure. and outcomes or measures of interest 4.3 Decide if, in addition Meta-analysis is appropriate if The approach to quantitative Describe why a meta-analysis is to a qualitative analysis, combining studies will give a synthesis should be decided at appropriate. The choice of meta- the systemic review will meaningful answer to a well- the outset of the review. analysis method should be stated, include a quantitative formulated research question. including whether a fixed-effect or analysis (meta-analysis) a random-effects model is used. 285 continued

OCR for page 281
TABLE F-1 Continued 286 Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration Meta-analysis is not always possible or sensible. The type of synthesis depends on the type of question posed and the type of studies that are available. Initial descriptive phase of synthesis will be helpful in confirming that studies are similar and reliable enough to synthesize and that it is appropriate to pool results. 4.3.1 Explain why a pooled Authors should explain the Not mentioned. Not mentioned. estimate might be useful to reason a combined estimate decision makers might be useful to decision makers. 4.4 If conducting a Provides guidance on Provides guidance on Provides guidance on conducting a meta-analysis, then do conducting a meta-analysis conducting a meta-analysis meta-analysis (see below). the following: (see below). (see below). 4.4.1 Use expert Review team must include The review team should ideally Review teams must include, methodologists to develop, an individual with statistical include expertise in statistics. or have access to, expertise in execute, and peer review the expertise. A peer reviewer with The team may wish to seek systematic review methodology meta-analyses statistical expertise should be advice from methodological (including statistical expertise). invited as appropriate. experts formally through an advisory group, or informally.

OCR for page 281
4.4.2 Address the Evaluate the amount of Variation in results across It is important to consider to what heterogeneity among heterogeneity for each meta- studies should be investigated extent the results of studies are study effects analysis. Explore statistical informally by visual consistent. A statistical test for heterogeneity using subgroup examination of the forest plot, heterogeneity is available, but a analysis or meta-regression or tested using chi square test or useful statistic for quantifying sensitivity analyses. Q statistic, quantified using the inconsistency is I2. It is clearly of I squared statistic. If statistical interest to determine the causes heterogeneity is observed, of heterogeneity among results of then the possible reasons for studies. However, most Cochrane differences should be explored. reviews do not have enough The influence of patient-level studies to allow the reliable characteristics or issues related investigation of the reasons for to equity can also be explored heterogeneity. through subgroup analyses, meta-regression, or other modeling approaches. 4.4.3 Accompany all Appropriate measures of Results should be expressed as Results should always be estimates with measures of variance should be included point estimates together with accompanied by a measure statistical uncertainty with point estimates from associated confidence intervals of uncertainty, such as a 95% meta-analyses. and exact p-values. confidence interval. 4.4.4 Assess the sensitivity Sensitivity analysis should be Sensitivity analyses should be Sensitivity analyses should be of conclusions to changes in conducted to investigate the used to explore the robustness used to examine whether overall the protocol, assumptions, robustness of the results. of the main meta-analysis by findings are robust to potentially and study selection repeating the analyses after influential decisions. (sensitivity analysis) having made some changes to the data or methods. NOTES: Some information on AHRQ-, CRD-, and Cochrane-recommended methods was provided via personal communication with Stephanie Chang, EPC Program Task Order Officer, AHRQ (October 5, 2010); Lesley Stewart, Director, CRD (October 14, 2010); and Julian Higgins, Senior Statistician, MRC Biostatistics Unit, Institute of Public Health, University of Cambridge (October 4, 2010). The 287 order of the standards does not indicate the sequence in which they are carried out.

OCR for page 281
288 FINDING WHAT WORKS IN HEALTH CARE REFERENCES Atkins, D., S. Chang, G. Gartlehner, D. I. Buckley, E. P. Whitlock, E. Berliner, and D. Matchar. 2010. Assessing the applicability of studies when comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?productid=603& pageaction=displayproduct (accessed January 19, 2011). Chou, R., N. Aronson, D. Atkins, A. S. Ismaila, P. Santaguida, D. H. Smith, E. Whitlock, T. J. Wilt, and D. Moher. 2010. AHRQ series paper 4: Assessing harms when com- paring medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):502–512. CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: York Publishing Services, Ltd. Fu, R., G. Gartlehner, M. Grant, T. Shamliyan, A. Sedrakyan, T. J. Wilt, L. Griffith, M. Oremus, P. Raina, A. Ismaila, P. Santaguida, J. Lau, and T. A. Trikalinos. 2010. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. In Methods guide for compara- tive effectiveness reviews, edited by Agency for Healthcare Research and Qual- ity. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides- reviews-and-reports/?pageaction=displayProduct&productID=554 (accessed January 19, 2011). Helfand, M., and H. Balshem. 2010. AHRQ series paper 2: Principles for develop - ing guidance: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):484–490. Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons. Norris, S., D. Atkins, W. Bruening, S. Fox, E. Johnson, R. Kane, S. C. Morton, M. Oremus, M. Ospina, G. Randhawa, K. Schoelles, P. Shekelle, and M. Viswanathan. 2010. Selecting observational studies for comparing medical inter- ventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct &productID=454 (accessed January 19, 2011). Owens, D. K., K. N. Lohr, D. Atkins, J. R. Treadwell, J. T. Reston, E. B. Bass, S. Chang, and M. Helfand. 2010. AHRQ series paper 5: Grading the strength of a body of evidence when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):513–523. Relevo, R., and H. Balshem. 2011. Finding evidence for comparing medical interven - tions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct &productID=605 (accessed January 19, 2011). Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ series paper 1: Comparing medical interventions: AHRQ and the Effective Health Care Pro - gram. Journal of Clinical Epidemiology 63(5):481–483. White, C. M., S. Ip, M. McPheeters, T. S. Carey, R. Chou, K. N. Lohr, K. Robinson, K. McDonald, and E. Whitlock. 2009. Using existing systematic reviews to replace de novo processes in CERs. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effective- healthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?page action=displayProduct&productID=329 (accessed January 19, 2011).

OCR for page 281
289 APPENDIX F Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2010. AHRQ series paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):491–501.

OCR for page 281