TABLE G-1 Comparison of Chapter 5 Guidance on Conducting Systematic Reviews (SRs) of Comparative Effectiveness Research
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
5.1 Prepare the final report using a structured format |
Use a structured format that adheres to the Evidence-based Practice Center (EPC) style guide. Report must meet Section 508 requirements for users with disabilities. |
Quality of Reporting of Metaanalyses (QUORUM)/Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) are useful guides for all authors of systematic review reports. (NOTE: The next edition of the guidance will recommend adhering to PRISMA.) Commissioning bodies and journals usually have specific requirements regarding presentation and layout of the review. |
Cochrane reviews all have the same format, which is facilitated by RevMan. |
Cochrane has endorsed PRISMA, and it will be incorporated into the next version of the Cochrane handbook. |
|||
5.1.1 Include a report title |
Required. |
Required. |
Required. |
5.1.2 Include an abstract |
Required. |
Include a structured abstract for reviews published as journal articles. |
Required. All full reviews must include an abstract of 400 words or fewer. The abstract should primarily target healthcare decision makers. |
5.1.3 Include an executive summary |
Required. An executive summary is published separately as well as with the full-length report. |
Include an executive summary for reviews published as fulllength reports. |
Not mentioned. |
5.1.4 Include a summary written for the lay public |
Required. Developed by the Eisenberg Center. |
Not mentioned. |
Required. Plain-language summaries provide findings in a straightforward style that can be understood by consumers. |
5.1.5 Include an introduction (rationale and objectives) |
Required in both full report as well as part of the executive summary. |
Required. Include a background/introduction. |
Required. |
5.1.6 Include a methods section. Describe the following: |
|
||
• Research protocol |
Same elements in protocol are required in methods section. |
Description of the protocol is not mentioned, but the next edition will recommend that reports indicate that a protocol was written and followed, and should report the protocol registration number. |
Review authors are encouraged to cite their protocol. |
• Eligibility criteria (criteria for including and excluding studies in the systematic review) |
Required in protocol and methods section. |
Required. |
Required. |
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
• Analytic framework and key questions key questions |
Required in protocol and methods section of the full report. |
Not mentioned. |
Not mentioned. |
• Databases and other information sources used to identify relevant studies |
Required in protocol and methods section. |
Required. The write-up of the search should include information about the databases and interfaces searched. |
Required. |
• Search strategy |
Required. Include a description of the search methods. Full-search strategy required in appendix. This description should be detailed enough to allow replication of search. |
Required. The search process should be documented in full, including information about the databases and interfaces searched (including the dates covered), full detailed search strategies (including any justifications for date or language restrictions), and the number of records retrieved or details provided on where the strategy can be obtained. An appendix documenting the search process should be included. |
Required. List all databases searched. Note the dates of the last search for each database and the period searched. Note any language or publication status restriction. List grey-literature sources. List individuals or organizations contacted. List any journals and conference proceedings specifically handsearched. List any other sources searched. |
Study selection process |
Required in protocol and methods section. |
Required. |
State the method used to apply the selection criteria. |
• Data extraction process |
Required in protocol and methods section. |
Required. |
Describe the methods for data collection. |
• Methods for handling missing information |
Required in protocol and methods. |
Required. |
Describe the strategies for dealing with missing data. |
• Information to be extracted from included studies |
Required in protocol and methods section. |
Required. |
Not mentioned. |
• Methods to appraise the quality of individual studies |
Required. Protocol and methods section should describe methods to assess risk of bias. |
Required. |
Describe the methods used to assess risk of bias. |
• Summary measures (e.g., risk ratio, difference in means) |
Required in protocol and methods section. |
Required. |
The effect measures of choice should be stated. |
• Rationale for pooling (or not pooling) of included studies |
Required in protocol and methods section. |
Not mentioned. |
Approach to determining whether a meta-analysis is considered appropriate should be included. |
• Methods of synthesizing the evidence (qualitative and meta-analysis) |
Required in protocol and methods section. Describing methods for grading of strength of evidence in general and of each domain is recommended. |
Required. |
The choice of meta-analysis method should be stated, including whether a fixed-effect or a random-effects model is used. Approaches to addressing clinical heterogeneity should be described. Method for identifying statistical heterogeneity should be stated (e.g., visually, using I2, using a chi-squared test). |
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
• Additional analyses, if done, indicating which were prespecified |
Required in protocol and methods section. |
Any secondary analyses (sensitivity analyses, etc.). |
All planned subgroup analyses should be listed (or independent variables for meta-regression). Any other methods for investigating heterogeneity of effects should be described. |
5.1.7 Include a results section. Organize the presentation of results around key questions. Describe the following (repeat for each key question): |
Organize presentation of results in logical format. This is typically done around key questions. |
The results of all analyses should be considered as a whole, and overall coherence discussed. |
The results section should directly address the objectives of the review. |
• Study selection process |
Required. Flow chart is required documenting excluded studies. |
Required. Describe the details of included and excluded studies. |
The results sections should start with a summary of the results of the search (e.g., how many references were retrieved by the electronic searches, and how many were considered as potentially eligible after screening?). It is essential that the number of included studies is clearly stated. |
• List of excluded studies and reasons for their exclusion |
Excluded studies are included in references/appendix. |
Required. |
Studies that may appear to meet the eligibility criteria, but which were excluded, should be listed and the reason for exclusion should be given. |
• Appraisal of individual studies’ quality |
Required. |
Required. |
Required. A risk of bias table is strongly recommended. |
• Qualitative synthesis |
Required. Describe the findings of the review. |
Required. Both quantitative and narrative synthesis should begin by constructing a clear descriptive summary of the included studies. An indication of study quality or risk of bias may also be given. |
Not required. However, the final report should include a characteristics of included studies table and Grading of Recommendations Assessment, Development and Evaluation (GRADE) evidence tables. It should also summarize the general risk of bias in results of the included studies, its variability across studies, and any important flaws in individual studies. |
The study characteristics of eligible studies are usually included in both a text summary and a summary table, and sometimes in an evidence map as well. |
|||
Where possible, results of individual studies should be presented graphically, most commonly using a forest plot that illustrates the effect estimates from individual studies. |
|||
Highlight where evidence indicates that benefits, harms, and trade-offs are different for distinct patient groups. |
|||
The justification for grade and domains are required, usually provided in a grading table, sometimes in appendix. |
|||
Consider how the relative effects may translate into different absolute effects for people with differing underlying prognoses. |
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
• Meta-analysis of results, if performed (explain rationale for doing one) |
Required, if appropriate. Describe the findings of the review. |
Required. Consistency across studies should be considered. |
Required. A summary of findings table may be included to present the main findings of a review in a tabular format. |
• Additional analyses, if done, indicating which were prespecified |
Required as appropriate. |
Required. Include any secondary analyses. |
Not mentioned. |
• Tables and figures |
Include tables summarizing the studies and quantitative syntheses. |
Where possible, results should be shown graphically. The most commonly used graphic is the forest plot. |
Required. Tables that may be included in a review: a characteristics of included studies table, a risk of bias table, a characteristics of excluded studies table, a characteristics of studies awaiting classification table, a characteristics of ongoing studies table, and a summary of findings table. Figures that may be included in a review: forest plot, funnel plot, risk of bias graph, risk of bias summary, and other figures. |
Synthesis should usually include tabulated details about study type, interventions, number of participants, a summary of participant characteristics, outcomes, and outcome measures. |
|||
5.1.8 Include a discussion section. Include the following: |
|
|
|
• Summary of the evidence |
Required, though usually in the conclusions section. |
Suggests a statement of principal findings. |
Required. |
• Strengths and limitations of the systematic review |
Recommended. Describe strengths and weaknesses of systematic review and of studies. |
Required. Describe the strengths and weaknesses of the review. Appraise the methodological quality of the review, and the relation to other reviews. |
Describe the quality of evidence, potential biases in the review process, and agreements/disagreements with other studies or reviews. |
• Conclusions for each key question |
Required. Present the benefits and harms in a manner that helps decision makers. Express benefits in absolute terms, rather than relative terms. |
Should include practical implications for clinicians and policy makers. |
Review authors should not make recommendations for clinical practice. May highlight different actions that might be consistent with particular patterns of values and preferences. |
• Gaps in evidence |
Description of gaps in evidence is required as a separate section (does not necessarily need to be in discussion section). |
Gaps in evidence should be highlighted. |
Describe the completeness and applicability of evidence to the review question. |
• Future research needs |
Required. Some reports will also require this as a more fully prioritized and fleshed-out separate paper. |
The report should describe any unanswered questions and implications for further research. |
Describe the implications for research. |
5.1.9 Include a section describing funding sources and COI |
This is done automatically in the editing process. |
Required. |
Required. |
5.2 Peer review the draft report |
Identify peer reviewers to ensure independent, unconflicted input from persons with particular clinical, methodological, and statistical expertise and submit a draft report to these individuals. |
The advisory group should review the draft report for scientific quality and completeness. The commissioning body may also organize an independent peer review of the draft report. |
The editorial team of the Cochrane Review Group is ultimately responsible for the decision to publish a Cochrane review on its module. The decision is made after peer review and appropriate revisions by the review authors. |
REFERENCES
Atkins, D., S. Chang, G. Gartlehner, D. I. Buckley, E. P. Whitlock, E. Berliner, and D. Matchar. 2010. Assessing the applicability of studies when comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?productid=603&pageaction=displayproduct (accessed January 19, 2011).
Chou, R., N. Aronson, D. Atkins, A. S. Ismaila, P. Santaguida, D. H. Smith, E. Whitlock, T. J. Wilt, and D. Moher. 2010. AHRQ series paper 4: Assessing harms when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):502–512.
CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: York Publishing Services, Ltd.
Fu, R., G. Gartlehner, M. Grant, T. Shamliyan, A. Sedrakyan, T. J. Wilt, L. Griffith, M. Oremus, P. Raina, A. Ismaila, P. Santaguida, J. Lau, and T. A. Trikalinos. 2010. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=554 (accessed January 19, 2011).
Helfand, M., and H. Balshem. 2010. AHRQ series paper 2: Principles for developing guidance: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):484–490.
Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons
Norris, S., D. Atkins, W. Bruening, S. Fox, E. Johnson, R. Kane, S. C. Morton, M. Oremus, M. Ospina, G. Randhawa, K. Schoelles, P. Shekelle, and M. Viswanathan. 2010. Selecting observational studies for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=454 (accessed January 19, 2011).
Owens, D. K., K. N. Lohr, D. Atkins, J. R. Treadwell, J. T. Reston, E. B. Bass, S. Chang, and M. Helfand. 2010. AHRQ series paper 5: Grading the strength of a body of evidence when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):513–523.
Relevo, R., and H. Balshem. 2011. Finding evidence for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=605 (accessed January 19, 2011).
Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ series paper 1: Comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):481–483.
White, C. M., S. Ip, M. McPheeters, T. S. Carey, R. Chou, K. N. Lohr, K. Robinson, K. McDonald, and E. Whitlock. 2009. Using existing systematic reviews to replace de novo processes in CERs. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effective-healthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=329 (accessed January 19, 2011).
Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2010. AHRQ series paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):491–501.