TABLE E-1 Comparison of Chapter 3 Guidance on Conducting Systematic Reviews (SR) of Comparative Effectiveness Research
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
3.1 Conduct a comprehensive, systematic search for evidence |
Provides guidance on searching for evidence (see below). |
Provides guidance on searching for evidence (see below). |
Provides guidance on searching for evidence (see below). |
3.1.1 Work with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy |
A person with library expertise is part of the review team whose responsibility is to plan the search. The person conducting the search should be involved in the development of key questions, PICOTS (population, intervention, comparator, outcome, timing, and setting), analytic frameworks, and inclusion/exclusion criteria. |
An information specialist should ideally be included as part of the project team. |
Review authors should work closely with the Trials Search Coordinator for assistance in searching for studies to include in their reviews. |
3.1.2 Design the search strategy to address each key research question |
The search strategy should be based on the concepts identified in the analytic framework, and the review question (PICOTS). |
Search strategies should be highly sensitive in order to retrieve all potentially relevant studies. Use PICOS (population, intervention, comparator, outcome, and setting) to help structure the search. Consult the topic experts and the advisory team for advice. |
Searches are targeted at the eligibility criteria for the review not the review question). |
3.1.3 Use an independent librarian or other information specialist to peer review the search strategy |
Evidence-based Practice Centers (EPC) frequently internally peer review the electronic search strategies. |
If at all possible, the final search strategy should be peer reviewed to check for errors (spelling mistakes, incorrect use of operators, or failure to include relevant MeSH) that could reduce the recall of papers. |
Not mentioned. |
3.1.4 Search bibliographic databases |
Search at least two electronic databases. Begin with MEDLINE (including inprocess and other nonindexed citations) and the Cochrane Central Register of Controlled Trials. If topic is researched primarily outside of the United States, search relevant subject-specific databases, as well as databases with stronger international coverage of languages(s) of interest, such as EMBASE. |
The selection of electronic databases to search will depend upon the review topic. Importance of MEDLINE, EMBASE, and Cochrane Central Register of Controlled Trials noted. Details of scope of additional databases with narrower focus listed. |
The three most important sources to search for studies are Cochrane Central Register of Controlled Trials, MEDLINE, and EMBASE. |
3.1.5 Search citation indexes |
Use citation indexes. If possible use Web of Science or Scopus. If you do not have access to these databases, use Google Scholar, a free citation tracking database. |
Citation searching is useful for identifying a cluster of related, and therefore highly relevant, papers. |
Citation searching can be conducted for additional studies. |
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
3.1.6 Search literature cited by eligible studies |
Do forward citation search for any key articles. Handsearch if necessary if sources such as journals and conference abstracts are identified by key informants or technical experts. |
Scanning reference lists of relevant studies may be helpful in identifying further studies of interest. |
Should search reference list of included (and excluded) studies for additional studies. |
3.1.7 Update the search at intervals appropriate to the pace of generation of new information for the research question being addressed |
Update search at peer review draft stage. |
If the initial searches were conducted some time (e.g., 6 months) before the final analysis, it may be necessary to update the literature searches. |
While conducting a review, authors may be able to judge if relevant research is being published frequently, and therefore may be able to predict and suggest the need for more frequent updating of the review. (Updating is defined as including a new search.) |
3.1.8 Search subject-specific databases if other databases are unlikely to provide all relevant evidence |
Consult subject-specific databases that are relevant to the review topic. |
Consult subject-specific databases that are relevant to the review topic. Guidance provides details of scope of additional databases with narrower focus listed. |
If possible, search subject-specific databases that are relevant to the topic of the review. Access to these databases may be limited. |
3.1.9 Search regional bibliographic databases if other databases are unlikely to provide all relevant evidence |
If topic is researched primarily outside of the United States, search relevant subject-specific databases, as well as databases with stronger international coverage of language(s) of interest, such as EMBASE. |
Using additional databases such as LILACS (Latin American and Caribbean Health Sciences Literature) that contain collections of non-English language can minimize potential language bias. |
National and regional databases can be an important source of additional studies from journals not indexed in international databases. |
3.2 Take action to address potentially biased reporting of research results |
Provides guidance on addressing reporting bias (see below). |
Provides guidance on addressing reporting bias (see below). |
Provides guidance on addressing reporting bias (see below). |
3.2.1 Search grey-literature databases, clinical trial registries, and other sources of unpublished information about studies |
At a minimum, search grey literature for regulatory documents, clinical trial registries, and indexed conference abstracts. Search for unpublished articles, especially in areas where there is little published evidence, where the field or intervention is new or changing, where the topic is interdisciplinary, and with alternative medicine. |
Searching databases of grey literature is important to minimize publication and language bias. Researchers should consult grey-literature databases and catalogues from major libraries (e.g., British Library and the U.S. National Library of Medicine). It is useful to search trials registers in order to identify unpublished or ongoing trials. |
Grey literature can be an important source of studies for inclusion in reviews. Efforts should be made to identify unpublished studies. Trials registers and trials results registers are increasingly important sources of information of ongoing and unpublished trials. |
3.2.2 Invite researchers to clarify information related to study eligibility, study characteristics, and risk of bias |
EPC authors should prespecify if they will contact study authors for further information and describe plans in protocol. |
Sometimes the amount of information reported about a study is insufficient to make a decision about inclusion, and it can be helpful to contact study authors to ask for more details. |
Authors are recommended to contact the original investigators for clarification of eligibility, details of the study, and the numerical results. |
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
3.2.3 Invite all study sponsors and researchers to submit unpublished data, including unreported outcomes, for possible inclusion in the systematic review |
When interventions identified in key questions involve drugs or devices, it is important to supplement the literature search with a request to the manufacturer for a scientific information packet (includes information about published and unpublished trials or studies). Public comment periods are also opportunities for industry or other study sponsors to submit other data for consideration. |
Contacting experts and manufacturers may be useful for supplying information about unpublished or ongoing trials. |
It may be helpful to contact colleagues, or send formal letters of request to first authors of included reports to identify unpublished data. It may be desirable to send a letter to experts and pharmaceutical companies or others with an interest in the area. |
3.2.4 Handsearch selected journals and conference abstracts |
Handsearch selected recent or relevant proceedings of journals if you identify journals that are highly relevant to your topic, but are not fully indexed or not indexed at all by MEDLINE, particularly as identified by key informants or technical experts. Search for information only published in abstract form. |
Handsearching is an important way of identifying very recent publications that have not yet been included and indexed by electronic databases or of including articles from journals that are not indexed by electronic databases. Ideally include conference abstracts and interim results in order to avoid publication bias. |
Authors are not routinely expected to handsearch journals for their reviews, but they should discuss with their Trials Search Coordinator whether in their particular case handsearching of any journals or conference proceedings might be beneficial. Conference abstracts can be an important source of studies for inclusion in reviews. |
3.2.5 Conduct a web search |
Use Google Scholar if you do not have access to Web of Science or Scopus. |
Internet searching is useful for retrieving grey literature. Identifying and scanning specific relevant web sites is more useful than using a general search engine, such as Google. |
There is little empirical evidence concerning the value of using general Internet search engines, but it might be fruitful. |
3.2.6 Search for studies reported in languages other than English if appropriate |
Consider when topic necessitates search of non-English studies. Discuss with expert panel whether exclusion of non-English studies would bias the report. Document decision. Consider tracking relevant non-English studies to assess the potential for bias from excluding them. |
Whenever feasible, all relevant studies should be included regardless of language. This may be impossible due to time, resources, and facilities of translation. It is advisable to at least identify all non-English language papers and document their existence, but record language as the reason for exclusion in cases where they cannot be dealt with. |
Whenever possible review authors should attempt to identify and assess for eligibility all possibly relevant reports of trials irrespective of language of publication. |
3.3 Screen and select studies |
Provides guidance on screening and selecting studies (see below). |
Provides guidance on screening and selecting studies (see below). |
Provides guidance on screening and selecting studies (see below). |
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
3.3.1 Include or exclude studies based on the protocol’s prespecified criteria |
Inclusion and exclusion criteria should be determined a priori. Determination of inclusion and exclusion criteria is made with input by technical experts. Any changes to criteria should be documented and justified. |
The process by which decisions on the selection of studies will be made should be specified in the protocol. |
The protocol prespecifies the criteria for including and excluding studies in the review. The eligibility criteria are a combination of relevant aspects of the clinical question (population, intervention, comparator, outcomes [PICO]) plus specification of the types of studies that have addressed these questions. |
3.3.2 Use observational studies in addition to randomized controlled trials to evaluate harms of interventions |
Observational studies are almost always necessary to assess harms adequately. They may provide the best (or only) evidence for evaluating harms in minority or vulnerable populations who are underrepresented in clinical trials. Observational studies should be included when there are gaps in randomized clinical trials (RCTs) evidence and when observational studies will provide valid and useful evidence. |
Observational studies can provide useful information about the unintentional effects of an intervention, and in such situations it is important to assess their quality. |
One of the most important roles of nonrandomized studies is to assess potential unexpected or rare harms of interventions. |
3.3.3 Use two or more members of the review team, working independently, to screen and select studies |
Ensure quality control mechanism; this is usually through use of independent researchers to assess studies for eligibility. Pilot testing of screening process is particularly important if there is not dual-review screening. |
Good to have more than one researcher to help minimize bias and error at all stages of the review. Parallel independent assessments should be conducted to minimize the risk of errors. |
At least two people, independently. Process must be transparent, and chosen to minimize biases and human error. |
3.3.4 Train screeners using written documentation; test and retest screeners to improve accuracy and consistency |
Pilot-testing of screening process is particularly important if there is not dual-review screening. |
The selection process should be piloted by applying the inclusion criteria to a sample of papers in order to check that they can be reliably interpreted and that they classify the studies appropriately. |
Pilot-testing the eligibility criteria can be used to train the people who will be applying them and ensure that the criteria can be applied consistently by more than one person. |
3.3.5 Use one of two strategies to select studies: (1) read all full-text articles identified in the search, or (2) screen titles and abstracts of all articles and then read the full text of articles identified in initial screening |
Screening is typically done at two stages—title/abstract and full text. Typically title/abstract-level screen may err on the side of being more inclusive. |
Screening of potential studies is usually conducted in two stages: (1) initial screening of titles and abstracts against inclusion criteria to identify potentially relevant papers, and (2) screening of full papers identified in initial screening. |
Typical process: (1) merge results with reference software and remove duplicates; (2) examine titles and abstracts; (3) retrieve full text of relevant reports; (4) link multiple reports of the same study; (5) examine full-text reports for compliance with eligibility criteria; (6) contact investigators, if appropriate, to clarify study eligibility; and (7) make final decisions on study inclusion. |
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
3.3.6 Taking account of the risk of bias, consider including observational studies to address gaps in the evidence from randomized clinical trials on the benefits of interventions |
Observational studies should be included when there are gaps in RCT evidence and when observational studies will provide valid and useful evidence. |
Because of the risk of bias, careful consideration should be given to the inclusion of quasi-experimental studies in a review to assess the effectiveness of an intervention. |
Cochrane reviews focus primarily on randomized trials. Nonrandomized studies might be included (1) to provide an explicit evaluation of their weaknesses; (2) to provide evidence on interventions that cannot be randomized; or (3) to provide evidence of effects that cannot be adequately studied in randomized trials. |
3.4 Document the search |
Provides guidance for documenting the search (see below). |
Provides guidance for documenting the search (see below). |
Provides guidance for documenting the search (see below). |
3.4.1 Provide a line-by-line description of the search strategy, including the date of every search for each database, web browser, etc. |
While conducting the search, detailed notes about the full search strategy should be kept (e.g., database used, dates covered by search, date search was conducted, search terms used, nondatabase methods used, language restrictions). |
Record the search process and results contemporaneously. Provide full detail of the searches, including the databases and interfaces searched, dates covered, full detailed search strategies (including justifications for date or language restrictions), and the number of records retrieved. |
The full search strategies for each database will need to be included in an appendix. The search strategies will need to be copied and pasted exactly as run and included in full, together with the search set numbers and the number of records received. A single date should be specified to indicate when the most recent comprehensive search was started. |
3.4.2 Document the disposition of each report identified including reasons for their exclusion if appropriate |
Account for all citations identified from all sources. Report the list of excluded references. A flow chart accounts for all citations identified from all sources as well as accounting for all citations that were later excluded and why. |
Have a record of decisions made for each article. A flow chart showing the number of studies/papers remaining at each stage is a simple and useful way of documenting the study selection process. A list of studies excluded from the review should be reported where possible, giving the reasons for exclusion. This is most useful if it is restricted to “near misses” rather than all the research evidence identified. |
The review includes a characteristics of excluded studies table. This lists studies that appear to meet the eligibility criteria, but which were excluded, and the reasons for exclusion. |
3.5 Manage data collection |
Provides guidance on managing data collection (see below). |
Provides guidance on managing data collection (see below). |
Provides guidance on managing data collection (see below). |
3.5.1 At a minimum, use two or more researchers, working independently, to extract quantitative or other critical data from each study. For other types of data, one individual could extract the data while the second individual checks for accuracy and completeness. Establish a fair procedure for resolving discrepancies—do not simply give final decision-making power to the senior reviewer |
Quality control process for data extraction should be defined a priori. |
Ideally two researchers should independently perform the data extraction. At a minimum, one researcher can extract the data, with a second researcher independently checking the data extraction forms for accuracy and completeness. The process for resolving disagreements should be specified in the protocol. Disagreements should, where possible, be resolved by consensus after referring to the protocol; if necessary a third person may be consulted. |
More than one person should extract data from every report. |
The methods section of both the protocol and the review should detail how disagreements are handled. Disagreements can generally be resolved by discussion, but may require arbitration by another person or obtain more information from the study authors. |
|||
Procedure for resolving discrepancies should be defined in the protocol. |
Standards and Elements |
Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program |
Centre for Reviews and Dissemination (CRD) |
The Cochrane Collaboration |
3.5.2 Link publications from the same study to avoid including data from the same study more than once |
Publications from the same study are typically linked. |
It is important to identify duplicate publications of research results to ensure they are not treated as separate studies in the review. |
Multiple reports of the same study need to be linked together. |
3.5.3 Use standard data extraction forms developed for the specific systematic review |
Data abstraction forms are developed prior to data abstraction. Protocol should list elements included in data abstraction forms. |
Standardized data extraction forms should be designed with both the review question and subsequent analysis in mind. Information on study characteristics should be sufficiently detailed to allow readers to assess the applicability of the findings to their area of interest. |
Data collection forms are invaluable. The form should be linked directly to the review question and criteria for assessing eligibility of studies and serve as the historical record of the SR and the source of data for any analysis. |
3.5.4 Pilot-test the data extraction forms and process |
Data abstraction forms should be pilot tested by a sampling of studies. |
Data extraction forms should be piloted to ensure that all the relevant information is captured and that resources are not wasted on extracting data not required. |
All forms should be pilot-tested using a representative sample of studies to be reviewed. |
3.6 Critically appraise each study |
Provides guidance on appraising individual studies (see below). |
Provides guidance on appraising individual studies (see below). |
Provides guidance on appraising individual studies (see below). |
3.6.1 Systematically assess the risk of bias, using predefined criteria |
There are three steps to rating the risk of bias of individual studies (quality): (1) classify the study design (e.g., review, RCT, observational), (2) apply a predefined criteria for quality and critical appraisal (e.g., scale, checklists), and (3) arrive at a summary judgment of the study’s quality (good, fair, and poor). |
It is important to assess the risk of bias in included studies caused by inadequacies in study design, conduct, or analysis that may have led to the treatment effect being over-or underestimated. |
A risk-of-bias table should be made for each study, including judgments of low risk of bias, high risk of bias, or unclear risk of bias for the six domains of bias. Judgments should be explicit and informed by empirical evidence, likely direction of bias, and likely magnitude of bias. |
3.6.2 Assess relevance of the study’s populations, interventions, and outcome measures |
Must assess the relevance of the study populations in terms of severity of illness, comorbidities, and demographics (age, sex, and race). |
Assessment of risk of bias should consider whether groups were similar at outset of the study, selection bias, and attrition bias. |
Not applicable. The applicability of endpoints and outcomes can only be assessed in relation to a specific decision that needs to be made. Cochrane reviews do not have a specific implementation decision, so assessment of applicability is irrelevant. |
It is important to consider the reliability or validity of the actual outcome measure being used. The outcome should also be relevant and meaningful to both the intervention and the evaluation. |
|||
Must assess the relevance of the intervention, including drug dosing and adherence. |
REFERENCES
Atkins, D., S. Chang, G. Gartlehner, D. I. Buckley, E. P. Whitlock, E. Berliner, and D. Matchar. 2010. Assessing the applicability of studies when comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?productid=603&pageaction=displayproduct (accessed January 19, 2011).
Chou, R., N. Aronson, D. Atkins, A. S. Ismaila, P. Santaguida, D. H. Smith, E. Whitlock, T. J. Wilt, and D. Moher. 2010. AHRQ series paper 4: Assessing harms when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):502–512.
CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: York Publishing Services, Ltd.
Fu, R., G. Gartlehner, M. Grant, T. Shamliyan, A. Sedrakyan, T. J. Wilt, L. Griffith, M. Oremus, P. Raina, A. Ismaila, P. Santaguida, J. Lau, and T. A. Trikalinos. 2010. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=554 (accessed January 19, 2011).
Helfand, M., and H. Balshem. 2010. AHRQ series paper 2: Principles for developing guidance: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):484–490.
Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons.
Norris, S., D. Atkins, W. Bruening, S. Fox, E. Johnson, R. Kane, S. C. Morton, M. Oremus, M. Ospina, G. Randhawa, K. Schoelles, P. Shekelle, and M. Viswanathan. 2010. Selecting observational studies for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=454 (accessed January 19, 2011).
Owens, D. K., K. N. Lohr, D. Atkins, J. R. Treadwell, J. T. Reston, E. B. Bass, S. Chang, and M. Helfand. 2010. AHRQ series paper 5: Grading the strength of a body of evidence when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):513–523.
Relevo, R., and H. Balshem. 2011. Finding evidence for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=605 (accessed January 19, 2011).
Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ series paper 1: Comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):481–483.
White, C. M., S. Ip, M. McPheeters, T. S. Carey, R. Chou, K. N. Lohr, K. Robinson, K. McDonald, and E. Whitlock. 2009. Using existing systematic reviews to replace de novo processes in CERs. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effective-healthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=329 (accessed January 19, 2011).