E
Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 265
E Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies 265

OCR for page 265
TABLE E-1 Comparison of Chapter 3 Guidance on Conducting Systematic Reviews (SR) of Comparative 266 Effectiveness Research Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration 3.1 Conduct a Provides guidance on searching Provides guidance on searching Provides guidance on searching comprehensive, systematic for evidence (see below). for evidence (see below). for evidence (see below). search for evidence 3.1.1 Work with a librarian A person with library expertise An information specialist Review authors should work or other information is part of the review team should ideally be included as closely with the Trials Search specialist trained in whose responsibility is to part of the project team. Coordinator for assistance in performing systematic plan the search. The person searching for studies to include in reviews to plan the search conducting the search should their reviews. strategy be involved in the development of key questions, PICOTS (population, intervention, comparator, outcome, timing, and setting), analytic frameworks, and inclusion/ exclusion criteria. 3.1.2 Design the search The search strategy should Search strategies should be Searches are targeted at the strategy to address each key be based on the concepts highly sensitive in order eligibility criteria for the review research question identified in the analytic to retrieve all potentially (not the review question). framework, and the review relevant studies. Use PICOS question (PICOTS). (population, intervention, comparator, outcome, and setting) to help structure the search. Consult the topic experts and the advisory team for advice.

OCR for page 265
3.1.3 Use an independent Evidence-based Practice If at all possible, the final Not mentioned. librarian or other Centers (EPC) frequently search strategy should be peer information specialist to internally peer review the reviewed to check for errors peer review the search electronic search strategies. (spelling mistakes, incorrect strategy use of operators, or failure to include relevant MeSH) that could reduce the recall of papers. 3.1.4 Search bibliographic Search at least two electronic The selection of electronic The three most important sources databases databases. Begin with databases to search will to search for studies are Cochrane MEDLINE (including in- depend upon the review topic. Central Register of Controlled process and other nonindexed Importance of MEDLINE, Trials, MEDLINE, and EMBASE. citations) and the Cochrane EMBASE, and Cochrane Central Register of Controlled Central Register of Controlled Trials. If topic is researched Trials noted. Details of scope primarily outside of the United of additional databases with States, search relevant subject- narrower focus listed. specific databases, as well as databases with stronger international coverage of languages(s) of interest, such as EMBASE. 3.1.5 Search citation indexes Use citation indexes. If possible Citation searching is useful for Citation searching can be use Web of Science or Scopus. identifying a cluster of related, conducted for additional studies. If you do not have access to and therefore highly relevant, these databases, use Google papers. Scholar, a free citation tracking database. 267 continued

OCR for page 265
TABLE E-1 Continued 268 Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration 3.1.6 Search literature cited Do forward citation search for Scanning reference lists of Should search reference list of by eligible studies any key articles. Handsearch relevant studies may be helpful included (and excluded) studies if necessary if sources such in identifying further studies of for additional studies. as journals and conference interest. abstracts are identified by key informants or technical experts. 3.1.7 Update the search at Update search at peer review If the initial searches were While conducting a review, intervals appropriate to the draft stage. conducted some time (e.g., authors may be able to judge pace of generation of new 6 months) before the final if relevant research is being information for the research analysis, it may be necessary to published frequently, and question being addressed update the literature searches. therefore may be able to predict and suggest the need for more frequent updating of the review. (Updating is defined as including a new search.) 3.1.8 Search subject-specific Consult subject-specific Consult subject-specific If possible, search subject-specific databases if other databases databases that are relevant to databases that are relevant to databases that are relevant to the are unlikely to provide all the review topic. the review topic. Guidance topic of the review. Access to these relevant evidence provides details of scope of databases may be limited. additional databases with narrower focus listed.

OCR for page 265
3.1.9 Search regional If topic is researched primarily Using additional databases National and regional databases bibliographic databases if outside of the United States, such as LILACS (Latin can be an important source of other databases are unlikely search relevant subject-specific American and Caribbean additional studies from journals to provide all relevant databases, as well as databases Health Sciences Literature) that not indexed in international evidence with stronger international contain collections of non- databases. coverage of language(s) of English language can minimize interest, such as EMBASE. potential language bias. 3.2 Take action to address Provides guidance on Provides guidance on Provides guidance on addressing potentially biased reporting addressing reporting bias (see addressing reporting bias (see reporting bias (see below). of research results below). below). 3.2.1 Search grey-literature At a minimum, search grey Searching databases of grey Grey literature can be an databases, clinical trial literature for regulatory literature is important to important source of studies for registries, and other sources documents, clinical trial minimize publication and inclusion in reviews. Efforts of unpublished information registries, and indexed language bias. Researchers should be made to identify about studies conference abstracts. Search for should consult grey-literature unpublished studies. Trials unpublished articles, especially databases and catalogues registers and trials results registers in areas where there is little from major libraries (e.g., are increasingly important sources published evidence, where the British Library and the U.S. of information of ongoing and field or intervention is new National Library of Medicine). unpublished trials. or changing, where the topic It is useful to search trials is interdisciplinary, and with registers in order to identify alternative medicine. unpublished or ongoing trials. 3.2.2 Invite researchers to EPC authors should prespecify Sometimes the amount of Authors are recommended to clarify information related if they will contact study information reported about a contact the original investigators to study eligibility, study authors for further information study is insufficient to make a for clarification of eligibility, characteristics, and risk of and describe plans in protocol. decision about inclusion, and it details of the study, and the bias can be helpful to contact study numerical results. authors to ask for more details. 269 continued

OCR for page 265
TABLE E-1 Continued 270 Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration 3.2.3 Invite all study When interventions identified Contacting experts and It may be helpful to contact sponsors and researchers in key questions involve drugs manufacturers may be useful colleagues, or send formal letters to submit unpublished or devices, it is important for supplying information of request to first authors of data, including unreported to supplement the literature about unpublished or ongoing included reports to identify outcomes, for possible search with a request to the trials. unpublished data. It may be inclusion in the systematic manufacturer for a scientific desirable to send a letter to review information packet (includes experts and pharmaceutical information about published companies or others with an and unpublished trials or interest in the area. studies). Public comment periods are also opportunities for industry or other study sponsors to submit other data for consideration. 3.2.4 Handsearch selected Handsearch selected recent Handsearching is an important Authors are not routinely journals and conference or relevant proceedings of way of identifying very recent expected to handsearch journals abstracts journals if you identify journals publications that have not yet for their reviews, but they should that are highly relevant to been included and indexed discuss with their Trials Search your topic, but are not fully by electronic databases or Coordinator whether in their indexed or not indexed at all of including articles from particular case handsearching by MEDLINE, particularly as journals that are not indexed of any journals or conference identified by key informants by electronic databases. Ideally proceedings might be beneficial. or technical experts. Search for include conference abstracts Conference abstracts can be an information only published in and interim results in order to important source of studies for abstract form. avoid publication bias. inclusion in reviews.

OCR for page 265
3.2.5 Conduct a web search Use Google Scholar if you Internet searching is useful There is little empirical evidence do not have access to Web of for retrieving grey literature. concerning the value of using Science or Scopus. Identifying and scanning general Internet search engines, specific relevant web sites but it might be fruitful. is more useful than using a general search engine, such as Google. 3.2.6 Search for studies Consider when topic Whenever feasible, all relevant Whenever possible review reported in languages other necessitates search of non- studies should be included authors should attempt to than English if appropriate English studies. Discuss with regardless of language. This identify and assess for eligibility expert panel whether exclusion may be impossible due to time, all possibly relevant reports of of non-English studies would resources, and facilities of trials irrespective of language of bias the report. Document translation. It is advisable to at publication. decision. Consider tracking least identify all non-English relevant non-English studies language papers and document to assess the potential for bias their existence, but record from excluding them. language as the reason for exclusion in cases where they cannot be dealt with. 3.3 Screen and select Provides guidance on screening Provides guidance on screening Provides guidance on screening studies and selecting studies (see and selecting studies (see and selecting studies (see below). below). below). continued 271

OCR for page 265
TABLE E-1 Continued 272 Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration 3.3.1 Include or exclude Inclusion and exclusion criteria The process by which decisions The protocol prespecifies studies based on the should be determined a priori. on the selection of studies will the criteria for including protocol’s prespecified Determination of inclusion and be made should be specified in and excluding studies in the criteria exclusion criteria is made with the protocol. review. The eligibility criteria input by technical experts. Any are a combination of relevant changes to criteria should be aspects of the clinical question documented and justified. (population, intervention, comparator, outcomes [PICO]) plus specification of the types of studies that have addressed these questions. 3.3.2 Use observational Observational studies are Observational studies can One of the most important roles studies in addition to almost always necessary to provide useful information of nonrandomized studies is to randomized controlled assess harms adequately. about the unintentional effects assess potential unexpected or rare trials to evaluate harms of They may provide the of an intervention, and in such harms of interventions. interventions best (or only) evidence for situations it is important to evaluating harms in minority assess their quality. or vulnerable populations who are underrepresented in clinical trials. Observational studies should be included when there are gaps in randomized clinical trials (RCTs) evidence and when observational studies will provide valid and useful evidence.

OCR for page 265
3.3.3 Use two or Ensure quality control Good to have more than one At least two people, more members of the mechanism; this is usually researcher to help minimize independently. Process must review team, working through use of independent bias and error at all stages be transparent, and chosen to independently, to screen researchers to assess studies of the review. Parallel minimize biases and human error. and select studies for eligibility. Pilot testing independent assessments of screening process is should be conducted to particularly important if there minimize the risk of errors. is not dual-review screening. 3.3.4 Train screeners using Pilot-testing of screening The selection process should Pilot-testing the eligibility criteria written documentation; process is particularly be piloted by applying the can be used to train the people test and retest screeners important if there is not dual- inclusion criteria to a sample who will be applying them and to improve accuracy and review screening. of papers in order to check that ensure that the criteria can be consistency they can be reliably interpreted applied consistently by more than and that they classify the one person. studies appropriately. 3.3.5 Use one of two Screening is typically done Screening of potential studies Typical process: (1) merge results strategies to select studies: at two stages—title/abstract is usually conducted in two with reference software and (1) read all full-text articles and full text. Typically title/ stages: (1) initial screening remove duplicates; (2) examine identified in the search, abstract-level screen may err of titles and abstracts against titles and abstracts; (3) retrieve or (2) screen titles and on the side of being more inclusion criteria to identify full text of relevant reports; (4) abstracts of all articles and inclusive. potentially relevant papers, link multiple reports of the same then read the full text of and (2) screening of full papers study; (5) examine full-text reports articles identified in initial identified in initial screening. for compliance with eligibility screening criteria; (6) contact investigators, if appropriate, to clarify study eligibility; and (7) make final decisions on study inclusion. 273 continued

OCR for page 265
TABLE E-1 Continued 274 Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration 3.3.6 Taking account of Observational studies should Because of the risk of bias, Cochrane reviews focus primarily the risk of bias, consider be included when there are careful consideration should on randomized trials. including observational gaps in RCT evidence and be given to the inclusion of Nonrandomized studies might be studies to address gaps when observational studies quasi-experimental studies included (1) to provide an explicit in the evidence from will provide valid and useful in a review to assess the evaluation of their weaknesses; randomized clinical trials on evidence. effectiveness of an intervention. (2) to provide evidence on the benefits of interventions interventions that cannot be randomized; or (3) to provide evidence of effects that cannot be adequately studied in randomized trials. 3.4 Document the search Provides guidance for Provides guidance for Provides guidance for documenting the search (see documenting the search (see documenting the search (see below). below). below). 3.4.1 Provide a line-by-line While conducting the search, Record the search process and The full search strategies for description of the search detailed notes about the full results contemporaneously. each database will need to be strategy, including the date search strategy should be kept Provide full detail of the included in an appendix. The of every search for each (e.g., database used, dates searches, including the search strategies will need to be database, web browser, etc. covered by search, date search databases and interfaces copied and pasted exactly as run was conducted, search terms searched, dates covered, full and included in full, together used, nondatabase methods detailed search strategies with the search set numbers and used, language restrictions). (including justifications for the number of records received. date or language restrictions), A single date should be specified and the number of records to indicate when the most recent retrieved. comprehensive search was started.

OCR for page 265
3.4.2 Document the Account for all citations Have a record of decisions The review includes a disposition of each report identified from all sources. made for each article. A flow characteristics of excluded studies identified including reasons Report the list of excluded chart showing the number table. This lists studies that appear for their exclusion if references. A flow chart of studies/papers remaining to meet the eligibility criteria, but appropriate accounts for all citations at each stage is a simple and which were excluded, and the identified from all sources useful way of documenting reasons for exclusion. as well as accounting for the study selection process. A all citations that were later list of studies excluded from excluded and why. the review should be reported where possible, giving the reasons for exclusion. This is most useful if it is restricted to “near misses” rather than all the research evidence identified. 3.5 Manage data collection Provides guidance on manag- Provides guidance on manag- Provides guidance on managing ing data collection (see below). ing data collection (see below). data collection (see below). 3.5.1 At a minimum, use Quality control process for data Ideally two researchers should More than one person should two or more researchers, extraction should be defined a independently perform the data extract data from every report. working independently, priori. extraction. At a minimum, one to extract quantitative or researcher can extract the data, The methods section of both other critical data from Procedure for resolving with a second researcher inde- the protocol and the review each study. For other types discrepancies should be pendently checking the data ex- should detail how disagreements of data, one individual defined in the protocol. traction forms for accuracy and are handled. Disagreements could extract the data completeness. The process for can generally be resolved by while the second individual resolving disagreements should discussion, but may require checks for accuracy and be specified in the protocol. Dis- arbitration by another person or completeness. Establish a agreements should, where pos- obtain more information from the fair procedure for resolving sible, be resolved by consensus study authors. discrepancies—do not after referring to the protocol; if 275 simply give final decision- necessary a third person may be continued making power to the senior consulted. reviewer

OCR for page 265
TABLE E-1 Continued 276 Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration 3.5.2 Link publications from Publications from the same It is important to identify Multiple reports of the same study the same study to avoid study are typically linked. duplicate publications of need to be linked together. including data from the research results to ensure they same study more than once are not treated as separate studies in the review. 3.5.3 Use standard data Data abstraction forms are Standardized data extraction Data collection forms are extraction forms developed developed prior to data forms should be designed invaluable. The form should be for the specific systematic abstraction. Protocol should with both the review question linked directly to the review review list elements included in data and subsequent analysis question and criteria for assessing abstraction forms. in mind. Information on eligibility of studies and serve as study characteristics should the historical record of the SR and be sufficiently detailed to the source of data for any analysis. allow readers to assess the applicability of the findings to their area of interest. 3.5.4 Pilot-test the data Data abstraction forms should Data extraction forms should All forms should be pilot-tested extraction forms and be pilot tested by a sampling of be piloted to ensure that all using a representative sample of process studies. the relevant information is studies to be reviewed. captured and that resources are not wasted on extracting data not required. 3.6 Critically appraise each Provides guidance on Provides guidance on Provides guidance on appraising study appraising individual studies appraising individual studies individual studies (see below). (see below). (see below).

OCR for page 265
. 3.6.1 Systematically assess There are three steps to rating It is important to assess the A risk-of-bias table should be the risk of bias, using the risk of bias of individual risk of bias in included studies made for each study, including predefined criteria studies (quality): (1) classify caused by inadequacies in judgments of low risk of bias, the study design (e.g., review, study design, conduct, or high risk of bias, or unclear risk RCT, observational), (2) apply analysis that may have led to of bias for the six domains of bias. a predefined criteria for quality the treatment effect being over- Judgments should be explicit and and critical appraisal (e.g., or underestimated. informed by empirical evidence, scale, checklists), and (3) arrive likely direction of bias, and likely at a summary judgment of the magnitude of bias. study’s quality (good, fair, and poor). 3.6.2 Assess relevance of Must assess the relevance Assessment of risk of bias Not applicable. The applicability the study’s populations, of the study populations should consider whether of endpoints and outcomes can interventions, and outcome in terms of severity of groups were similar at outset only be assessed in relation to a measures illness, comorbidities, and of the study, selection bias, and specific decision that needs to be demographics (age, sex, attrition bias. made. Cochrane reviews do not and race). have a specific implementation It is important to consider the decision, so assessment of Must assess the relevance reliability or validity of the applicability is irrelevant. of the intervention, actual outcome measure being including drug dosing used. The outcome should also and adherence. be relevant and meaningful to both the intervention and the evaluation. continued 277

OCR for page 265
TABLE E-1 Continued 278 Agency for Healthcare Research and Quality (AHRQ) Centre for Reviews and Standards and Elements Effective Health Care Program Dissemination (CRD) The Cochrane Collaboration Must assess the applicability It is often helpful to assess the of the study’s outcomes. quality of the intervention and Outcomes should include the its implementation. most important clinical benefits and harms. Surrogate outcomes are evaluated if they are defined as important outcomes in the key questions. They may also be considered indirect outcomes of a final health outcome. Relationship between surrogate and final health outcome should be depicted by analytic framework. 3.6.3 Assess the fidelity Not mentioned. A review should assess Not applicable. The applicability of the implementation of whether the intervention was of interventions can only be interventions implemented as planned in the assessed in relation to a specific individual studies. decision that needs to be made and Cochrane reviews do not have a specific implementation decision. NOTE: Some information on AHRQ-, CRD-. and Cochrane-recommended methods was provided via personal communication with Stephanie Chang, EPC Program Task Order Officer, AHRQ (October 5, 2010); Lesley Stewart, Director, CRD (October 14, 2010); and Julian Higgins, Senior Statistician, MRC Biostatistics Unit, Institute of Public Health, University of Cambridge (October 4, 2010).

OCR for page 265
279 APPENDIX E REFERENCES Atkins, D., S. Chang, G. Gartlehner, D. I. Buckley, E. P. Whitlock, E. Berliner, and D. Matchar. 2010. Assessing the applicability of studies when comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?productid=603& pageaction=displayproduct (accessed January 19, 2011). Chou, R., N. Aronson, D. Atkins, A. S. Ismaila, P. Santaguida, D. H. Smith, E. Whitlock, T. J. Wilt, and D. Moher. 2010. AHRQ series paper 4: Assessing harms when com- paring medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):502–512. CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: York Publishing Services, Ltd. Fu, R., G. Gartlehner, M. Grant, T. Shamliyan, A. Sedrakyan, T. J. Wilt, L. Griffith, M. Oremus, P. Raina, A. Ismaila, P. Santaguida, J. Lau, and T. A. Trikalinos. 2010. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. In Methods guide for compara- tive effectiveness reviews, edited by Agency for Healthcare Research and Qual- ity. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides- reviews-and-reports/?pageaction=displayProduct&productID=554 (accessed January 19, 2011). Helfand, M., and H. Balshem. 2010. AHRQ series paper 2: Principles for develop - ing guidance: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):484–490. Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons. Norris, S., D. Atkins, W. Bruening, S. Fox, E. Johnson, R. Kane, S. C. Morton, M. Oremus, M. Ospina, G. Randhawa, K. Schoelles, P. Shekelle, and M. Viswanathan. 2010. Selecting observational studies for comparing medical inter- ventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct &productID=454 (accessed January 19, 2011). Owens, D. K., K. N. Lohr, D. Atkins, J. R. Treadwell, J. T. Reston, E. B. Bass, S. Chang, and M. Helfand. 2010. AHRQ series paper 5: Grading the strength of a body of evidence when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):513–523. Relevo, R., and H. Balshem. 2011. Finding evidence for comparing medical interven - tions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct &productID=605 (accessed January 19, 2011). Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ series paper 1: Comparing medical interventions: AHRQ and the Effective Health Care Pro - gram. Journal of Clinical Epidemiology 63(5):481–483. White, C. M., S. Ip, M. McPheeters, T. S. Carey, R. Chou, K. N. Lohr, K. Robinson, K. McDonald, and E. Whitlock. 2009. Using existing systematic reviews to replace de novo processes in CERs. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effective- healthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?page action=displayProduct&productID=329 (accessed January 19, 2011).

OCR for page 265
280 FINDING WHAT WORKS IN HEALTH CARE Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2010. AHRQ series paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):491–501.