National Academies Press: OpenBook

Finding What Works in Health Care: Standards for Systematic Reviews (2011)

Chapter: Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies

« Previous: Appendix D: Expert Guidance for Chapter 2: Standards for Initiating a Systematic Review
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

E
Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

TABLE E-1 Comparison of Chapter 3 Guidance on Conducting Systematic Reviews (SR) of Comparative Effectiveness Research

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

3.1 Conduct a comprehensive, systematic search for evidence

Provides guidance on searching for evidence (see below).

Provides guidance on searching for evidence (see below).

Provides guidance on searching for evidence (see below).

3.1.1 Work with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy

A person with library expertise is part of the review team whose responsibility is to plan the search. The person conducting the search should be involved in the development of key questions, PICOTS (population, intervention, comparator, outcome, timing, and setting), analytic frameworks, and inclusion/exclusion criteria.

An information specialist should ideally be included as part of the project team.

Review authors should work closely with the Trials Search Coordinator for assistance in searching for studies to include in their reviews.

3.1.2 Design the search strategy to address each key research question

The search strategy should be based on the concepts identified in the analytic framework, and the review question (PICOTS).

Search strategies should be highly sensitive in order to retrieve all potentially relevant studies. Use PICOS (population, intervention, comparator, outcome, and setting) to help structure the search. Consult the topic experts and the advisory team for advice.

Searches are targeted at the eligibility criteria for the review not the review question).

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

3.1.3 Use an independent librarian or other information specialist to peer review the search strategy

Evidence-based Practice Centers (EPC) frequently internally peer review the electronic search strategies.

If at all possible, the final search strategy should be peer reviewed to check for errors (spelling mistakes, incorrect use of operators, or failure to include relevant MeSH) that could reduce the recall of papers.

Not mentioned.

3.1.4 Search bibliographic databases

Search at least two electronic databases. Begin with MEDLINE (including inprocess and other nonindexed citations) and the Cochrane Central Register of Controlled Trials. If topic is researched primarily outside of the United States, search relevant subject-specific databases, as well as databases with stronger international coverage of languages(s) of interest, such as EMBASE.

The selection of electronic databases to search will depend upon the review topic. Importance of MEDLINE, EMBASE, and Cochrane Central Register of Controlled Trials noted. Details of scope of additional databases with narrower focus listed.

The three most important sources to search for studies are Cochrane Central Register of Controlled Trials, MEDLINE, and EMBASE.

3.1.5 Search citation indexes

Use citation indexes. If possible use Web of Science or Scopus. If you do not have access to these databases, use Google Scholar, a free citation tracking database.

Citation searching is useful for identifying a cluster of related, and therefore highly relevant, papers.

Citation searching can be conducted for additional studies.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

3.1.6 Search literature cited by eligible studies

Do forward citation search for any key articles. Handsearch if necessary if sources such as journals and conference abstracts are identified by key informants or technical experts.

Scanning reference lists of relevant studies may be helpful in identifying further studies of interest.

Should search reference list of included (and excluded) studies for additional studies.

3.1.7 Update the search at intervals appropriate to the pace of generation of new information for the research question being addressed

Update search at peer review draft stage.

If the initial searches were conducted some time (e.g., 6 months) before the final analysis, it may be necessary to update the literature searches.

While conducting a review, authors may be able to judge if relevant research is being published frequently, and therefore may be able to predict and suggest the need for more frequent updating of the review. (Updating is defined as including a new search.)

3.1.8 Search subject-specific databases if other databases are unlikely to provide all relevant evidence

Consult subject-specific databases that are relevant to the review topic.

Consult subject-specific databases that are relevant to the review topic. Guidance provides details of scope of additional databases with narrower focus listed.

If possible, search subject-specific databases that are relevant to the topic of the review. Access to these databases may be limited.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

3.1.9 Search regional bibliographic databases if other databases are unlikely to provide all relevant evidence

If topic is researched primarily outside of the United States, search relevant subject-specific databases, as well as databases with stronger international coverage of language(s) of interest, such as EMBASE.

Using additional databases such as LILACS (Latin American and Caribbean Health Sciences Literature) that contain collections of non-English language can minimize potential language bias.

National and regional databases can be an important source of additional studies from journals not indexed in international databases.

3.2 Take action to address potentially biased reporting of research results

Provides guidance on addressing reporting bias (see below).

Provides guidance on addressing reporting bias (see below).

Provides guidance on addressing reporting bias (see below).

3.2.1 Search grey-literature databases, clinical trial registries, and other sources of unpublished information about studies

At a minimum, search grey literature for regulatory documents, clinical trial registries, and indexed conference abstracts. Search for unpublished articles, especially in areas where there is little published evidence, where the field or intervention is new or changing, where the topic is interdisciplinary, and with alternative medicine.

Searching databases of grey literature is important to minimize publication and language bias. Researchers should consult grey-literature databases and catalogues from major libraries (e.g., British Library and the U.S. National Library of Medicine). It is useful to search trials registers in order to identify unpublished or ongoing trials.

Grey literature can be an important source of studies for inclusion in reviews. Efforts should be made to identify unpublished studies. Trials registers and trials results registers are increasingly important sources of information of ongoing and unpublished trials.

3.2.2 Invite researchers to clarify information related to study eligibility, study characteristics, and risk of bias

EPC authors should prespecify if they will contact study authors for further information and describe plans in protocol.

Sometimes the amount of information reported about a study is insufficient to make a decision about inclusion, and it can be helpful to contact study authors to ask for more details.

Authors are recommended to contact the original investigators for clarification of eligibility, details of the study, and the numerical results.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

3.2.3 Invite all study sponsors and researchers to submit unpublished data, including unreported outcomes, for possible inclusion in the systematic review

When interventions identified in key questions involve drugs or devices, it is important to supplement the literature search with a request to the manufacturer for a scientific information packet (includes information about published and unpublished trials or studies). Public comment periods are also opportunities for industry or other study sponsors to submit other data for consideration.

Contacting experts and manufacturers may be useful for supplying information about unpublished or ongoing trials.

It may be helpful to contact colleagues, or send formal letters of request to first authors of included reports to identify unpublished data. It may be desirable to send a letter to experts and pharmaceutical companies or others with an interest in the area.

3.2.4 Handsearch selected journals and conference abstracts

Handsearch selected recent or relevant proceedings of journals if you identify journals that are highly relevant to your topic, but are not fully indexed or not indexed at all by MEDLINE, particularly as identified by key informants or technical experts. Search for information only published in abstract form.

Handsearching is an important way of identifying very recent publications that have not yet been included and indexed by electronic databases or of including articles from journals that are not indexed by electronic databases. Ideally include conference abstracts and interim results in order to avoid publication bias.

Authors are not routinely expected to handsearch journals for their reviews, but they should discuss with their Trials Search Coordinator whether in their particular case handsearching of any journals or conference proceedings might be beneficial. Conference abstracts can be an important source of studies for inclusion in reviews.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

3.2.5 Conduct a web search

Use Google Scholar if you do not have access to Web of Science or Scopus.

Internet searching is useful for retrieving grey literature. Identifying and scanning specific relevant web sites is more useful than using a general search engine, such as Google.

There is little empirical evidence concerning the value of using general Internet search engines, but it might be fruitful.

3.2.6 Search for studies reported in languages other than English if appropriate

Consider when topic necessitates search of non-English studies. Discuss with expert panel whether exclusion of non-English studies would bias the report. Document decision. Consider tracking relevant non-English studies to assess the potential for bias from excluding them.

Whenever feasible, all relevant studies should be included regardless of language. This may be impossible due to time, resources, and facilities of translation. It is advisable to at least identify all non-English language papers and document their existence, but record language as the reason for exclusion in cases where they cannot be dealt with.

Whenever possible review authors should attempt to identify and assess for eligibility all possibly relevant reports of trials irrespective of language of publication.

3.3 Screen and select studies

Provides guidance on screening and selecting studies (see below).

Provides guidance on screening and selecting studies (see below).

Provides guidance on screening and selecting studies (see below).

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

3.3.1 Include or exclude studies based on the protocol’s prespecified criteria

Inclusion and exclusion criteria should be determined a priori. Determination of inclusion and exclusion criteria is made with input by technical experts. Any changes to criteria should be documented and justified.

The process by which decisions on the selection of studies will be made should be specified in the protocol.

The protocol prespecifies the criteria for including and excluding studies in the review. The eligibility criteria are a combination of relevant aspects of the clinical question (population, intervention, comparator, outcomes [PICO]) plus specification of the types of studies that have addressed these questions.

3.3.2 Use observational studies in addition to randomized controlled trials to evaluate harms of interventions

Observational studies are almost always necessary to assess harms adequately. They may provide the best (or only) evidence for evaluating harms in minority or vulnerable populations who are underrepresented in clinical trials. Observational studies should be included when there are gaps in randomized clinical trials (RCTs) evidence and when observational studies will provide valid and useful evidence.

Observational studies can provide useful information about the unintentional effects of an intervention, and in such situations it is important to assess their quality.

One of the most important roles of nonrandomized studies is to assess potential unexpected or rare harms of interventions.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

3.3.3 Use two or more members of the review team, working independently, to screen and select studies

Ensure quality control mechanism; this is usually through use of independent researchers to assess studies for eligibility. Pilot testing of screening process is particularly important if there is not dual-review screening.

Good to have more than one researcher to help minimize bias and error at all stages of the review. Parallel independent assessments should be conducted to minimize the risk of errors.

At least two people, independently. Process must be transparent, and chosen to minimize biases and human error.

3.3.4 Train screeners using written documentation; test and retest screeners to improve accuracy and consistency

Pilot-testing of screening process is particularly important if there is not dual-review screening.

The selection process should be piloted by applying the inclusion criteria to a sample of papers in order to check that they can be reliably interpreted and that they classify the studies appropriately.

Pilot-testing the eligibility criteria can be used to train the people who will be applying them and ensure that the criteria can be applied consistently by more than one person.

3.3.5 Use one of two strategies to select studies: (1) read all full-text articles identified in the search, or (2) screen titles and abstracts of all articles and then read the full text of articles identified in initial screening

Screening is typically done at two stages—title/abstract and full text. Typically title/abstract-level screen may err on the side of being more inclusive.

Screening of potential studies is usually conducted in two stages: (1) initial screening of titles and abstracts against inclusion criteria to identify potentially relevant papers, and (2) screening of full papers identified in initial screening.

Typical process: (1) merge results with reference software and remove duplicates; (2) examine titles and abstracts; (3) retrieve full text of relevant reports; (4) link multiple reports of the same study; (5) examine full-text reports for compliance with eligibility criteria; (6) contact investigators, if appropriate, to clarify study eligibility; and (7) make final decisions on study inclusion.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

3.3.6 Taking account of the risk of bias, consider including observational studies to address gaps in the evidence from randomized clinical trials on the benefits of interventions

Observational studies should be included when there are gaps in RCT evidence and when observational studies will provide valid and useful evidence.

Because of the risk of bias, careful consideration should be given to the inclusion of quasi-experimental studies in a review to assess the effectiveness of an intervention.

Cochrane reviews focus primarily on randomized trials.

Nonrandomized studies might be included (1) to provide an explicit evaluation of their weaknesses; (2) to provide evidence on interventions that cannot be randomized; or (3) to provide evidence of effects that cannot be adequately studied in randomized trials.

3.4 Document the search

Provides guidance for documenting the search (see below).

Provides guidance for documenting the search (see below).

Provides guidance for documenting the search (see below).

3.4.1 Provide a line-by-line description of the search strategy, including the date of every search for each database, web browser, etc.

While conducting the search, detailed notes about the full search strategy should be kept (e.g., database used, dates covered by search, date search was conducted, search terms used, nondatabase methods used, language restrictions).

Record the search process and results contemporaneously. Provide full detail of the searches, including the databases and interfaces searched, dates covered, full detailed search strategies (including justifications for date or language restrictions), and the number of records retrieved.

The full search strategies for each database will need to be included in an appendix. The search strategies will need to be copied and pasted exactly as run and included in full, together with the search set numbers and the number of records received. A single date should be specified to indicate when the most recent comprehensive search was started.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

3.4.2 Document the disposition of each report identified including reasons for their exclusion if appropriate

Account for all citations identified from all sources. Report the list of excluded references. A flow chart accounts for all citations identified from all sources as well as accounting for all citations that were later excluded and why.

Have a record of decisions made for each article. A flow chart showing the number of studies/papers remaining at each stage is a simple and useful way of documenting the study selection process. A list of studies excluded from the review should be reported where possible, giving the reasons for exclusion. This is most useful if it is restricted to “near misses” rather than all the research evidence identified.

The review includes a characteristics of excluded studies table. This lists studies that appear to meet the eligibility criteria, but which were excluded, and the reasons for exclusion.

3.5 Manage data collection

Provides guidance on managing data collection (see below).

Provides guidance on managing data collection (see below).

Provides guidance on managing data collection (see below).

3.5.1 At a minimum, use two or more researchers, working independently, to extract quantitative or other critical data from each study. For other types of data, one individual could extract the data while the second individual checks for accuracy and completeness. Establish a fair procedure for resolving discrepancies—do not simply give final decision-making power to the senior reviewer

Quality control process for data extraction should be defined a priori.

Ideally two researchers should independently perform the data extraction. At a minimum, one researcher can extract the data, with a second researcher independently checking the data extraction forms for accuracy and completeness. The process for resolving disagreements should be specified in the protocol. Disagreements should, where possible, be resolved by consensus after referring to the protocol; if necessary a third person may be consulted.

More than one person should extract data from every report.

The methods section of both the protocol and the review should detail how disagreements are handled. Disagreements can generally be resolved by discussion, but may require arbitration by another person or obtain more information from the study authors.

Procedure for resolving discrepancies should be defined in the protocol.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

3.5.2 Link publications from the same study to avoid including data from the same study more than once

Publications from the same study are typically linked.

It is important to identify duplicate publications of research results to ensure they are not treated as separate studies in the review.

Multiple reports of the same study need to be linked together.

3.5.3 Use standard data extraction forms developed for the specific systematic review

Data abstraction forms are developed prior to data abstraction. Protocol should list elements included in data abstraction forms.

Standardized data extraction forms should be designed with both the review question and subsequent analysis in mind. Information on study characteristics should be sufficiently detailed to allow readers to assess the applicability of the findings to their area of interest.

Data collection forms are invaluable. The form should be linked directly to the review question and criteria for assessing eligibility of studies and serve as the historical record of the SR and the source of data for any analysis.

3.5.4 Pilot-test the data extraction forms and process

Data abstraction forms should be pilot tested by a sampling of studies.

Data extraction forms should be piloted to ensure that all the relevant information is captured and that resources are not wasted on extracting data not required.

All forms should be pilot-tested using a representative sample of studies to be reviewed.

3.6 Critically appraise each study

Provides guidance on appraising individual studies (see below).

Provides guidance on appraising individual studies (see below).

Provides guidance on appraising individual studies (see below).

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

3.6.1 Systematically assess the risk of bias, using predefined criteria

There are three steps to rating the risk of bias of individual studies (quality): (1) classify the study design (e.g., review, RCT, observational), (2) apply a predefined criteria for quality and critical appraisal (e.g., scale, checklists), and (3) arrive at a summary judgment of the study’s quality (good, fair, and poor).

It is important to assess the risk of bias in included studies caused by inadequacies in study design, conduct, or analysis that may have led to the treatment effect being over-or underestimated.

A risk-of-bias table should be made for each study, including judgments of low risk of bias, high risk of bias, or unclear risk of bias for the six domains of bias. Judgments should be explicit and informed by empirical evidence, likely direction of bias, and likely magnitude of bias.

3.6.2 Assess relevance of the study’s populations, interventions, and outcome measures

Must assess the relevance of the study populations in terms of severity of illness, comorbidities, and demographics (age, sex, and race).

Assessment of risk of bias should consider whether groups were similar at outset of the study, selection bias, and attrition bias.

Not applicable. The applicability of endpoints and outcomes can only be assessed in relation to a specific decision that needs to be made. Cochrane reviews do not have a specific implementation decision, so assessment of applicability is irrelevant.

It is important to consider the reliability or validity of the actual outcome measure being used. The outcome should also be relevant and meaningful to both the intervention and the evaluation.

Must assess the relevance of the intervention, including drug dosing and adherence.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Standards and Elements

Agency for Healthcare Research and Quality  (AHRQ) Effective Health Care Program

Centre for Reviews and Dissemination (CRD)

The Cochrane  Collaboration

 

Must assess the applicability of the study’s outcomes. Outcomes should include the most important clinical benefits and harms. Surrogate outcomes are evaluated if they are defined as important outcomes in the key questions. They may also be considered indirect outcomes of a final health outcome. Relationship between surrogate and final health outcome should be depicted by analytic framework.

It is often helpful to assess the quality of the intervention and its implementation.

 

3.6.3 Assess the fidelity of the implementation of interventions

Not mentioned.

A review should assess whether the intervention was implemented as planned in the individual studies.

Not applicable. The applicability of interventions can only be assessed in relation to a specific decision that needs to be made and Cochrane reviews do not have a specific implementation decision.

NOTE: Some information on AHRQ-, CRD-. and Cochrane-recommended methods was provided via personal communication with Stephanie Chang, EPC Program Task Order Officer, AHRQ (October 5, 2010); Lesley Stewart, Director, CRD (October 14, 2010); and Julian Higgins, Senior Statistician, MRC Biostatistics Unit, Institute of Public Health, University of Cambridge (October 4, 2010).

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

REFERENCES

Atkins, D., S. Chang, G. Gartlehner, D. I. Buckley, E. P. Whitlock, E. Berliner, and D. Matchar. 2010. Assessing the applicability of studies when comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?productid=603&pageaction=displayproduct (accessed January 19, 2011).

Chou, R., N. Aronson, D. Atkins, A. S. Ismaila, P. Santaguida, D. H. Smith, E. Whitlock, T. J. Wilt, and D. Moher. 2010. AHRQ series paper 4: Assessing harms when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):502–512.

CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: York Publishing Services, Ltd.

Fu, R., G. Gartlehner, M. Grant, T. Shamliyan, A. Sedrakyan, T. J. Wilt, L. Griffith, M. Oremus, P. Raina, A. Ismaila, P. Santaguida, J. Lau, and T. A. Trikalinos. 2010. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=554 (accessed January 19, 2011).

Helfand, M., and H. Balshem. 2010. AHRQ series paper 2: Principles for developing guidance: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):484–490.

Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons.

Norris, S., D. Atkins, W. Bruening, S. Fox, E. Johnson, R. Kane, S. C. Morton, M. Oremus, M. Ospina, G. Randhawa, K. Schoelles, P. Shekelle, and M. Viswanathan. 2010. Selecting observational studies for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=454 (accessed January 19, 2011).

Owens, D. K., K. N. Lohr, D. Atkins, J. R. Treadwell, J. T. Reston, E. B. Bass, S. Chang, and M. Helfand. 2010. AHRQ series paper 5: Grading the strength of a body of evidence when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):513–523.

Relevo, R., and H. Balshem. 2011. Finding evidence for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=605 (accessed January 19, 2011).

Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ series paper 1: Comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):481–483.

White, C. M., S. Ip, M. McPheeters, T. S. Carey, R. Chou, K. N. Lohr, K. Robinson, K. McDonald, and E. Whitlock. 2009. Using existing systematic reviews to replace de novo processes in CERs. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www.effective-healthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayProduct&productID=329 (accessed January 19, 2011).

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×

Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2010. AHRQ series paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):491–501.

Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 265
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 266
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 267
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 268
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 269
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 270
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 271
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 272
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 273
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 274
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 275
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 276
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 277
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 278
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 279
Suggested Citation:"Appendix E: Expert Guidance for Chapter 3: Standards for Finding and Assessing Individual Studies." Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. doi: 10.17226/13059.
×
Page 280
Next: Appendix F: Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence »
Finding What Works in Health Care: Standards for Systematic Reviews Get This Book
×
Buy Paperback | $61.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Healthcare decision makers in search of reliable information that compares health interventions increasingly turn to systematic reviews for the best summary of the evidence. Systematic reviews identify, select, assess, and synthesize the findings of similar but separate studies, and can help clarify what is known and not known about the potential benefits and harms of drugs, devices, and other healthcare services. Systematic reviews can be helpful for clinicians who want to integrate research findings into their daily practices, for patients to make well-informed choices about their own care, for professional medical societies and other organizations that develop clinical practice guidelines.

Too often systematic reviews are of uncertain or poor quality. There are no universally accepted standards for developing systematic reviews leading to variability in how conflicts of interest and biases are handled, how evidence is appraised, and the overall scientific rigor of the process.

In Finding What Works in Health Care the Institute of Medicine (IOM) recommends 21 standards for developing high-quality systematic reviews of comparative effectiveness research. The standards address the entire systematic review process from the initial steps of formulating the topic and building the review team to producing a detailed final report that synthesizes what the evidence shows and where knowledge gaps remain.

Finding What Works in Health Care also proposes a framework for improving the quality of the science underpinning systematic reviews. This book will serve as a vital resource for both sponsors and producers of systematic reviews of comparative effectiveness research.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!