FINDING WHAT WORKS IN HEALTH CARE
STANDARDS FOR SYSTEMATIC REVIEWS
Jill Eden, Laura Levit, Alfred Berg, and Sally Morton, Editors
INSTITUTE OF MEDICINE
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W.
Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract No. HHSP23320042509X1 between the National Academy of Sciences and the Department of Health and Human Services. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project.
Library of Congress Cataloging-in-Publication Data
Institute of Medicine (U.S.). Committee on Standards for Systematic Reviews of Comparative Effectiveness Research.
Finding what works in health care : standards for systematic reviews / Committee on Standards for Systematic Reviews of Comparative Effectiveness Research, Board on Health Care Services, Institute of Medicine of the National Academies ; Jill Eden … [et al.], editors.
p. ; cm.
Includes bibliographical references.
ISBN 978-0-309-16425-2 (pbk.) — ISBN 978-0-309-16426-9 (pdf)
1. Medical care—Standards—United States. 2. Medical care—United States—Quality control. I. Eden, Jill. II. Title.
[DNLM: 1. Comparative Effectiveness Research—standards—United States. 2. Outcome and Process Assessment (Health Care)—United States. W 84.3]
RA399.A3I565 2011
610.28’9—dc23
2011017455
Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
For more information about the Institute of Medicine, visit the IOM home page at: www.iom.edu.
Copyright 2011 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
The serpent has been a symbol of long life, healing, and knowledge among almost all cultures and religions since the beginning of recorded history. The serpent adopted as a logotype by the Institute of Medicine is a relief carving from ancient Greece, now held by the Staatliche Museen in Berlin.
Suggested citation: IOM (Institute of Medicine). 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON STANDARDS FOR SYSTEMATIC REVIEWS OF COMPARATIVE EFFECTIVENESS RESEARCH
ALFRED O. BERG (Chair), Professor,
Department of Family Medicine, University of Washington School of Medicine, Seattle, WA
SALLY C. MORTON (Vice Chair), Professor and Chair,
Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh
JESSE A. BERLIN, Vice President,
Epidemiology, Johnson & Johnson Pharmaceutical Research and Development, Titusville, NJ
MOHIT BHANDARI, Canada Research Chair in Musculoskeletal Trauma,
McMaster University, Orthopaedic Research Unity, Clarity Research Group, at the Hamilton Health Sciences–General Site (joined October 2009; resigned July 2010)
GISELLE CORBIE-SMITH, Professor,
Social Medicine, Medicine and Epidemiology, Departments of Social Medicine, Medicine and Epidemiology, University of North Carolina–Chapel Hill, Chapel Hill, NC
KAY DICKERSIN, Professor of Epidemiology, Director,
Center for Clinical Trials and U.S. Cochrane Center, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD
JEREMY M. GRIMSHAW, Director,
Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
MARK HELFAND, Director,
Oregon Evidence-Based Practice Center,
Professor of Medicine and Medical Informatics and Clinical Epidemiology,
Oregon Health and Science University, Portland, OR
VINCENT E. KERR, President,
Care Solutions, UnitedHealthcare, Fairfield, CT
MARGUERITE A. KOSTER, Practice Leader,
Technology Assessment and Guidelines Unit, Kaiser Permanente Southern California, Pasadena, CA
KATIE MASLOW, Scholar-in-Residence,
Institute of Medicine, Washington, DC
DAVID A. MRAZEK, Chair,
Department of Psychiatry and Psychology,
Professor of Pediatrics and Psychiatry,
College of Medicine, Mayo Clinic, Rochester, MN
CHRISTOPHER H. SCHMID, Director,
Biostatistics Research Center;
Professor of Medicine,
Tufts Medical Center and Tufts University, Boston, MA
ANNA MARIA SIEGA-RIZ, Professor of Epidemiology and Nutrition, Associate Chair of Epidemiology,
University of North Carolina Gillings School of Global Public Health, Chapel Hill, NC
HAROLD C. SOX, Editor Emeritus,
Annals of Internal Medicine, American College of Physicians of Internal Medicine, Hanover, NH
PAUL WALLACE, Medical Director,
The Permanente Federation, Kaiser Permanente, Oakland, CA
Study Staff
JILL EDEN, Study Director
LAURA LEVIT, Program Officer
LEA BINDER, Research Associate (through May 2010)
MAI LE, Research Assistant (starting September 2010)
ALLISON McFALL, Senior Program Assistant (through August 2010)
JILLIAN LAFFREY, Senior Program Assistant (starting July 2010)
ROGER HERDMAN, Director,
Board on Health Care Services
Consultants
JULIA KREIS, Harkness/Bosch Fellow in Health Care Policy and Practice,
Johns Hopkins Bloomberg School of Public Health, Baltimore, MD
DAVID MOHER, Senior Scientist,
Ottawa Hospital Research Institute (OHRI), Ottawa, Ontario, Canada
ALEXANDER TSERTSVADZE,
Ottawa Hospital Research Institute (OHRI), Ottawa, Ontario, Canada
SALLY HOPEWELL,
National Institute for Health Research, U.K. Cochrane Centre, Oxford, United Kingdom
BRYAN LUCE, Senior Vice President,
Science Policy, United BioSource Corporation, Chevy Chase, MD
RACHAEL FLEURENCE, Senior Research Scientist,
United BioSource Corporation, Chevy Chase, MD
Reviewers
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:
NAOMI ARONSON, Executive Director, Blue Cross and Blue Shield Association, Technology Evaluation Center
VIVIAN COATES, Vice President, Information Services and Technology Assessment, ECRI Institute
JOHN P. A. IOANNIDIS, C.F. Rehnborg Professor in Disease Prevention, Professor of Medicine, and Director, Stanford Prevention Research Center, Stanford University
ERIC B. LARSON, Executive Director, Group Health Research Institute
JOSEPH LAU, Director, Tufts Evidence-based Practice Center, Tufts Medical Center
ART LEVIN, Director, Center for Medical Consumers
ALVIN I. MUSHLIN, Professor and Chairman, Department of Public Health, Weill Medical College, Cornell University
PAUL SHEKELLE, Director, Southern California Evidence-Based Practice Center, The RAND Corporation
RICHARD N. SHIFFMAN, Associate Director, Center for Medical Informatics, Yale School of Medicine
LESLEY STEWART, Director, Centre for Reviews and Dissemination, University of York
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by ENRIQUETA C. BOND of the Burroughs Wellcome Fund, and MARK R. CULLEN of Stanford University. Appointed by the National Research Council and the Institute of Medicine, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Foreword
Knowing what works in health care is of highest importance for patients, healthcare providers, and other decision makers. The most reliable way to identify benefits and harms associated with various treatment options is a systematic review of comparative effectiveness research. Increasingly recognized for their importance, systematic reviews are now being sponsored and conducted by a number of organizations across the United States. When conducted well, a systematic review identifies, appraises, and synthesizes the available body of evidence for a specific clinical question. However, not all of these reviews meet the appropriate standards of quality and methodology. At the request of the U.S. Congress, the Institute of Medicine (IOM) undertook this study to develop a set of standards for conducting systematic reviews of comparative effectiveness research.
The report will have direct implications for implementation of the Patient Protection and Affordable Care Act of 2010. This law established the first nonprofit, public–private Patient-Centered Outcomes Research Institute (PCORI). PCORI will be responsible for setting methodological standards for clinical effectiveness research, including systematic reviews of research findings. I hope this study will support PCORI’s development of standards to ensure that systematic reviews meet a minimum level of objectivity, transparency, and scientific rigor. The IOM study should also help to inform other
public sponsors of systematic reviews of comparative effectiveness research.
To conduct this study, the Institute of Medicine convened a highly qualified committee with diverse backgrounds, ably led by Alfred Berg, chair, and Sally Morton, vice chair. The committee was assisted by dedicated IOM staff led by Jill Eden. This report draws on available evidence, review of expert guidance, and careful consideration of alternative standards according to specified criteria. While this report presents an initial list of standards for improving the quality of publicly funded systematic reviews, it also calls for continued investment in methodological research to identify better practices for future reviews. A companion report establishes standards for developing clinical practice guidelines. I hope these documents will help guide a robust systematic review enterprise for health in the United States.
Harvey V. Fineberg, M.D., Ph.D.
President, Institute of Medicine
February 2011
Preface
Page through any volume of a medical journal from the 1970s and read a clinical review. The authors are likely to be recognized as experts in the field, and the introduction will often open with “we reviewed the world’s medical literature,” moving on to reach clinical conclusions based as much on the experience and opinions of the authors as on the published evidence. Systematic literature searches, critical appraisal, quantitative meta-analysis, and documented pathways linking the evidence to reaching clinical conclusions were virtually unknown. Today’s explicit, scientifically rigorous, transparent, and publicly accountable systematic reviews (SRs) and clinical practice guidelines (CPGs) are the barely recognizable heirs to that earlier convention for giving clinical advice.
Enormous progress has been made by a large and growing international community of clinicians, methodologists, statisticians, and other stakeholders in developing SRs and CPGs, yet problems remain. There are many competing systems for evaluating and synthesizing evidence, and there are no internationally agreed-upon standards for how to conduct an SR or create a CPG. In the United States, the decades-old interest in SRs and CPGs among public and private agencies is receiving a boost from the highlighting of the importance of both in debates about healthcare reform; a specific provision in the Medicare Improvements for Patients and Providers Act of 2008 brought two Institute of Medicine (IOM) committees into being, aimed at setting standards for SRs and CPGs. Furthermore, in the United States there is enormous interest in and high expecta-
tions for the newly created Patient-Centered Outcomes Research Institute, whose authorizing legislation specifically names SRs and CPGs as important components in developing a national program of comparative effectiveness research.
As both SR and CPG reports indicate, the term “standard” is problematic. Our two committees found a sparse evidence base that directly evaluates alternative approaches to SRs and CPGs. The SR committee thus relied on available literature, expert guidance from organizations engaged in SRs, and its own criteria and internal discussions to propose a set of standards, recognizing that any such recommendations must be considered provisional pending further development of the evidence. Collectively the standards set a high bar that will be difficult to achieve for many SRs, yet the evidence and experience are not reassuring that it is safe to cut corners if resources are limited. The standards will be especially valuable for SRs of high-stakes clinical questions with broad population impact, where the use of public funds to get the right answer justifies careful attention to the rigor with which the SR is conducted. The best practices collected in this report should be thoughtfully considered by anyone conducting an SR. In the end the most important standard is to be transparent in reporting what was done and why. Importantly, the committee concludes with recommendations that the United States invest in a program to improve both the science of SRs (with attention to both scientific rigor and feasibility/cost) and the environment that supports them, including a process to update standards as the evidence improves.
Finally, one of the most professionally satisfying benefits of leading an IOM committee is the opportunity to work with committee members with an amazing breadth and depth of experience, and IOM staff whose anticipation and completion of the next steps always appears effortless. We are deeply grateful that this committee and staff have again demonstrated the process at its best.
In conclusion, the committee believes we are at an important juncture in the development of SRs and CPGs, and that timely investment in both will produce an excellent return in improving health care and patient outcomes. We hope our recommended standards will serve as a useful milestone as the United States joins international partners to advance the science and improve the environment for SRs and CPGs.
Alfred O. Berg, Chair
Sally C. Morton, Vice Chair
Committee on Standards for Systematic Reviews of Comparative Effectiveness Research
Acknowledgments
The committee and staff are indebted to a number of individuals and organizations for their contributions to this report. The following individuals provided testimony to the committee:
Naomi Aronson, Executive Director, Technology Evaluation Center, Blue Cross and Blue Shield Association
Kalipso Chalkidou, Director, NICE International, National Institute for Health and Clinical Excellence
Yngve Falck-Ytter, Assistant Professor of Medicine, Case Western Reserve University; Director of Hepatology, VA Medical Center, Cleveland
Rebekah Gee, Assistant Professor of Clinical Medicine, Department of Obstetrics and Gynecology, Tulane University, American College of Obstetricians and Gynecologists
Louis B. Jacques, Director, Coverage & Analysis Group, Office of Clinical Standards & Quality, Centers for Medicare & Medicaid Services
Sandra Zelman Lewis, Assistant Vice President, Health & Science Policy, American College of Chest Physicians
Virginia Moyer, Section Head, Academic General Pediatrics, Baylor College of Medicine, American Academy of Pediatrics
Edmund Pezalla, National Medical Director and Chief Clinical Officer, Aetna Pharmacy Management
Alan Rosenberg, Vice President of Technology Assessment, Wellpoint Health Networks
Carol Sakala, Director of Programs, Childbirth Connection
Gail Shearer, Former Director, Consumer Reports Best Buy Drugs, and Former Director, Health Policy Analysis, Consumers Union
David Shern, President and CEO, Mental Health America
David B. Wilson, Crime and Justice Group Cochair, Steering Committee, The Campbell Collaboration
We also extend special thanks to the following individuals who were essential sources of information, generously giving their time and knowledge to further the committee’s efforts:
Stephanie Chang, EPC Program Director, AHRQ
Julian Higgins, Senior Statistician, MRC Biostatistics Unit, Institute of Public Health, University of Cambridge
Alison Little, Medical Director, Drug Effectiveness Review Project
Rose Relevo, Research Librarian, AHRQ Scientific Resource Center
Karen M. Schoelles, Director, ECRI Institute Evidence-based Practice Center
Lesley Stewart, Director, Centre for Reviews and Dissemination
Funding for this study was provided by the Agency for Healthcare Research and Quality (AHRQ). The committee appreciates the opportunity and support extended by AHRQ for the development of this report.
Finally, many within the Institute of Medicine were helpful to the study staff. We especially would like to thank the staff of the Clinical Practice Guidelines We Can Trust report for their collaborative spirit: Robin Graham, Michelle Mancher, and Dianne Wolman. Additionally, the staff would like to thank Clyde Behney, Michelle O. Crosby-Nagy, Greta Gorman, Cheryl Levey, William McLeod, Abbey Meltzer, Vilija Teel, and Lauren Tobias
Boxes, Figures, and Tables
Summary
Boxes
S-1 |
Charge to the Committee on Standards for Systematic Reviews of Comparative Effectiveness Research, |
|||
S-2 |
Recommended Standards for Initiating a Systematic Review, |
|||
S-3 |
Recommended Standards for Finding and Assessing Individual Studies, |
|||
S-4 |
Recommended Standards for Synthesizing the Body of Evidence, |
|||
S-5 |
Recommended Standards for Reporting Systematic Reviews, |
Chapter 1
Boxes
1-1 |
Charge to the Committee on Standards for Systematic Reviews of Comparative Effectiveness Research, |
|||
1-2 |
Types of Comparative Effectiveness Research Studies, |
|||
1-3 |
Four Examples of the Use of Observational Studies in Systematic Reviews of Comparative Effectiveness Research, |
|||
1-4 |
Charge to the Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, |
|||
1-5 |
Numbering System for the Committee’s Recommended Systematic Review Standards, |
Tables
1-1 |
Statutory Requirements for the Patient-Centered Outcomes Research Institute, |
|||
1-2 |
Committee Criteria for Assessing Potential Standards and Elements for Systematic Reviews, |
|||
1-3 |
Examples of U.S. Governmental Organizations That Produce Systematic Reviews, |
|||
1-4 |
Examples of Private U.S. Organizations That Produce Systematic Reviews, |
|||
1-5 |
Examples of International Organizations That Produce Systematic Reviews, |
|||
1-6 |
Examples of Organizations That Use Systematic Reviews, |
Chapter 2
Boxes
2-1 |
Recommended Standards for Initiating a Systematic Review, |
|||
2-2 |
International Committee of Medical Journal Editors Types of Conflict-of-Interest Disclosures, |
|||
2-3 |
Organizations and Groups Included in the Commissioned Paper on Consumer Involvement in Systematic Reviews, |
|||
2-4 |
Examples of Interaction Between Systematic Review (SR) and Clinical Practice Guideline (CPG) Teams, |
Figure
2-1 |
Analytic framework for a new enteral supplement to heal bedsores, |
Tables
2-1 |
Models of Interaction Between Systematic Review (SR) and Clinical Practice Guideline (CPG) Teams, |
|||
2-2 |
PICO Format for Formulating an Evidence Question, |
Chapter 3
Boxes
3-4 |
Grey-Literature Databases, |
|||
3-5 |
Study Selection Criteria for a Systematic Review of Therapies for Children with Autism Spectrum Disorders (ASD), |
|||
3-6 |
Types of Data Extracted from Individual Studies, |
|||
3-7 |
Cochrane Back Pain Group Criteria for Internal Validity of Randomized Trials of Back Pain, |
|||
3-8 |
Cochrane Risk of Bias Tool Domains, |
|||
3-9 |
USPSTF Criteria for Grading the Internal Validity of Individual Studies (RCTs and Cohort Studies), |
Figure
3-1 |
Example of a flow chart, |
Tables
3-1 |
Types of Reporting Biases, |
|||
3-2 |
Expert Suggestions for Conducting the Search Process and Addressing Reporting Bias, |
|||
3-3 |
Expert Suggestions for Screening Publications and Extracting Data from Eligible Studies, |
|||
3-4 |
Types of Bias in Individual Studies, |
|||
3-5 |
Characteristics of Individual Studies That May Affect Applicability, |
Chapter 4
Boxes
4-1 |
Recommended Standards for Synthesizing the Body of Evidence, |
|||
4-2 |
Key Concepts Used in the GRADE Approach to Assessing the Quality of a Body of Evidence, |
|||
4-3 |
Key Concepts Used in the AHRQ Approach to Assessing the Quality of a Body of Evidence, |
|||
4-4 |
Research Trends in Meta-Analysis, |
Figure
4-1 |
Forest plot, |
Tables
Chapter 5
Boxes
5-1 |
Recommended Standards for Reporting Systematic Reviews, |
|||
5-2 |
Requirements for Research Funded by the Patient-Centered Outcomes Research Institute, |
|||
5-3 |
A History of Reporting Guidelines for Comparative Effectiveness Research, |
|||
5-4 |
Example of a Structured Abstract: Clinical Utility of Cancer Family History Collection in Primary Care, |
|||
5-5 |
Example of a Plain-Language Summary: Antenatal Corticosteroids for Accelerating Fetal Lung Maturation for Women at Risk of Preterm Birth, |
|||
5-6 |
Reporting Funding and Conflict of Interest: Selected Examples, |
Tables