KNOWING WHAT WORKS IN HEALTH CARE

A ROADMAP FOR THE NATION

Committee on Reviewing Evidence to Identify Highly Effective Clinical Services

Board on Health Care Services

Jill Eden, Ben Wheatley, Barbara McNeil, and Harold Sox, Editors

INSTITUTE OF MEDICINE OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS

Washington, D.C.
www.nap.edu



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
KNOWING WHAT WORKS IN HEALTH CARE A ROADMAP FOR THE NATION Committee on Reviewing Evidence to Identify Highly Effective Clinical Services Board on Health Care Services Jill Eden, Ben Wheatley, Barbara McNeil, and Harold Sox, Editors

OCR for page R1
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by Grant No. 56822 between the National Academy of Sci- ences and the Robert Wood Johnson Foundation. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project. Library of Congress Cataloging-in-Publication Data Knowing what works in health care : a roadmap for the nation / Committee on Reviewing Evidence to Identify Highly Effective Clinical Services, Board on Health Care Services ; Jill Eden ... [et al.], editors. p. ; cm. Includes bibliographical references. ISBN 978-0-309-11356-4 (hardcover) 1. Medical care--Standards—United States. 2. Medical care—United States—Quality control. 3. Evidence-based medicine—United States. I. Eden, Jill. II. Institute of Medicine (U.S.). Committee on Reviewing Evidence to Identify Highly Effective Clinical Services. [DNLM: 1. Quality Assurance, Health Care—standards—United States. 2. Evidence- Based Medicine—standards—United States. 3. Organizational Innovation—United States. 4. Practice Guidelines as Topic—standards—United States. W 84 AA1 K73 2008] RA399.A3K56 2008 362.1--dc22 2008008578 Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu. For more information about the Institute of Medicine, visit the IOM home page at: www.iom.edu. Copyright 2008 by the National Academy of Sciences. All rights reserved. Printed in the United States of America. The serpent has been a symbol of long life, healing, and knowledge among almost all cultures and religions since the beginning of recorded history. The serpent adopted as a logotype by the Institute of Medicine is a relief carving from ancient Greece, now held by the Staatliche Museen in Berlin. Suggested citation: Institute of Medicine (IOM). 2008. Knowing what works in health care: A roadmap for the nation. Washington, DC: The National Academies Press.

OCR for page R1
“Knowing is not enough; we must apply. Willing is not enough; we must do.” —Goethe Advising the Nation. Improving Health.

OCR for page R1
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Acad- emy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding en- gineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineer- ing programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is presi- dent of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Insti- tute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sci- ences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Coun- cil is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1
COMMITTEE ON REVIEWING EVIDENCE TO IDENTIFY HIGHLY EFFECTIVE CLINICAL SERVICES BARBARA J. MCNEIL (Chair), Ridley Watts Professor and Head of the Department of Health Care Policy, Harvard Medical School; Professor of Radiology, Harvard Medical School and Brigham and Women’s Hospital, Boston, Massachusetts HAROLD C. SOX (Vice Chair), Editor, Annals of Internal Medicine, American College of Physicians, Philadelphia, Pennsylvania ALLEN DANIELS, Chief Executive Officer, Alliance Behavioral Care, and Professor of Clinical Psychiatry, University of Cincinnati, Ohio KAY DICKERSIN, Director of the Center for Clinical Trials, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland ROBERT S. GALVIN, Director, Global Health Care, General Electric Company, Fairfield, Connecticut DANA P. GOLDMAN, Chair and Director of Health Economics, Finance, and Organization, RAND Corporation, Santa Monica, California RICHARD A. JUSTMAN, National Medical Director, UnitedHealthcare, Edina, Minnesota ARTHUR A. LEVIN, Director, Center for Medical Consumers, New York RICHARD E. MARSHALL, Medical Director for Research and Pediatrician, Harvard Vanguard Medical Associates, Boston, Massachusetts WILHELMINE MILLER, Associate Research Professor, George Washington University School of Public Health and Health Services, Washington, District of Columbia SALLY C. MORTON, Vice President for Statistics and Epidemiology, RTI International, Research Triangle Park, North Carolina SAMUEL R. NUSSBAUM, Executive Vice President and Chief Medical Officer, WellPoint, Inc., Indianapolis, Indiana DIANA B. PETITTI, Adjunct Professor, Department of Preventive Medicine, University of Southern California, Keck School of Medicine, Los Angeles, California STEVEN SHAK, Chief Medical Officer, Genomic Health, Inc., Redwood City, California LISA SIMPSON, Professor and Director, Child Policy Research Center, Cincinnati Children’s Hospital Medical Center and the University of Cincinnati Department of Pediatrics, Cincinnati, Ohio GLENN D. STEELE, President and Chief Executive Officer, Geisinger Health System, Danville, Pennsylvania 

OCR for page R1
Study Staff JILL EDEN, Study Director ROGER HERDMAN, Director, Board on Health Care Services (starting October 2007) MICHELE ORZA, Acting Director, Board on Health Care Services (December 2006 to October 2007) TRACY HARRIS, Program Officer (through December 2006) BEN WHEATLEY, Program Officer (through September 2007) SAMANTHA CHAO, Senior Health Policy Associate (through August 2006) LEA GREENSTEIN, Research Associate (starting February 2007) DANITZA VALDIVIA, Program Associate (through September 2007) i

OCR for page R1
Reviewers This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with proce- dures approved by the National Research Council’s (NRC’s) Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: STEVEN FINDLAY, Consumers Union R. BRIAN HAYNES, Department of Clinical Epidemiology and Biostatistics, McMaster University KATHLEEN McCORMICK, SAIC: Health Solutions CYNTHIA MULROW, Annals of Internal Medicine MARY O’NEIL MUNDINGER, School of Nursing, Columbia University PETER NEUMANN, Center for the Evaluation of Value & Risk in Health, Tufts-New England Medical Center SANDRA SCHNEIDER, Department of Emergency Medicine, University of Rochester CARY S. SENNETT, American Board of Internal Medicine PAUL SHEKELLE, Southern California Evidence-Based Practice Center at The RAND Corporation ii

OCR for page R1
iii KNOWING WHAT WORKS IN HEALTH CARE SHOSHANNA SOFAER, School of Public Affairs, Baruch College STEVEN TEUTSCH, Outcomes Research & Management, Merck & Co., Inc. PAUL WALLACE, Health and Productivity Management Programs and The Care Management Institute and KP-Healthy Solutions, The Permanente Federation, Kaiser Permanente Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclu- sions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by SHELDON GREENFIELD, Center for Health Policy Research, University of California, Irvine, and JOHANNA T. DWYER, Tufts University School of Medicine & Friedman School of Nutrition Science, Tufts-New England Medical Center. Appointed by the NRC and the Institute of Medicine, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.

OCR for page R1
Preface The United States has the most expensive health care in the world by a large margin. However, by many measures of the health of the public, the United States ranks well down the list of nations. How can we understand this paradox? The wide regional variation in practice style implies that our knowledge about effective health care is weak enough to support a wide range of accepted practice. Since health care outcomes are the same in high- and low-intensity regions, a lean style of practice is safe and an extravagant style is wasteful. The regional variation story offers further hints about a way out of this problem. Variation is very low for some practices (e.g., coronary bypass surgery or surgery for fractured hip), which implies secure knowledge and strong consensus. Regional variation is very high for other practices (e.g., MRI and CT scans, ICU admissions in the last six months of life, referral to a specialist), which implies weak knowledge and no consensus. Taken as a whole, the evidence implies that better knowledge could lead to a stronger consensus, less regional variation, and probably lower costs. In short, we need better knowledge of which health care services are the most effective and which patients are most likely to benefit from them. Concern about the cost of health care has grown in the past 20 years, and organizations that pay for health care have sought to obtain trust- worthy information about what works in the practice of medicine. Payers, government, health care delivery systems, and professional organizations have taken the lead in efforts to develop standards of care. The result has been movement in the right direction, but also chaos. The positive features of these efforts include steady movement away from sole reliance on expert ix

OCR for page R1
x PREFACE opinion and toward scientific, systematic reviews of the pertinent medical literature and increasing recognition that we need a common language for rating the evidence. The negative features are those inherent in a pluralistic, uncoordinated health care system: large-scale duplication of effort, wide variation in process, far too little attention to avoiding conflict of interest, and lack of standards. We must build on the developing strengths of the present system as we correct these problems. In the past several years, people have begun to talk about impos- ing order on the system for identifying effective health services. Many people—ranging from health care experts to payers to presidential candi- dates—have proposed a national organization to identify the most effective health care services. Somewhat in advance of these proposals, the Robert Wood Johnson Foundation asked the Institute of Medicine (IOM) to con- vene a committee to recommend methods to identify highly effective health care services. The confluence of these two developments creates what the committee hopes will be a useful contribution to the emerging consensus that the United States needs a more systematic approach to evaluating the evidence for clinical effectiveness. The IOM committee has focused on specifying the principles underly- ing the methods to accomplish three crucial tasks for a national system for identifying highly effective health services: priority setting, evidence review, and development of recommendations. We believe that this report would serve to guide an organization tasked with putting a working system into place. In effect, it would be the starting point for a detailed manual of operations for a new organization. In accord with its charge, the commit- tee did not make recommendations about funding for clinical effectiveness research or the institutional home of a national organization for clinical effectiveness. We expect considerable debate about the committee’s recommendation about the structure of this organization. The committee proposes a hybrid structure that exerts control over the processes of setting priorities for which services to evaluate and conducting evidence reviews on the high- priority topics. For the last step—the development of clinical recommen- dations—the committee proposes to use the nation’s existing capacity for developing practice guidelines and insurance coverage policy. The commit- tee proposes standards to guide these organizations in making clinical rec- ommendations and strongly recommends that organizations preferentially use recommendations that are developed according to these standards. For all three tasks, the committee has specific recommendations about minimiz- ing bias due to conflict of interest. The committee wrote this report for several audiences. One will de- cide how to allocate resources for a national clinical effectiveness assess- ment system. Among these are members of the U.S. Congress and private

OCR for page R1
xi PREFACE organizations that would benefit from a national clinical effectiveness as- sessment program. Another audience consists of the organizations that would use the evidence that the new system would produce: payers, health insurance companies, and health care delivery systems. A third audience is the organizations that develop recommendations that will shape practice measures, practice guidelines, and insurance coverage policy. Finally, we hope that members of the general public—the ultimate beneficiaries of the committee’s work—will read the report and support efforts to move in the directions proposed in this report. The IOM chose committee members who—individually and collec- tively—have the expertise to make credible proposals. Among its members are medical directors of large health insurance companies, health care delivery systems, and companies. The committee also includes physicians with experience in evidence-based guideline programs, experts on extract- ing evidence from the medical literature, and experienced advocates for the public interest. The breadth of interests represented on the committee is the best guarantee that its recommendations would meet the needs of a diverse community of interest. The committee developed a common vision early in its deliberations, and it speaks with one voice in this report. In a series of workshops, the committee listened to an array of experts who kindly donated their time to help the committee. Above all, the committee had a remarkable group of IOM staff members who supported the committee’s efforts and kept the project moving forward. To all, we give thanks. Barbara J. McNeil, Chair Harold C. Sox, Vice Chair

OCR for page R1

OCR for page R1
Acknowledgments The committee and staff are indebted to a number of individuals and organizations for their contributions to this report. The following individu- als testified before the committee during public meetings or workshops: Marilyn Albert, Johns Hopkins University School of Medicine, Division of Cognitive Neuroscience Naomi Aronson, Blue Cross and Blue Shield Association Technology Evaluation Center Mary Barton, U.S. Preventive Services Task Force Barry Berger, Exact Sciences Corporation Kathy Buto, Johnson & Johnson, Health Policy Daniel Cain, Cain Brothers Carolyn Clancy, Agency for Healthcare Research and Quality Vivian Coates, ECRI Institute Janet Corrigan, National Quality Forum Steven Findlay, Consumers Union Ray Gibbons, American Heart Association Richard Goldberg, University of North Carolina, Chapel Hill, School of Medicine Division of Hematology & Oncology Winifred Hayes, Hayes, Inc. Clarion Johnson, ExxonMobil, Global Medicine and Population Health Peter Juhn, Johnson & Johnson, Health Policy and Evidence Michael Maciosek, HealthPartners Foundation Daniel Martin, Emory University School of Medicine xiii

OCR for page R1
xi ACKNOWLEDGMENTS David Matchar, Duke University Medical School, Center for Clinical Health Policy Research Susan Molchan, Alzheimer’s Disease Neuroimaging Initiative, National Institute on Aging Cynthia Mulrow, University of Texas/American College of Physicians Dennis O’Leary, Joint Commission Gregory Pawlson, National Committee for Quality Assurance Stephen Phurrough, Centers for Medicare & Medicaid Services, Coverage and Analysis Group Margaret Piper, Blue Cross and Blue Shield Association Technology Evaluation Center Atiqur Rahman, Food and Drug Administration, Center for Drug Evaluation and Research David Ransohoff, Lineberger Comprehensive Cancer Center, University of North Carolina, Chapel Hill Reginald Sanders, American Society of Retina Specialists Cary Sennett, AMA-Convened Physician Consortium for Performance Improvement Jean Slutsky, Agency for Healthcare Research and Quality Arthur Small, Genentech Earl Steinberg, Resolution Health Sean Tunis, Center for Medical Technology Policy Jim Weinstein, Dartmouth Hitchcock Medical Center We also extend special thanks to the following individuals who were es- sential sources of information, generously giving their time and knowledge to further the committee’s efforts. Michael A. Stoto, Georgetown University, and Perry W. Payne, Jr., George Washington University, drafted important background papers. Kerry Kemp provided valuable editorial assistance. The following individuals answered many inquiries and were patient with staff’s many requests for information: Mary Barton, U.S. Preventive Ser- vices Task Force; Kristen K. Bronner, The Dartmouth Atlas; Vivian Coates, ECRI Institute; Winifred Hayes and Wendy Schneider, Hayes, Inc.; Mark Helfand, Oregon Health & Science University, Evidence-based Practice Center; Marguerite Koster, Medical Technology Assessment & Guidelines, Kaiser Permanente Southern California; Alison S. Little, Oregon Health & Science University, Drug Effectiveness Review Project; Robert McDonough, Technology Assessment and Clinical Guidelines, Aetna/US Healthcare; Peter Neumann and Jenny Palmer, Tufts-New England Medical Center, Center for the Evaluation of Value and Risk in Health; Chuck Phelps, University of Rochester; Richard N. Shiffman, Yale University Center for Medical

OCR for page R1
x ACKNOWLEDGMENTS Informatics; Jean Slutsky, AHRQ Effective Health Care Program; and Gail Wilensky, Project Hope. Funding for this study was provided by the Robert Wood Johnson Foundation (RWJF). The committee appreciates the opportunity and sup- port extended by RWJF for the development of this report. Finally, many within the IOM were helpful to the study staff. The staff would especially like to thank Lara Andersen, Clyde Behney, Michelle Bruno, Evalyne Bryant-Ward, Abbey Burchman, Bethany Hardy, Bronwyn Schrecker Jamrok, William McLeod, and Janice Mehler.

OCR for page R1

OCR for page R1
Contents SUMMARY 1 1 INTRODUCTION 17 Study Scope, 18 Study Methods, 20 Context for This Report, 21 Orientation to the Organization of the Report, 25 2 AN IMPERATIVE FOR CHANGE 33 Background, 34 Persistent Health Policy Challenges, 36 Current Landscape, 41 3 SETTING PRIORITIES FOR EVIDENCE ASSESSMENT 57 Background, 58 Findings, 72 Recommendations, 73 4 SYSTEMATIC REVIEWS: THE CENTRAL LINK BETWEEN EVIDENCE AND CLINICAL DECISION MAKING 81 Background, 82 Fundamentals of a Systematic Review, 87 Recommendations, 108 Other Program Challenges, 111 xii

OCR for page R1
xiii CONTENTS 5 DEVELOPING TRUSTED CLINICAL PRACTICE GUIDELINES 121 Background, 122 Current Landscape, 125 Critical Program Challenges and Recommendations, 139 6 BUILDING A FOUNDATION FOR KNOWING WHAT WORKS IN HEALTH CARE 153 Guiding Principles, 154 Building the Program’s Foundation, 160 Recommendations for Building the Program Infrastructure, 170 Unanswered Questions, 173 APPENDIXES A Acronyms and Abbreviations 179 B Workshop Agendas and Questions to Panelists 181 C Template for Submissions of Topics to the Agency for Healthcare Research and Quality 193 D Standards for Reporting Meta-Analyses of Clinical Trials and Observational Studies: QUOROM and MOOSE 199 E Examples of ECRI Institute and Hayes, Inc., Quick Turnaround Reports 205 F Guideline Standards: The AGREE Instrument and COGS Checklist 231 G Committee Biographies 237 INDEX 247

OCR for page R1
List of Boxes, Figures, and Tables Summary Boxes S-1 Charge to the IOM Committee, 2 S-2 Recommendations, 10 S-3 Program Principles, 12 Figure S-1 Continuum from research studies to systematic review to development of clinical guidelines and recommendations, 4 Chapter 1 Boxes 1-1 Selected Terms Used in the Report, 24 1-2 Selected Milestones in U.S. Efforts to Identify Effective Health Care Services, 26 Figure 1-1 Continuum from research studies to systematic review to development of clinical guidelines and recommendations, 23 xix

OCR for page R1
xx LIST OF BOXES, FIGURES, AND TABLES Chapter 2 Figures 2-1 Total federal spending for Medicare and Medicaid under assumptions about the health care cost growth differential, 38 2-2 Medicare spending per capita in the United States, by hospital referral region, 2003, 39 Tables 2-1 Examples of Organizations That Conduct Evidence Syntheses, 44 2-2 Public-Sector Activities That Use Evidence Syntheses, 45 2-3 Examples of Organizations That Establish Clinical Guidelines and Recommendations, 46 2-4 Examples of Organizations That Measure Performance, 48 Chapter 3 Boxes 3-1 Sources of Topic Nominations, Evidence-based Practice Centers, 2005-2006, 65 3-2 Topics of Evidence-based Practice Center Reports, 2005 to present, 66 3-3 Examples of Widely Adopted Health Interventions Found to Be Ineffective or Harmful, 68 Tables 3-1 Context for Setting Priorities for Evidence Assessment, 2006, 60 3-2 Methods Used to Identify Topics for Systematic Reviews in Selected Organizations, 62 3-3 AHRQ EPC Study Nominations by Source and Topic Area, 2005-2006, 64 3-4 Duplicated Efforts by Selected Health Plans and Technology Assessment Firms, 2006, 69 3-5 Priority Setting Criteria That Selected Organizations Use, 71 3-6 Definitions of Commonly Used Priority Setting Criteria, 72 3-7 Principles for Setting Evidence Assessment Priorities, 75 Chapter 4 Boxes 4-1 Medicare National Coverage Decisions with Poor Evidence, 95 4-2 Prevention Topics with Insufficient Evidence for One or More Population Subgroups, 96

OCR for page R1
xxi LIST OF BOXES, FIGURES, AND TABLES 4-3 Sources of Bias in Individual Studies and Systematic Reviews, 98 4-4 Key Concepts in Appraising Evidence, 101 4-5 Unresolved Methodological Issues in Conducting Systematic Reviews, 110 Figure 4-1 Analytic framework used by the U.S. Preventive Services Task Force, 89 Tables 4-1 Key U.S. Producers and Users of Systematic Reviews, 84 4-2 PICO Format for Formulating an Evidence Question, 88 4-3 Matching the Clinical Question with the Appropriate Evidence, 92 4-4 Quality of Evidence for Technologies Subject to Medicare National Coverage Determinations, 1998-2003, 94 4-5 Selected Examples of Evidence Hierarchies for Three Cardiology Interventions, 103 4-6 Comparison of Reporting Standards in CONSORT, QUOROM, and MOOSE, 106 4-7 Use of Reporting Standards in Leading Biomedical Journals, 107 Chapter 5 Boxes 5-1 Conflicting Guidelines for the Treatment of Epilepsy, 133 5-2 Potential Sources of Bias in Clinical Practice Guidelines, 143 Tables 5-1 USPSTF Strength of Recommendations, 128 5-2 Dueling Evidence Hierarchies and Recommendation Grades in Cardiology, 136 5-3 Measures to Address Conflicts of Interest, 144 Chapter 6 Boxes 6-1 Program Principles, 156 6-2 Committee Recommendations, 172 Tables 6-1 Focus of Selected National Efforts to Identify Effective Health Care Services, 161

OCR for page R1
xxii LIST OF BOXES, FIGURES, AND TABLES 6-2 Key Features of National Clinical Effectiveness Programs in Australia, Canada, and England and Wales, 162 6-3 Alternative Approaches to Organizing the Program: Administrative Structure and Primary Functions, 164 6-4 Summary Assessment of Organizational Alternatives Based on Committee Principles, 167