Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page R1
--> Health Performance Measurement in the Public Sector Principles and Policies for Implementing an Information Network Edward B. Perrin, Jane S. Durch, and Susan M. Skillman, Editors Panel on Performance Measures and Data for Public Health Performance Partnership Grants Committee on National Statistics Commission on Behavioral and Social Sciences and Education National Research Council NATIONAL ACADEMY PRESS Washington, D.C. 1999
OCR for page R2
--> NATIONAL ACADEMY PRESS 2101 Constitution Avenue, N.W. Washington, D.C. 20418 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by Contract No. 282-95-0034 between the National Academy of Sciences and the U.S. Department of Health and Human Services. Support of the work of the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (No. SBR-9709489). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project. Library of Congress Cataloging-in-Publication Data Health performance measurement in the public sector : principles and policies for implementing an information network / Edward B. Perrin, Jane S. Durch, and Susan M. Skillman, editors ; Panel on Performance Measures and Data for Public Health Performance Partnership Grants, Committee on National Statistics, Commission on Behavioral and Social Sciences and Education, National Research Council. p. cm. Includes bibliographical references (p. ). ISBN 0-309-06436-8 (pbk.) 1. Public health—United States—Information services. 2. Public health—United States—Evaluation—Data processing. I. Perrin, Edward. II. Durch, Jane. III. Skillman, Susan M. IV. National Research Council (U.S.). Panel on Performance Measures and Data for Public Health Performance Partnership Grants. RA423.2 .H43 1999 362.1'0973—dc21 99-6159 Additional copies of this report are available from National Academy Press, 2101 Constitution Avenue, N.W., Box 285, Washington, D.C. 20055. Call (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area). This report is also available on line at http://www.nap.edu Printed in the United States of America Copyright 1999 by the National Academy of Sciences. All rights reserved.
OCR for page R3
--> PANEL ON PERFORMANCE MEASURES AND DATA FOR PUBLIC HEALTH PERFORMANCE PARTNERSHIP GRANTS EDWARD B. PERRIN (Chair), Department of Health Services, School of Public Health and Community Medicine, University of Washington JOHN C. BAILAR III, Department of Health Studies, University of Chicago THOMAS J. BARRETT, Mental Health Services, Colorado Department of Human Services DAVID W. FLEMING, Center for Disease Prevention and Epidemiology, Oregon Health Division V. JOSEPH HOTZ, Departments of Economics and Policy Studies, University of California, Los Angeles IRA R. KAUFMAN, University of Medicine and Dentistry of New Jersey, Department of Environmental and Community Medicine, Robert Wood Johnson Medical School JOHN R. LUMPKIN, Illinois Department of Public Health WILLIAM A. MORRILL, Mathtech, Inc., Princeton, New Jersey R. HEATHER PALMER, Center for Quality of Care Research and Education, Harvard School of Public Health J. SANFORD SCHWARTZ, School of Medicine and the Wharton School, University of Pennsylvania MARY E. STUART, Maryland Collaboratory for Business and Health, Department of Sociology and Anthropology, University of Maryland, Baltimore County CYNTHIA P. TURNURE, Minnesota Department of Health PAUL J. WIESNER, DeKalb County Board of Health, Decatur, Georgia GAIL R. WILENSKY, Project Hope, Bethesda, Maryland JANE S. DURCH, Study Director JEFFREY J. KOSHEL, Study Director (through December 1997) SUSAN M. SKILLMAN, Senior Research Associate TELISSIA M. THOMPSON, Senior Project Assistant
OCR for page R4
--> COMMITTEE ON NATIONAL STATISTICS 1998–1999 JOHN E. ROLPH (Chair), Department of Information and Operations Management, University of Southern California JOSEPH G. ALTONJI, Institute for Research on Poverty and Department of Economics, Northwestern University JULIE DaVANZO, RAND, Santa Monica, California WILLIAM F. EDDY, Department of Statistics, Carnegie Mellon University WILLIAM KALSBEEK, Department of Biostatistics, University of North Carolina RODERICK J.A. LITTLE, School of Public Health, University of Michigan THOMAS A. LOUIS, School of Public Health, University of Minnesota CHARLES F. MANSKI, Department of Economics, Northwestern University WILLIAM NORDHAUS, Department of Economics, Yale University JANET L. NORWOOD, Urban Institute, Washington, D.C. EDWARD B. PERRIN, Department of Health Services, School of Public Health and Community Medicine, University of Washington PAUL R. ROSENBAUM, Department of Statistics, Wharton School, University of Pennsylvania FRANCISCO J. SAMANIEGO, Division of Statistics, University of California, Davis RICHARD L. SCHMALENSEE, Sloan School of Management, Massachusetts Institute of Technology MIRON L. STRAF, Director ANDREW A. WHITE, Deputy Director
OCR for page R5
--> Contents Preface ix Executive Summary 1 1 Introduction 15 Origins of the Study 16 Performance Partnership Grants 16 Other Influences 17 Charge to the Panel 18 Essential Definitions 19 Performance Measurement and Related Concepts 19 Categories of Performance Measures 20 Phase I: Focus on Selection of Performance Measures 22 Use of Measures of Process and Capacity as Well as Outcomes 22 Guidelines for Selecting Performance Measures 24 Limitations of a Program-Specific Approach to Performance Measurement 25 Need to Strengthen State and Local Capacity for Data Collection and Analysis 26 Inadvisability of Using Performance Measures Alone for Resource Allocation Purposes 26 Phase II: Data and Information System Development to Support Performance Measurement 27 A Vision for a National Health Information Network 27 Critical Issues 28 Structure of the Report 29 References 29
OCR for page R6
--> 2 Putting Performance Measurement in Context 30 Use of Performance Measurement in Accountability Systems 30 Performance-Based Accountability Systems 31 Operation of Performance-Based Systems 33 Applications of Performance Measurement 37 Examples of Performance Monitoring and Accountability Systems 40 Government Performance and Results Act 41 Use of Performance Measures in the Maternal and Child Health Services Block Grant Program 42 Reporting Requirements for the Temporary Assistance for Needy Families Program 44 State Developments in Performance-Based Budgeting 45 Health Care Performance Measurement in the Private Sector 48 Conclusions 52 References 55 3 Performance Measurement Considerations for Publicly Funded Health Programs 59 Broad Array of Health-Related Services and Service Relationships 60 Measurement Considerations for Population-Based Health Services 61 Population Health Services 61 Monitoring Population Health Services 62 Monitoring the Infrastructure for Publicly Funded Health Programs 64 Some Performance Measurement Considerations Related to Program-Specific Matters 66 Environmental Health Programs 66 Mental Health Programs 68 Substance Abuse Programs 71 Process Guidelines as a Basis for Performance Measurement 74 Guidelines for Personal Health Services 75 Guidelines for Population-Based Health Services 76 Research Needs for Practice Guidelines and Performance Measurement 76 Conclusions 78 References 79 4 Data and Information Systems: Issues for Performance Measurement 83 Health Data Resources 84 Registries 85 Surveys 86 Patient Records and Related Clinical Encounter Data 87 Administrative Data 89 Claims Data 90 Linkage of Data Sets 91 Steps Toward Integration of Data Sets 93
OCR for page R7
--> Assuring the Quality of Data and Data Analysis 95 Random Variation and Bias 96 Data Management 97 Challenges in Data Analysis 98 Developing and Implementing Standards for Data and Data Systems 103 Standardization Activities 103 Balancing Standardization and Change 108 Enhancing Performance Measurement through Advances in Information Technology 109 Data Collection and Transmission Technologies 109 Data Management and Analysis 110 Computer-Based Patient Records 111 The Internet and the World Wide Web 111 Limits of Technology 114 Privacy, Confidentiality, and Security of Health Data 114 Investing in Health Data and Data Systems 116 Data and Information Systems 117 Technology 119 Training and Technical Assistance 120 Taking a Collaborative Approach to the Development of Health Data and Information Systems 122 Meeting the Needs of Many Data Users 123 Collaboration in the Design and Implementation of Data Systems 125 Conclusions 126 References 127 5 Strategies for Supporting Performance Measurement Through a National Information Network 132 Conclusions Regarding Performance Partnership Agreements 133 A National Information Network for Health-Related Data 134 Essential Features of a Health Information Network to Serve Different Levels of Decision Making 135 Major Policy Actions Needed 138 National Collaboration 138 Integration of Data Systems 143 Technology Policy 144 Operational Principles 145 Involve a Broad Range of Public Agencies 145 Build on Existing Information Systems 146 Implementation Standards 147 Required Investment for an Information Network 152 Data Systems 152 Training and Technical Assistance 154 A National Research Agenda 156 Final Observations 157 References 158
OCR for page R8
--> Appendices A Potential Health Outcome and Risk Status Measures 163 B Workshop Agenda 168 C Biographical Sketches 170
OCR for page R9
--> Preface This is the second of two reports from the Panel on Performance Measures and Data for Public Health Performance Partnership Grants. In summer 1995, as part of its response to the need to ensure that public funding of health programs is related to documented program performance, the Office of the Assistant Secretary for Health of the U.S. Department of Health and Human Services (DHHS) requested that the National Research Council (NRC) convene an expert panel to examine and report on the technical issues involved in establishing performance measures in ten substantive program areas. Such measures were to be required as part of the proposed Performance Partnership Grants (PPG) Program, under which each state was to negotiate with DHHS an action plan with performance objectives that would be specific in terms of outcomes, processes, and capacity to be achieved within a 3- to 5-year period. The panel divided its work into two phases. The objective of the first phase was to develop a report to the Secretary of DHHS on performance measures in specified areas that would be useful to the PPG Program in its first 3–5 years. The report Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health (National Academy Press, Washington, D.C., 1997) presents the findings and recommendations resulting from the first phase of the panel's work. The focus of the second phase was on recommended improvements to existing local, state, and federal data systems that would facilitate the collection of data for use with existing and proposed performance measures. This two-phase approach was adopted by the panel because of our conclusion that developmental work is needed in both the public and private sectors to adapt, refine, or add to existing data systems to make them more useful in performance measure-
OCR for page R10
--> ment, and to delineate more clearly the complex causal links between program processes and outcomes. Although formal legislation mandating PPGs has not been adopted, various DHHS programs and a growing number of states and communities are moving to monitor and analyze health outcomes on their own. Moreover, there appears to be a growing consensus within the public health, substance abuse, and mental health communities about the value of performance measurement. Indeed, many people believe that the case for increasing, or even maintaining, public funding will depend on documented program performance. The development of performance measures is a continuation of earlier efforts to assess progress toward important public health goals. The broad acceptance of immunization rates and other such measures developed for Healthy People 2000, for example, has been instrumental in the creation of databases and the mobilization of resources in many jurisdictions to assess progress toward those objectives. The work of this panel was furthered significantly by four regional meetings of state officials and consumers convened by DHHS early in the process—in Portland, Oregon; San Francisco, California; Chicago, Illinois; and Philadelphia, Pennsylvania—and by input from several national associations of state agency administrators (the Association of State and Territorial Health Officials, the National Association of County and City Health Officials, the National Association of State Alcohol and Drug Abuse Directors, and the National Association of State Mental Health Program Directors). Our work during the second phase of the project was aided by the contributions of participants in a workshop held by the panel in July 1997. The workshop agenda and participants are listed in Appendix B. We also wish to thank Winnie Mitchell of the Substance Abuse and Mental Health Services Administration and David Moriarty of the Centers for Disease Control and Prevention, who provided special assistance by arranging several meetings with agency staff members. The panel was fortunate to have for its deliberations two specially commissioned working papers. ''Improving Federal-State Data Collection to Monitor Program Performance" was prepared by the Science and Epidemiology Committee of the Association of State and Territorial Chronic Disease Program Directors and the Council of State and Territorial Epidemiologists. Christopher Maylahn, New York State Department of Health, chaired the group that prepared this paper. "Quality Education Data: Unprecedented Opportunity for a Decade to Build" was prepared by Emerson Elliott, National Council for Accreditation of Teacher Education, and John Ralph, National Center for Education Statistics. Copies of these papers are available from the Committee on National Statistics (2101 Constitution Avenue, N.W., Washington, DC 20418). This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institu-
OCR for page R11
--> tion in making the published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following individuals for their participation in the review of this report: Thomas F. Babor, School of Medicine, University of Connecticut Health Center; Maureen M. Henderson, School of Medicine, University of Washington (emeritus); Ed Lichtenstein, Oregon Research Institute, Eugene; Thomas Louis, School of Public Health, University of Minnesota; J. Michael McGinnis, scholar in residence, National Research Council; Barbara J. McNeil, Department of Health Care Policy, Harvard Medical School; Donald M. Steinwachs, School of Hygiene and Public Health, Johns Hopkins University; Bernard Turnock, Community Health Sciences, University of Illinois, Chicago; and Albert W. Wu, Health Services Research Center, Johns Hopkins University. While the individuals listed above have provided constructive comments and suggestions, it must be emphasized that responsibility for the final content of this report rests entirely with the authoring committee and the institution. The panel appreciates the assistance of the staff of the Committee on National Statistics (CNSTAT) in preparing this report. We wish to thank in particular Miron Straf, staff director of CNSTAT, for developing the original project design; Telissia Thompson for organizing panel meetings and carrying out innumerable administrative tasks; and Kathleen Saslaw for providing financial management for the study. Several former CNSTAT staff members provided valuable assistance at various stages in the production of the panel's two reports: Anu Das, Theresa Raphael, Michelle Ruddick, and Ashley Bowers. We appreciate the editorial work of Rona Briere and the guidance of Eugenia Grohman, associate director for reports of the Commission on Behavioral and Social Sciences and Education. Thanks also go to Sue Skillman, University of Washington, for her valued assistance to the committee chair. Finally, the panel extends its special thanks to Jeff Koshel, the panel's study director for the first part of the study, and to Jane Durch, for her very capable management of and contribution to the process leading to the production of this second report. EDWARD B. PERRIN, CHAIR PANEL ON PERFORMANCE MEASURES AND DATA FOR PUBLIC HEALTH PERFORMANCE PARTNERSHIP GRANTS
OCR for page R12
--> The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council.
OCR for page R13
Health Performance Measurement in the Public Sector
OCR for page R14
This page in the original is blank.