Health Performance Measurement in the Public Sector

Principles and Policies for Implementing an Information Network

Edward B. Perrin, Jane S. Durch, and Susan M. Skillman, Editors

Panel on Performance Measures and Data for Public Health Performance Partnership Grants

Committee on National Statistics

Commission on Behavioral and Social Sciences and Education

National Research Council

NATIONAL ACADEMY PRESS
Washington, D.C.
1999



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
--> Health Performance Measurement in the Public Sector Principles and Policies for Implementing an Information Network Edward B. Perrin, Jane S. Durch, and Susan M. Skillman, Editors Panel on Performance Measures and Data for Public Health Performance Partnership Grants Committee on National Statistics Commission on Behavioral and Social Sciences and Education National Research Council NATIONAL ACADEMY PRESS Washington, D.C. 1999

OCR for page R1
--> NATIONAL ACADEMY PRESS 2101 Constitution Avenue, N.W. Washington, D.C. 20418 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by Contract No. 282-95-0034 between the National Academy of Sciences and the U.S. Department of Health and Human Services. Support of the work of the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (No. SBR-9709489). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project. Library of Congress Cataloging-in-Publication Data Health performance measurement in the public sector : principles and policies for implementing an information network / Edward B. Perrin, Jane S. Durch, and Susan M. Skillman, editors ; Panel on Performance Measures and Data for Public Health Performance Partnership Grants, Committee on National Statistics, Commission on Behavioral and Social Sciences and Education, National Research Council.  p. cm. Includes bibliographical references (p. ).  ISBN 0-309-06436-8 (pbk.)  1. Public health—United States—Information services. 2. Public health—United States—Evaluation—Data processing. I. Perrin, Edward. II. Durch, Jane. III. Skillman, Susan M. IV. National Research Council (U.S.). Panel on Performance Measures and Data for Public Health Performance Partnership Grants.  RA423.2 .H43 1999 362.1'0973—dc21 99-6159 Additional copies of this report are available from National Academy Press, 2101 Constitution Avenue, N.W., Box 285, Washington, D.C. 20055. Call (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area). This report is also available on line at http://www.nap.edu Printed in the United States of America Copyright 1999 by the National Academy of Sciences. All rights reserved.

OCR for page R1
--> PANEL ON PERFORMANCE MEASURES AND DATA FOR PUBLIC HEALTH PERFORMANCE PARTNERSHIP GRANTS EDWARD B. PERRIN (Chair), Department of Health Services, School of Public Health and Community Medicine, University of Washington JOHN C. BAILAR III, Department of Health Studies, University of Chicago THOMAS J. BARRETT, Mental Health Services, Colorado Department of Human Services DAVID W. FLEMING, Center for Disease Prevention and Epidemiology, Oregon Health Division V. JOSEPH HOTZ, Departments of Economics and Policy Studies, University of California, Los Angeles IRA R. KAUFMAN, University of Medicine and Dentistry of New Jersey, Department of Environmental and Community Medicine, Robert Wood Johnson Medical School JOHN R. LUMPKIN, Illinois Department of Public Health WILLIAM A. MORRILL, Mathtech, Inc., Princeton, New Jersey R. HEATHER PALMER, Center for Quality of Care Research and Education, Harvard School of Public Health J. SANFORD SCHWARTZ, School of Medicine and the Wharton School, University of Pennsylvania MARY E. STUART, Maryland Collaboratory for Business and Health, Department of Sociology and Anthropology, University of Maryland, Baltimore County CYNTHIA P. TURNURE, Minnesota Department of Health PAUL J. WIESNER, DeKalb County Board of Health, Decatur, Georgia GAIL R. WILENSKY, Project Hope, Bethesda, Maryland JANE S. DURCH, Study Director JEFFREY J. KOSHEL, Study Director (through December 1997) SUSAN M. SKILLMAN, Senior Research Associate TELISSIA M. THOMPSON, Senior Project Assistant

OCR for page R1
--> COMMITTEE ON NATIONAL STATISTICS 1998–1999 JOHN E. ROLPH (Chair), Department of Information and Operations Management, University of Southern California JOSEPH G. ALTONJI, Institute for Research on Poverty and Department of Economics, Northwestern University JULIE DaVANZO, RAND, Santa Monica, California WILLIAM F. EDDY, Department of Statistics, Carnegie Mellon University WILLIAM KALSBEEK, Department of Biostatistics, University of North Carolina RODERICK J.A. LITTLE, School of Public Health, University of Michigan THOMAS A. LOUIS, School of Public Health, University of Minnesota CHARLES F. MANSKI, Department of Economics, Northwestern University WILLIAM NORDHAUS, Department of Economics, Yale University JANET L. NORWOOD, Urban Institute, Washington, D.C. EDWARD B. PERRIN, Department of Health Services, School of Public Health and Community Medicine, University of Washington PAUL R. ROSENBAUM, Department of Statistics, Wharton School, University of Pennsylvania FRANCISCO J. SAMANIEGO, Division of Statistics, University of California, Davis RICHARD L. SCHMALENSEE, Sloan School of Management, Massachusetts Institute of Technology MIRON L. STRAF, Director ANDREW A. WHITE, Deputy Director

OCR for page R1
--> Contents     Preface   ix     Executive Summary   1 1   Introduction   15     Origins of the Study   16     Performance Partnership Grants   16     Other Influences   17     Charge to the Panel   18     Essential Definitions   19     Performance Measurement and Related Concepts   19     Categories of Performance Measures   20     Phase I: Focus on Selection of Performance Measures   22     Use of Measures of Process and Capacity as Well as Outcomes   22     Guidelines for Selecting Performance Measures   24     Limitations of a Program-Specific Approach to Performance Measurement   25     Need to Strengthen State and Local Capacity for Data Collection and Analysis   26     Inadvisability of Using Performance Measures Alone for Resource Allocation Purposes   26     Phase II: Data and Information System Development to Support Performance Measurement   27     A Vision for a National Health Information Network   27     Critical Issues   28     Structure of the Report   29     References   29

OCR for page R1
--> 2   Putting Performance Measurement in Context   30     Use of Performance Measurement in Accountability Systems   30     Performance-Based Accountability Systems   31     Operation of Performance-Based Systems   33     Applications of Performance Measurement   37     Examples of Performance Monitoring and Accountability Systems   40     Government Performance and Results Act   41     Use of Performance Measures in the Maternal and Child Health Services Block Grant Program   42     Reporting Requirements for the Temporary Assistance for Needy Families Program   44     State Developments in Performance-Based Budgeting   45     Health Care Performance Measurement in the Private Sector   48     Conclusions   52     References   55 3   Performance Measurement Considerations for Publicly Funded Health Programs   59     Broad Array of Health-Related Services and Service Relationships   60     Measurement Considerations for Population-Based Health Services   61     Population Health Services   61     Monitoring Population Health Services   62     Monitoring the Infrastructure for Publicly Funded Health Programs   64     Some Performance Measurement Considerations Related to Program-Specific Matters   66     Environmental Health Programs   66     Mental Health Programs   68     Substance Abuse Programs   71     Process Guidelines as a Basis for Performance Measurement   74     Guidelines for Personal Health Services   75     Guidelines for Population-Based Health Services   76     Research Needs for Practice Guidelines and Performance Measurement   76     Conclusions   78     References   79 4   Data and Information Systems: Issues for Performance Measurement   83     Health Data Resources   84     Registries   85     Surveys   86     Patient Records and Related Clinical Encounter Data   87     Administrative Data   89     Claims Data   90     Linkage of Data Sets   91     Steps Toward Integration of Data Sets   93

OCR for page R1
-->     Assuring the Quality of Data and Data Analysis   95     Random Variation and Bias   96     Data Management   97     Challenges in Data Analysis   98     Developing and Implementing Standards for Data and Data Systems   103     Standardization Activities   103     Balancing Standardization and Change   108     Enhancing Performance Measurement through Advances in Information Technology   109     Data Collection and Transmission Technologies   109     Data Management and Analysis   110     Computer-Based Patient Records   111     The Internet and the World Wide Web   111     Limits of Technology   114     Privacy, Confidentiality, and Security of Health Data   114     Investing in Health Data and Data Systems   116     Data and Information Systems   117     Technology   119     Training and Technical Assistance   120     Taking a Collaborative Approach to the Development of Health Data and Information Systems   122     Meeting the Needs of Many Data Users   123     Collaboration in the Design and Implementation of Data Systems   125     Conclusions   126     References   127 5   Strategies for Supporting Performance Measurement Through a National Information Network   132     Conclusions Regarding Performance Partnership Agreements   133     A National Information Network for Health-Related Data   134     Essential Features of a Health Information Network to Serve Different Levels of Decision Making   135     Major Policy Actions Needed   138     National Collaboration   138     Integration of Data Systems   143     Technology Policy   144     Operational Principles   145     Involve a Broad Range of Public Agencies   145     Build on Existing Information Systems   146     Implementation Standards   147     Required Investment for an Information Network   152     Data Systems   152     Training and Technical Assistance   154     A National Research Agenda   156     Final Observations   157     References   158

OCR for page R1
-->     Appendices         A Potential Health Outcome and Risk Status Measures   163     B Workshop Agenda   168     C Biographical Sketches   170

OCR for page R1
--> Preface This is the second of two reports from the Panel on Performance Measures and Data for Public Health Performance Partnership Grants. In summer 1995, as part of its response to the need to ensure that public funding of health programs is related to documented program performance, the Office of the Assistant Secretary for Health of the U.S. Department of Health and Human Services (DHHS) requested that the National Research Council (NRC) convene an expert panel to examine and report on the technical issues involved in establishing performance measures in ten substantive program areas. Such measures were to be required as part of the proposed Performance Partnership Grants (PPG) Program, under which each state was to negotiate with DHHS an action plan with performance objectives that would be specific in terms of outcomes, processes, and capacity to be achieved within a 3- to 5-year period. The panel divided its work into two phases. The objective of the first phase was to develop a report to the Secretary of DHHS on performance measures in specified areas that would be useful to the PPG Program in its first 3–5 years. The report Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health (National Academy Press, Washington, D.C., 1997) presents the findings and recommendations resulting from the first phase of the panel's work. The focus of the second phase was on recommended improvements to existing local, state, and federal data systems that would facilitate the collection of data for use with existing and proposed performance measures. This two-phase approach was adopted by the panel because of our conclusion that developmental work is needed in both the public and private sectors to adapt, refine, or add to existing data systems to make them more useful in performance measure-

OCR for page R1
--> ment, and to delineate more clearly the complex causal links between program processes and outcomes. Although formal legislation mandating PPGs has not been adopted, various DHHS programs and a growing number of states and communities are moving to monitor and analyze health outcomes on their own. Moreover, there appears to be a growing consensus within the public health, substance abuse, and mental health communities about the value of performance measurement. Indeed, many people believe that the case for increasing, or even maintaining, public funding will depend on documented program performance. The development of performance measures is a continuation of earlier efforts to assess progress toward important public health goals. The broad acceptance of immunization rates and other such measures developed for Healthy People 2000, for example, has been instrumental in the creation of databases and the mobilization of resources in many jurisdictions to assess progress toward those objectives. The work of this panel was furthered significantly by four regional meetings of state officials and consumers convened by DHHS early in the process—in Portland, Oregon; San Francisco, California; Chicago, Illinois; and Philadelphia, Pennsylvania—and by input from several national associations of state agency administrators (the Association of State and Territorial Health Officials, the National Association of County and City Health Officials, the National Association of State Alcohol and Drug Abuse Directors, and the National Association of State Mental Health Program Directors). Our work during the second phase of the project was aided by the contributions of participants in a workshop held by the panel in July 1997. The workshop agenda and participants are listed in Appendix B. We also wish to thank Winnie Mitchell of the Substance Abuse and Mental Health Services Administration and David Moriarty of the Centers for Disease Control and Prevention, who provided special assistance by arranging several meetings with agency staff members. The panel was fortunate to have for its deliberations two specially commissioned working papers. ''Improving Federal-State Data Collection to Monitor Program Performance" was prepared by the Science and Epidemiology Committee of the Association of State and Territorial Chronic Disease Program Directors and the Council of State and Territorial Epidemiologists. Christopher Maylahn, New York State Department of Health, chaired the group that prepared this paper. "Quality Education Data: Unprecedented Opportunity for a Decade to Build" was prepared by Emerson Elliott, National Council for Accreditation of Teacher Education, and John Ralph, National Center for Education Statistics. Copies of these papers are available from the Committee on National Statistics (2101 Constitution Avenue, N.W., Washington, DC 20418). This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institu-

OCR for page R1
--> tion in making the published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following individuals for their participation in the review of this report: Thomas F. Babor, School of Medicine, University of Connecticut Health Center; Maureen M. Henderson, School of Medicine, University of Washington (emeritus); Ed Lichtenstein, Oregon Research Institute, Eugene; Thomas Louis, School of Public Health, University of Minnesota; J. Michael McGinnis, scholar in residence, National Research Council; Barbara J. McNeil, Department of Health Care Policy, Harvard Medical School; Donald M. Steinwachs, School of Hygiene and Public Health, Johns Hopkins University; Bernard Turnock, Community Health Sciences, University of Illinois, Chicago; and Albert W. Wu, Health Services Research Center, Johns Hopkins University. While the individuals listed above have provided constructive comments and suggestions, it must be emphasized that responsibility for the final content of this report rests entirely with the authoring committee and the institution. The panel appreciates the assistance of the staff of the Committee on National Statistics (CNSTAT) in preparing this report. We wish to thank in particular Miron Straf, staff director of CNSTAT, for developing the original project design; Telissia Thompson for organizing panel meetings and carrying out innumerable administrative tasks; and Kathleen Saslaw for providing financial management for the study. Several former CNSTAT staff members provided valuable assistance at various stages in the production of the panel's two reports: Anu Das, Theresa Raphael, Michelle Ruddick, and Ashley Bowers. We appreciate the editorial work of Rona Briere and the guidance of Eugenia Grohman, associate director for reports of the Commission on Behavioral and Social Sciences and Education. Thanks also go to Sue Skillman, University of Washington, for her valued assistance to the committee chair. Finally, the panel extends its special thanks to Jeff Koshel, the panel's study director for the first part of the study, and to Jane Durch, for her very capable management of and contribution to the process leading to the production of this second report. EDWARD B. PERRIN, CHAIR PANEL ON PERFORMANCE MEASURES AND DATA FOR PUBLIC HEALTH PERFORMANCE PARTNERSHIP GRANTS

OCR for page R1
--> The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council.

OCR for page R1
Health Performance Measurement in the Public Sector

OCR for page R1
This page in the original is blank.