NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competencies and with regard for appropriate balance.
This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences , the National Academy of Engineering, and the Institute of Medicine.
This study was supported by Contract No.282-95-0034 between the National Academy of Sciences and the U.S. Department of Health and Human Services. Any opinions, findings, conclusions , or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project.
Library of Congress Cataloging-in-Publication Data
Assessment of performance measures in public health, substance abuse, and mental health/
Edward B. Perrin and Jeffrey J. Koshel, editors ; Panel on Performance Measures and Data for Public Health Performance Partnership Grants, Committee on National Statistics, Commission on Behavioral and Social Sciences and Education, National Research Council.p. cm.
Includes bibliographical references (p.).
ISBN 0-309-05796-5 (pbk.)
1. Public health-Evaluation. 2. Public health-United States-Evaluation. I. Perrin,
Edward. II. Koshel, Jeffrey. III. National Research Council (U.S.). Panel on Performance
Measures and Data for Public Health Performance Partnership Grants
RA427.A75 1997
362.1'0973-dc2l 97-4915 CIP
Printed in the United States of America
Copyright 1997 by the National Academy of Sciences. All rights reserved.
EDWARD B. PERRIN (Chair), Department of Health Services, School of Public Health and Community Medicine, University of Washington
JOHN C. BAILAR, III, Department of Health Studies, University of Chicago
THOMAS J. BARRETT, Colorado Mental Health Services, Denver, Colorado
DAVID W. FLEMING, Center for Disease Prevention and Epidemiology,Oregon Health Division
V. JOSEPH HOTZ, Irving B. Harris Graduate School of Public Policy Studies, University of Chicago
IRA R. KAUFMAN, University of Medicine and Dentistry of New Jersey, Department of Environmental and Community Medicine, Robert Wood Johnson Medical School
JOHN LUMPKIN, Illinois Department of Public Health
WILLIAM A. MORRILL, Mathtech, Inc., Princeton, New Jersey
R. HEATHER PALMER, Center for Quality of Care Research and Education,Harvard School of Public Health
J. SANFORD SCHWARTZ, Leonard Davis Institute of Health Economics
MARY E. STUART, Maryland Collaboratory for Business and Health, Department of Sociology and Anthropology, University of Maryland, Baltimore County
CYNTHIA P. TURNURE, Chemical Dependency Program Division, Minnesota Department of Human Services
PAUL J. WIESNER, DeKalb County Board of Health, Decatur, Georgia
GAIL R. WILENSKY, Project Hope, Bethesda, Maryland
JEFFREY J. KOSHEL, Study Director
SUSAN M. SKILLMAN, Senior Research Associate
ASHLEY BOWERS, Research Assistant
ANU PEMMARAZU, Research Assistant
MICHELLE M. RUDDICK, Research Assistant
TELISSIA M. THOMPSON, Senior Project Assistant
THERESA RAPHAEL, Consultant
COMMITTEE ON NATIONAL STATISTICS
1996-1997
NORMAN M. BRADBURN (Chair), National Opinion Research Center,University of Chicago
JULIE DaVANZO, RAND, Santa Monica, California
WILLIAM F. EDDY, Department of Statistics, Carnegie Mellon University
JOHN F. GEWEKE, Department of Economics, University of Minnesota, Minneapolis
JOEL B. GREENHOUSE, Department of Statistics, Carnegie Mellon University
ERIC A. HANUSHEK, W. Allen Wallis Institute of Political Economy and Department of Economics, University of Rochester
RODERICK J.A. LITTLE, Department of Biostatistics, University of Michigan
CHARLES F. MANSKI, Department of Economics, University of Wisconsin
WILLIAM NORDHAUS, Department of Economics, Yale University
JANET L. NORWOOD, Urban Institute, Washington, D.C.
EDWARD B. PERRIN, Department of Health Services, School of Public Health and Community Medicine, University of Washington
PAUL R. ROSENBAUM, Department of Statistics, Wharton School, University of Pennsylvania
KEITH F. RUST, Westat, Inc., Rockville, Maryland
FRANCISCO J. SAMANIEGO, Division of Statistics, University of California, Davis
MIRON L. STRAF, Director
KATHLEEN SASLAW, Administrative Associate
1 INTRODUCTION AND FRAMEWORK
2 CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES
3 POTENTIAL PPG MEASURES FOR 1997-2002
-
Chronic Diseases
STDs, HIV, and Tuberculosis
Mental Health
Immunization
Substance Abuse
Sexual Assault, Disabilities, and Emergency Medical Services
4 IMPLEMENTING PERFORMANCE-BASED AGREEMENTS
REFERENCES
APPENDICES
A PERFORMANCE MEASURES: SOURCE MATERIALS
B OVERVIEW OF FEDERAL DATA SOURCES FOR PPG MEASURES
C POTENTIAL HEALTH OUTCOME AND RISK STATUS MEASURES
D ANALYSIS OF COMMENTS ON DRAFT REPORT
In summer 1995, as part of its response to the need for assuring that public funding of health programs be related to documented program performance, the Office of the Assistant Secretary for Health of the U.S. Department of Health and Human Services (DHHS) requested that the National Research Council convene an expert panel to examine and report on the technical issues involved in establishing performance measures in ten substantive program areas. Such measures would be required as part of the proposed Performance Partnership Grants (PPG) Program, under which each state will negotiate with DHHS an action plan with performance objectives that are specific in terms of outcomes, processes, and capacity to be achieved over 3-5 years.
The panel divided its work into two phases. The objective of the first phase was a report to the Secretary of DHHS on performance measures in specified areas that would be useful to the PPG Program over the next 3-5 years. This report presents the panel's findings and recommendations of the first phase. In the second phase, the panel will consider and report on what needs to be done to improve performance measures, for example, by designing new data systems and surveys and increasing understanding of the relationship between programmatic interventions and health outcomes. This two-phase approach was adopted by the panel because of our conclusion that developmental work is needed in both the public and private sectors to adapt, refine, or add to existing data systems to make them more useful in performance measurement and to more clearly delineate the complex causal links between program processes and outcomes.
Because of the uncertainty about the structure of the substance abuse and mental health block grants legislation, as well as funding levels for various public health programs being considered for conversion to a PPG format, it is unlikely that PPG contracts between the federal government and state agencies will go into effect before fiscal 1998. Yet a growing number of states are moving to monitor and analyze outcomes on their own. Moreover, there appears to be a growing consensus within the public health, substance abuse, and mental health communities about the value of performance measurement. Indeed, many people believe that the case for increasing, or even maintaining, public funding will depend on documented program performance.
The development of performance measures is a continuation of earlier efforts forts to assess progress toward important public health goals. The broad acceptance of immunization rates and other such measures developed for Healthy People 2000, for example, has been instrumental in the creation of data bases and the mobilization of resources in many jurisdictions to assess progress toward those objectives.
The work of this panel has been furthered significantly by four regional meetings of state officials and consumers convened by DHHS-in Portland, Oregon; San Francisco, California; Chicago, Illinois; and Philadelphia, Pennsylvania -and by input from several national associations of state agency administrators (the National Association of State Alcohol and Drug Abuse Directors; the National Association of State Mental Health Program Directors; the Association of State and Territorial Health Officials; and the National Association of County and City Health Officials). DHHS stated that the purpose of the regional meet meetings was to "develop comprehensive lists of desired program results to initiate the process of developing performance objectives." Prior to the meetings, organizations representing potential PPG grant recipients were asked to consult on a state, regional, or national basis to solicit opinions from their membership about the PPG measures that are important for individual programs. Representatives of these institutional interests were invited to attend the regional meetings, and participants included representatives of state and local governments, public health groups, tribal governments, professional associations, providers, consumers, and advocacy groups. DHHS has actively involved these associations in soliciting information on potential measures and data sources for the PPG process.
Other efforts are under way to examine performance measures for areas of public health that are not included in the panel's charge. For example, two studies were recently completed at the Institute of Medicine (IBM), one focusing on community health performance measures and the other examining performance indicators, standards, and measures for accreditation and quality assurance for managed behavioral health care. The first study, conducted by the Committee on Using Performance Monitoring to Improve Community Health, developed prototypical sets of indicators for specific public health concerns that communities can use to monitor the performance of public health agencies, per personal health care organizations, and other entities that can contribute to health improvement (Institute of Medicine, 1997b). The second study, conducted by the Committee on Quality Assurance and Accreditation Guidelines for Managed Behavioral Health Care, developed a framework for accreditation standards and quality improvements for managed behavioral health care and for developing, using, and evaluating performance indicators (Institute of Medicine, 1997c).
The panel is mindful of the great interest that surrounds the PPG concept:the eventual usefulness of our reports will depend on understanding and agreement by the federal and state officials and others who will be key players in implementing it. For that reason, this report was released in draft form for public comment. The panel was especially interested in receiving comments on its assessment of data availability and quality and the usefulness of the suggested measures in light of the limited empirical data that link program interventions to health outcomes. The panel received many comments, which contributed to this report; see Appendix D for an overview of the comments received and the panel's responses.
The panel appreciates the assistance of the staff of the Committee on National Statistics (CNSTAT) in preparing this report: Miron Straf, staff director of CNSTAT for developing the original project design; Telissia Thompson for organizing panel meetings; Anu Pemmarazu for preparing key technical materials; Theresa Raphael for preparing the literature review for the panel and preparing the annotated bibliography; and Michelle Ruddick and Ashley Bowers for coding and analyzing the responses from the field. Thanks also go to Sue Skillman, University of Washington, for her valued assistance to the committee chair. Finally, special thanks go to Jeff Koshel, study director for this panel, for his very capable management of the study process.
Edward Perrin,Chair
Panel on Performance Measures and Data for
Public Health Performance Partnership Grants
The Panel on Performance Measures and Data for Public Health Performance Partnership Grants was established at the request of the U.S. Department of Health and Human Services (DHHS) Its charge is to examine the state of the art in performance measurement for public health and to recommend measures that could be used to monitor the Performance Partnership Grant agreements to be negotiated between each state and the federal government. The panel was asked to consider performance measures in ten areas, which are clearly a subset of the full range of traditional public health concerns: chronic diseases; sexually transmitted diseases (STDs), human immunodeficiency virus (HIV) infection, and tuberculosis; mental health; immunization; substance abuse; and three areas of prevention of special interest to DHHS-sexual assault, disabilities, and emergency medical services. This report focuses on measures that states and the federal government can use over the next 3 to 5 years to negotiate agreements and monitor performance in these areas. A later report will examine additional mea measures that might be developed from new research findings on program effective effectiveness or as improvements are made to state and federal surveys and data systems.
More than 3,200 measures were proposed to the panel through various outreach efforts. The panel used four guidelines for assessing them: (1) the measure should be specific and result oriented; (2) the measure should be meaningful and understandable; (3) data should be adequate to support the measure; (4) the measure should be valid, reliable, and responsive. The measures that scored the highest are those we recommend for use in performance monitoring. They cover health status, social functioning, consumer satisfaction, and risk status.
In assessing the adequacy of data for specific performance measures, the panel concluded that there are few available data sources that are ideal for performance monitoring. Understanding the limits of available data is important if appropriate inferences are to be drawn. Many federal efforts to collect health related data, for example, provide national rates, but do not collect data that provide state-level rates. Even when data are available at the state level, if comparisons are to be made among states, attention must be paid to the effect of different data collection methods on the comparability of results. Other issues that need to be considered include whether or not specific populations of interest are included in samples from which data are drawn and whether data are collected sufficiently often, or are made available soon enough, to be useful in the monitor monitoring process.
It is important to note that many of the performance measures presented in this report can, and should, be subdivided to focus on specific high-risk populations in a state. These populations may be defined demographically, such as minorities, children, or elderly persons; by conditions, such as not having health insurance or being homeless; or by geographic area, such as central cities, high risk neighborhoods, or rural communities. Specific subpopulations of interest vary across states. Rather than create multiple submeasures for each proposed measure, the panel chose, in most cases, to identify broad population measures that can be tailored by each state to focus on its specific population group priorities.
Despite their widespread use and intuitive appeal, health outcome measures are insufficient by themselves for monitoring the efforts of a given program in reducing complex public health problems. Many measures that are recognized as valid for tracking health outcomes are affected by many factors (inputs or processes), so changes in outcomes cannot be attributed only to specific program effectiveness. Attribution of responsibility for outcomes becomes even more difficult when the services in question are supported by multiple funding sources or multiple provider organizations. The panel concludes that performance monitoring must make use of process and capacity measures to complement available measures of outcomes. The panel recommends that each process and capacity measure be accompanied by reference to published clinical guidelines or other professional standards that describe the relationship between the process measure or capacity measure and the desired health outcome.
Given the current and potential uses of performance measurement in public health, substance abuse, and mental health, the panel recommends that a combination nation of measures of health outcome, process, and capacity be used in the agreements between the federal government and states. Because in some cases actual health status outcomes are impractical to measure or because there are many factors that affect the ultimate health outcome, the panel recommends using "intermediate" outcome measures, such as risk status, for which there is general consensus that the result being measured is related to the health status outcome. The panel uses the following definitions in this report:
Because of data limitations and differing health and defined population priorities among states, the panel' s list of health outcome measures should be considered an important subset, but not an exhaustive listing, of those that will be of interest to state agencies around the country. Few states have adequate data to support every health outcome measure, and virtually all states have major priorities in addition to the ones indicated by these particular measures. Similarly, for process and capacity measures, there are many reasonable strategies that states can pursue to improve health outcomes, and each strategy requires a different set of process and capacity measures. Therefore, the panel offers representative examples of relevant process and capacity measures in each program area.
The potential health outcomes and risk status measures to be used for monitoring purposes are presented in Chapter 3 and described in detail in Appendix C. For each health topic covered, the report includes examples of process and capacity measures that complement the outcome and risk status measures suggested by the panel. Potential measures for chronic disease focus on improvement of health risk status for tobacco use, nutrition, exercise, and clinical screenings. For STDs, HIV infection, and tuberculosis, the potential outcome measures target reporting of incidence and prevalence rates for specific diseases; client satisfaction with treatment, and reduction of high-risk behavior among specific subpopulations at high risk of contracting or spreading the diseases. The immunization measures include a set for monitoring the incidence of vaccine-preventable disease and a set to be used to track vaccination rates for specific diseases. Most of the potential mental health measures focus on health outcomes for the treated population. Lack of data for measures of mental health outcomes in more general populations severely limits the number of potential measures the panel proposes. In sub stance abuse, the potential measures cover both treated and general populations for health status outcome, social functioning outcome, and risk status. For the three areas of prevention-sexual assault, disabilities, and emergency medical services-the narrowness of this charge to the panel and the general dearth of supporting data have resulted in a short list of potential measures.
Of course, use of even a large number of health outcome, process, and capacity measures may still result in less than conclusive evidence of agency or program performance in reducing multifaceted health problems. Therefore, the panel recommends that public health performance measures be considered as a central but not the only element of a continuous program of technical assistance. For example, if one measure or a combination of measures suggests that a given state is having unusual difficulty in making progress in meeting its performance objectives, such information should trigger an alert that some additional resources or technical assistance may be needed. The panel believes that this approach is consistent with the National Performance Review initiative at the federal level and with the total quality management activities that are being undertaken by state and local agencies around the country.
A major goal of this report is to provide an analytic framework for states and DHHS to use when assessing the appropriateness of specific outcome, process, and capacity measures for individual performance agreements. Recognizing that data resources and measurement methods need improvement, the panel recommends that DHHS continue to work with states toward several infrastructure goals: developing common definitions and measurement methods; encouraging efficient development of data resources that support multiple public health, mental health, and substance abuse needs; incorporating state data priorities in national infrastructure development efforts; and promoting states' data collection and analytic capabilities.
During the next stage of the study, the panel will examine the adequacy of existing databases to support improved health outcome measures, assess the quality of the empirical evidence of the effectiveness of specific interventions and the health outcomes discussed in this report, and suggest modifications to existing data sources or new databases necessary to support refined or new performance measures. Based on that assessment, the panel will recommend priority areas of research and data collection and infrastructure development for each of the health areas covered in this report, as well as for more general areas of public health concern.