Publicly funded health programs are being asked to account for their performance, and methods of performance measurement have emerged as essential tools for operationalizing this quest for accountability. A system of performance measurement promises improved documentation of the contributions of public agencies, and can serve as a quality improvement tool by drawing attention to practices shown to promote the achievement of desired outcomes and by identifying areas needing improvement. In fact, informed observers in public health, health policy, health economics, and related subjects believe that we cannot expect public funding to increase or even be maintained at current levels without better documentation of the return on program investments.
In 1995, the U.S. Department of Health and Human Services (DHHS) proposed legislative changes to establish Performance Partnership Grants (PPGS) for a set of federal block grant programs that provide funding to states for public health, substance abuse, and mental health activities.1 The proposal called for DHHS and each state to negotiate an agreement on program objectives and a set of performance measures to be used to monitor progress toward those objectives over a 3- to 5-year period. PPGs were proposed because it was felt that they would provide a mechanism for increasing state flexibility in the use of federal grant funds while enhancing accountability for progress toward program goals.
Although formal legislative requirements for PPGs have not been established, DHHS and state and local health agencies remain very interested in developing agreements through which these performance measurement and performance partnership concepts can be applied.
The Panel on Performance Measures and Data for Public Health Performance Partnership Grants was assembled in fall 1995 at the request of DHHS to assess the state of the art in performance measurement for the specific block grant programs included in the original PPG proposal, recommend measures that could be used to monitor the proposed PPG agreements to be negotiated between each state and the federal government, and recommend steps to improve performance measures and performance measurement for health-related programs.
The panel's work has resulted in two reports, of which this is the second. In its first report, the panel presented an analytic framework for use by states and DHHS in assessing the appropriateness of specific outcome, process, and capacity measures for individual performance agreements. That report also provided examples of health outcome and risk status measures, and related process and capacity measures, that might be used in conjunction with PPGs in the program areas covered by the original proposal.
In this second report, the panel addresses broader data and information system issues that require attention at the federal, state, and local levels to advance the practice of performance measurement for publicly funded health programs. This report broadens the discussion begun in the first report by looking beyond the federal-state PPG framework and beyond the specific program areas covered by the PPG proposal. Although the discussion focuses on the public-sector perspective, closely related private-sector interests and developments in clinical health care are also considered.
Principles of Performance Measurement
Performance measurement is not new, but its increasingly widespread use in both the public sector (e.g., in federal agencies under the Government Performance and Results Act and in state and local government accountability systems) and the private sector (e.g., in accreditation programs for health care organizations) reflects changing attitudes and expectations regarding accountability and management. An attractive feature of performance measurement is that it presents the opportunity to focus attention on defining and using evidence-based best practices to achieve desired outcomes. To define those best practices and select appropriate performance measures, evidence is needed not only on the causal links between interventions and outcomes, but also on how interventions work. This evidence should guide the organization, operation, and improvement of the services, as well as the selection and use of meaningful process and capacity performance measures. When evidence is not yet conclusive, the selection of performance measures should be guided by professional consensus regarding the
relationship between capacity and process factors and health outcomes. Explicit accountability for those processes and intermediate outcomes that are under programmatic control will encourage the further development of standards of practice in publicly funded health programs.
Performance measurement, as it has been implemented in various settings, involves (1) developing an explicit set of goals and objectives and articulating a strategy for achieving them, (2) developing and implementing strategies for measuring performance, and (3) using performance information to improve management practices or resource allocation. However, the panel cautions against using performance measures as the sole basis for causal inferences regarding program performance because of the diversity of factors beyond program activities that affect most health outcomes.
Successful implementation of performance measurement will require substantial and continuing efforts to overcome challenges involved in the development and use of performance measures and data systems. There must be a firm commitment to ongoing research to develop new and better measures, relate these measures to program actions, and evaluate and improve the performance measurement system. Performance measurement should also provide a framework for initiating systematic efforts, especially at the state and local levels, to identify and collect data for appropriate measures of program capacity, processes, and intermediate outcomes. Performance measurement activities will benefit from collaborations among agency staff and others who can bring to bear the perspectives and expertise of a variety of disciplines (e.g., public health, clinical services, statistics, epidemiology, data processing, data management).
The panel concluded that the following principles should guide current efforts to implement performance measurement for publicly funded health programs:
- Performance measurement activities must proceed from clearly defined program goals. The performance measurement process and the information it produces should be viewed as a tool for monitoring and promoting progress toward those goals, not as an end in itself.
- Because health needs and priorities vary across the country, a performance measurement system should promote the development of identifiable sets of measures from which states and communities can select subsets appropriate for the program priorities and strategies they have adopted. Ideally, all of the measures used should be recognized as valid, reliable, and responsive to change and have agreed-upon definitions.
- Performance measurement activities must recognize and meet information needs that may differ in terms of content and quantity for different functions (e.g., program operation, management, policy making, funding). Ideally, measures for specific functional purposes should be linked, conceptually or in practice, to provide a consistent assessment of performance across these different functions.
- Performance measurement must consider the feasibility of data collection and analysis for proposed measures. If appropriate data cannot be produced, the measures will not be useful. Considerations should include the quality and comparability of the available data and the cost of producing those data.
- The performance monitoring system, including the performance measures and the data supporting them, should be evaluated periodically. Such evaluation will help ensure that the system's goals are being met and decrease the likelihood of manipulation or inadvertent promotion of undesired effects, such as reducing services to groups that may be likely to have poor outcomes.
- Performance measurement should be viewed as a developmental activity that will continue to evolve. Measures should be refined or replaced as understanding of the linkages between health outcomes and program activities ("processes") improves, as better sources of data are developed, and as program priorities change. The panel cautions that because performance measurement is a new and largely unfamiliar policy mechanism, it should be tested in the context of goal setting, progress monitoring, and signaling before being used for resource allocation or regulatory purposes. Research and evaluation studies should assess performance measurement's effectiveness as a tool for improving health outcomes and program management.
Performance Partnership Agreements
Examination of performance measurement in the context of the proposal for PPGs led the panel to propose a set of general principles to guide further performance partnership efforts. First, those who are affected by decisions resulting from the application of performance measures (e.g., state and local health agencies) must share fully in the creation and selection of the measures and, where possible, the measurement process. Second, state- and local-level data are essential for achieving federal goals for performance measurement, and ensuring adequate support for data collection mechanisms to produce those data is important to the success of the system. Third, to make efficient and effective use of resources, performance measurement should, to the extent possible, rely on existing data systems that serve other managerial and operational purposes as well. Collaboration across and within the federal, state, and local levels can reduce the inconsistencies and incompatibilities that can arise in independently developed monitoring programs or data systems while preserving sufficient flexibility to accommodate the differences among states and communities in their programs and goals. Finally, performance measurement should focus on monitoring progress toward the overall goals of an activity and not on measuring the impact of a particular source of funding for that activity. The health outcomes that are linked to program goals are influenced by many factors, making inferences concerning the effect of individual funding sources problematic.
A National Information Network for Health-Related Data
Further advances in performance measurement for publicly funded health programs will require thoughtful and continuing attention to a varied set of policy, programmatic, and data and information system issues. The current focus on performance-based accountability has helped highlight the limitations of existing health-related data and data sources. Despite their diversity and breadth, the data presently suitable for performance measurement are limited. The need for stronger and more coherent health data systems is a concern at the federal, state, and local levels and in the private sector as well.
DHHS has an important leadership role to play, but all of the participants must share responsibility for ensuring that health data and data systems receive adequate support to operate efficiently and effectively. An investment must be made in the data collection programs and information technology that are at the core of these information systems, as well as in training and technical assistance for the people who produce and use health data.
The panel's conclusion is that over the long-term, performance measurement will be easier and more effective with the development of a broadly based national health information network that can promote a coordinated and collaborative approach to meeting diverse needs for a variety of health-related information, including performance data. This does not necessarily require the creation of an entirely new data system or a federally managed system. Instead, the panel envisions building on existing data systems across the local, state, and federal levels to produce the information required for performance measurement.
The development of a truly comprehensive health information network of national scope is a substantial endeavor that will almost certainly require an incremental approach with financial investment from a variety of sources, as well as commitment and persistence. The challenge is to develop a reasonably efficient and effective network that accommodates the many different data sources and information requirements that currently exist. The resulting network should meet real managerial and accountability needs; reflect important interdependencies and relationships across governmental and programmatic lines; and recognize that multiple, specialized data systems may no longer be affordable or consistent with other critical priorities.
As envisioned by the panel, the proposed national health information network should have the following features:
National Collaboration Collaboration among local, state, federal, and private efforts must be supported by adequate resources and must recognize the essential needs and contributions of each participant. The aim is to achieve a network of national scope that is not specifically a federal enterprise.
Linkage but not Consolidation The proposed network does not depend on
the development of a single national health database or information system. Compatible structure and architecture in information systems will promote linkage of comparable data and sharing of information. The confidentiality and security of personal health information must be ensured.
Ability to Meet the Needs of Varied Users The network should support multiple purposes, including monitoring for performance-based accountability of population-based and personal health services, operation of health programs, and delivery and management of clinical care. It should serve the information needs of managers, planners, health care providers, evaluators, policy makers, and the public at the national, state, and local levels, including ensuring that states and communities have the opportunity to produce or obtain timely data of local interest and of sufficient detail to be representative of their populations.
Standardization of Data and Measures Clear and common definitions of data elements, measures, and coding systems and standard approaches to data collection must be established so that information can be aggregated across multiple populations and regions, and comparisons, where appropriate, can be made over time or among populations and geographic areas.
Appropriate Performance Measures The measures adopted should represent multiple perspectives, specifically including that of the consumer (e.g., satisfaction with access to services). They should be applicable to varied types of services (e.g., individual and population-based), to different age groups, and to people of differing ethnic and cultural backgrounds.
Efficient and Effective Use of Resources Coordination and integration of information systems will often prove more efficient than the development and operation of program-specific systems. Data collection should be guided by careful judgments about the intensity of the information required for performance measurement in terms of frequency of data collection, level of detail, and completeness of coverage (e.g., sample or census data). The network should provide access to appropriate information from sources that are not primarily health-related (e.g., highway safety, corrections). Staff at all levels must have sufficient training and technical assistance to manage data systems and use information effectively.
Adaptability to Change Performance measurement and a national health information network must be able to change and evolve as information sources grow, knowledge expands, or program priorities and activities change. New measures must be developed and tested to respond to these evolving needs.
The panel concluded that further progress in performance measurement and in the development of a multilevel, user-oriented national health information network requires work in four broad areas: (1) policy actions that promote collaboration by federal, state, and local stakeholders in the performance measurement process and more effective integration of data systems; (2) operational principles that support efficient use of resources and promote use of appropriate perfor-
mance measures and data; (3) essential investment in data systems and in the staff who collect and use performance information; and (4) research to improve the evidence base for performance measures, as well as the use of performance measurement. The panel recommends several specific steps in each of these areas.
To succeed, performance measurement and the national health information network proposed by the panel must be supported by a broad and continuing commitment to a collaborative process that brings together as partners stakeholders from across federal, state, and local governments. The panel advocates such a national collaborative partnership to ensure that performance measurement practices and health data systems are responsive to program priorities and information needs at all levels of government. In addition to intergovernmental collaboration, the panel advocates greater intergovernmental collaboration to reduce duplication of effort and to promote data sharing and the development of comparable measures and definitions for data related to health outcomes and program activities. Where appropriate, these public-sector efforts should interact with related activities in the private sector.
The national collaboration recommended by the panel will require a process for initiating and continuing consensus-building discussions. DHHS is a key participant and may be an essential catalyst for this process, but it must act as a partner with state and local stakeholders. To ensure full and fair consideration of multiple points of view, participants may wish to identify an interested party without a direct stake in the outcomes (e.g., a foundation, a university) that can convene local, state, and national stakeholders in a neutral setting. Stakeholder groups may also wish to establish well-defined mechanisms for designating their representatives in these discussions. Participants should include both staff with policy and programmatic responsibilities who use health data and those with technical expertise in data collection and data analysis who produce and manage health data.
A well-designed and effectively operating performance monitoring system promises benefits for all of its participants. Those participants must, however, share responsibility for the design and maintenance of the system and for investment of the resources needed to give the system sufficient capacity to operate effectively. Participants' responsibilities also include working toward compromise solutions in such matters as uniformity in definitions and procedures, the choice of data items and data collection methods, and the timeliness and format of data forwarded to other collaborators.
To establish a collaborative base for performance measurement and a health information network, the panel recommends the following steps:
Federal, state, and local governments should commit to a common and national strategic goal of incorporating performance measurement into the practices of publicly funded health programs.
Federal, state, and local governments, with input from private partner organizations, should plan and implement all steps of the performance measurement process in full collaboration with one another.
DHHS should work in partnership with members of the relevant groups representing policy, program, and technical officials of states and local entities to establish a process for developing policies and procedures that can facilitate the implementation of performance measurement efforts in health-related areas.
Federal, state, and local governments should accept explicit responsibilities, determined in collaboration with other stakeholders, in return for their share in the governance of and benefits from broader efforts to improve performance monitoring.
Integration of Data Systems
The categorical nature of much of the federal funding for state and local health-related programs has often encouraged both a fragmented approach to health problems and the development of program-specific data systems and reporting requirements. A strictly programmatic perspective may discourage a more comprehensive approach that can capitalize on the complementary, overlapping, and even synergistic interactions among programs and their information system needs.
Even though programmatic funding streams are likely to remain a prominent feature of federal funding, additional opportunities are needed to improve data systems at the state and local levels by coordinating and integrating a broader array of health data. Some federal agencies are supporting a more integrated approach, and the panel encourages other agencies to facilitate a broader perspective in the planning for information system changes and to improve the likelihood of generating additional funds for the implementation of those changes. Specifically, the panel makes the following recommendations:
DHHS should lead efforts to integrate data systems across categorical health program lines.
DHHS, in collaboration with state and local partners, should review restrictions on the use of grant funds to determine whether
- they represent a significant barrier to progress in the development of integrated health information systems. If so, DHHS should pursue changes in the terms of those grant programs that would permit greater flexibility in the use of the funds.
Rapidly evolving information and communications technologies will enhance the feasibility of performance measurement and the health information network envisioned by the panel. Many state and local health agencies, however, lack the resources to keep up with the rapid pace of the revolution in information technology. Effective use of these technologies requires the development and implementation of standards to facilitate the transmission, aggregation, and linkage of data from multiple sources without requiring the standardization of equipment or operating systems and software. DHHS should serve as a catalyst for consensus building on information collection and transmission standards, and as a resource for technical assistance in the application of new information technologies for a broad range of health-related data. The panel emphasizes that development and use of information technologies must always address protection of the confidentiality and security of health-related data. Specifically, the panel makes the following recommendation:
DHHS should provide leadership in the development and use of data transmission standards and of new information technologies to collect, analyze, and disseminate health-related data.
The panel identified several matters related to the availability of data and the further development of performance measures that should be addressed to promote the successful implementation and operation of a performance measurement system for publicly funded health programs.
The development of a performance monitoring system requires consideration of the broad range of factors that influence desired health outcomes, as well as the administrative, analytic, and technical resources needed to collect data and use performance measures. State and local government agencies whose responsibilities are not primarily health-related must be part of the process. This includes agencies with programmatic responsibilities in nonhealth areas (e.g., criminal justice, housing, transportation) and those that collect and manage data on basic socioeconomic characteristics of the population and the state (e.g., population estimates, economic development data).
Although few existing data systems have been designed specifically for performance measurement, they nonetheless provide an essential base from which to
build an information network that can meet a broad range of health information needs, including performance data. The panel recommends that, instead of creating an entirely new data system for performance measurement, data needs be met to the extent possible by using existing systems, such as vital records, notifiable disease systems, adult and youth behavioral risk factor surveillance, cancer registries, and records on client services (e.g., prenatal care, substance abuse treatment). Other data systems beyond the purview of health agencies can also provide information valuable for tracking health risks and outcomes.
A performance monitoring system requires not only continuing data collection activities, but also a mechanism for ongoing review and refinement of performance measures. Measures must change to reflect the evolving knowledge base on which they rest, changes in health needs and opportunities for intervention, and changes in the health policy environment. A broad range of stakeholders must participate in the review process to ensure that performance measures are consistent with state and local public health priorities. Policy, programmatic, and technical perspectives must all be considered. The collaboration between the Centers for Disease Control and Prevention and the Council of State and Territorial Epidemiologists for periodic review and revision of the case definitions of specific infectious diseases might serve as a model in this regard.
As the panel observed in its first report, data limitations, differing health problems, and differing program priorities preclude prescribing a single set of performance measures for use by all states and communities. Instead, DHHS should work with states and communities to assemble sets of measures that meet the basic tests of validity, reliability, responsiveness, and data adequacy; users can then select smaller subsets of measures that meet their specific needs. Because states and localities may reasonably pursue many different strategies to target a single health outcome, a large number of process and capacity measures should be available for user choice.
Standard definitions for performance measures and standards for data collection should be adopted to enhance the comparability of performance data over time and across states and localities. Use of common measures and data definitions may encourage cost efficiencies by reducing the need to redesign data collection instruments, electronic processing protocols, and similar infrastructure elements. With sufficient comparability across state data systems, greater reliance might be placed on aggregating state data to produce national measures, rather than requiring separate data collection systems at the federal and state levels. Stakeholders must have a means of achieving consensus on these standards and harmonizing the implementation of their performance measurement activities. The advantages of greater standardization should not, however, obscure the need for continued critical assessment of the appropriateness of the measures and methods being used.
It is essential to ensure that performance measurement rests on data and data analysis of high-quality. Moreover, differences in data quality across individual
information systems may reduce the comparability of performance data. Although no data system or data set is ever perfect, and costs to reduce residual errors can be high, quality standards must be adopted. These standards should be based on informed assessments of how the data are to be used and the degree of accuracy and precision required to serve those uses.
The panel makes the following recommendations regarding these issues:
As states and communities work to implement performance monitoring systems for health-related programs, they should ensure that all relevant public agencies, including those outside traditional health areas, have the opportunity to participate.
When possible, partners should obtain performance measurement information from existing or enhanced federal, state, and local information systems.
DHHS, in partnership with state and local stakeholders, should lead the implementation of a process for ongoing development and review of performance measures to be used in conjunction with state and local health programs.
DHHS, in partnership with state and local stakeholders, should lead a process for assembling and evaluating sets of performance measures from which users can identify and agree upon those appropriate for specific applications.
DHHS should work in partnership with state and local stakeholders to promote the development and adoption of standard definitions for performance measures and standards for associated data collection and data quality in performance measurement systems.
Performance measurement activities are likely to impose new demands on those whose performance is being assessed (e.g., additional data collection or data system development, new data analysis and reporting). Adequate resources must be made available to meet those demands, as well as to maintain the effective elements of current data systems. Furthermore, efforts to enhance a health information network should not compromise funding for program services. To respond to these concerns, the panel recommends investments both in data systems and in training and technical assistance.
Existing health data systems provide a strong base for performance measurement, but they generally have not been developed for this purpose. Adequate resources are needed to maintain key information systems and to enhance or develop new systems for performance data that cannot currently be produced. Investments in data for state-and local-level performance measures should be a high priority. Innovative ways to use these resources should be explored, as should opportunities to improve the current investment of resources in data collection and analysis activities at the federal, state, and local levels. For example, careful examination of duplication in current data collection and data systems may suggest more efficient ways of meeting information needs, potentially freeing resources to improve or expand data systems. To address these matters, the panel makes the following recommendations:
DHHS and state and local users of performance measurement data should each commit resources to reduce gaps in the supporting information systems.
DHHS should sponsor a review of the current array of federal, state, and local data collection and analysis activities to begin an assessment of how existing resources might be used most effectively to meet performance measurement and other needs for health data. This review must include participation by appropriate state and local representatives.
Training and Technical Assistance
An investment is also necessary in state and local capacity for data collection and analysis. Staff vary in their knowledge of the relevant disciplines and methodologies (e.g., epidemiology, statistics, social science research) and in their experience with the use of data to plan, evaluate, and revise community programs. Expertise is also required in such areas as hardware, software, systems design and integration, and applications development. State and local health agencies are often understaffed, making it difficult for them to assume additional tasks associated with performance measurement. Relatively low salaries also place most health agencies at a disadvantage in the current highly competitive information technology market. If DHHS were to support a central resource for information and guidance on technology matters, state and local health agencies might be able to make more rapid use of a broader range of expertise than they could assemble on their own. Therefore, the panel recommends the following steps:
To ensure the success of performance measurement, all stakeholders, with substantial leadership from DHHS, should contribute ongoing technical assistance, training, and resources to enhance state and local data systems and analytic capacity.
DHHS should develop and maintain information technology expertise to assist states and communities as they use new technologies to improve the quality of and capacity for data collection, analysis, and dissemination.
A National Research Agenda
Research must be an integral part of any ongoing program of performance measurement for health-related programs. Because experience with performance measurement is still limited, studies are needed to improve understanding of what measures and methods of data collection are appropriate. Further research must also be done to establish the evidence base for causal links between program interventions and desired outcomes. This evidence, essential for selecting demonstrably meaningful capacity, process, and risk status measures, is currently limited in many fields. Studies will need to draw on expertise from a variety of disciplines, and they must be informative for a variety of settings at the local, state, and national levels. However, research by itself is not sufficient for informing and improving the performance monitoring process; resources must also be available to ensure that significant findings are communicated to those involved in performance measurement. The panel makes the following specific recommendations:
Federal agencies, foundations, and other private-sector groups should develop and fund a research agenda to support performance measurement activities, including the testing of intervention effectiveness, the investigation of the links between program capacity and processes and program outcomes, the development of measures, the refinement of data collection and information system technologies, and the use of performance measurement systems and performance-based decision making.
DHHS, foundations and other private organizations, and other partners involved in performance measurement activities should contribute in an appropriate manner to a process of information gathering and dissemination to support the use of evidence-based performance measures.
The broad national health information network envisioned by the panel should enhance the capacity of federal, state, and local health programs to meet performance measurement obligations and to use performance data and other information to achieve desired health outcomes. The leadership of DHHS and its agencies is critical in this process, but the department must participate as a partner with stakeholders at the state and local levels, in the public and private sectors, and across a variety of program areas. Several significant challenges lie ahead. Appropriate performance measures must be developed, and the data needed to use those measures must be available. Greater consensus must be achieved regarding standards for measures and data that will promote comparability in performance measurement. Necessary technical and analytic skills must be developed and applied to the creation and use of performance data. Sufficient financial and nonfinancial resources must be obtained to support both near-term efforts to introduce performance measurement activities and longer-term data collection, analysis, and research necessary to sustain those efforts. The need to address these issues is great, and the current commitment to performance measurement presents an opportunity to make significant progress toward meeting this need.