5
Strategies for Supporting Performance Measurement Through a National Health Information Network

As the previous chapters have shown, further advances in performance measurement for publicly funded health programs will require thoughtful and continuing attention to a varied set of policy, programmatic, and data and information system issues. The current focus on performance-based accountability has helped highlight the limitations of existing health-related data and data sources. Despite their diversity and breadth, the data suitable for performance measurement are limited. In fact, the U.S. Department of Health and Human Services (DHHS) (1997:10) has acknowledged that lack of suitable data is a ''critical limiting factor" in developing departmental performance objectives as required under the Government Performance and Results Act (GPRA). The need for stronger and more coherent data systems for public health surveillance and personal health services is a concern at the federal, state, and local levels and in the private sector as well (e.g., Thacker and Stroup, 1994; Gold, 1995; Lasker et al., 1995; Mendelson and Salinsky, 1997; National Committee for Quality Assurance, 1997; Starr, 1997; U.S. Department of Health and Human Services, 1997; Public Health Foundation, 1998).

This final chapter reviews the panel's essential conclusions regarding performance partnership agreements, outlines its vision for a national health information network to facilitate performance measurement for publicly funded health programs, and recommends steps that can and should be taken to realize that vision. Although much in these conclusions and recommendations is applicable in principle to the broadest spectrum of health services in the public and private sectors, the panel has anchored its positions in the context of its discussions of



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 132
--> 5 Strategies for Supporting Performance Measurement Through a National Health Information Network As the previous chapters have shown, further advances in performance measurement for publicly funded health programs will require thoughtful and continuing attention to a varied set of policy, programmatic, and data and information system issues. The current focus on performance-based accountability has helped highlight the limitations of existing health-related data and data sources. Despite their diversity and breadth, the data suitable for performance measurement are limited. In fact, the U.S. Department of Health and Human Services (DHHS) (1997:10) has acknowledged that lack of suitable data is a ''critical limiting factor" in developing departmental performance objectives as required under the Government Performance and Results Act (GPRA). The need for stronger and more coherent data systems for public health surveillance and personal health services is a concern at the federal, state, and local levels and in the private sector as well (e.g., Thacker and Stroup, 1994; Gold, 1995; Lasker et al., 1995; Mendelson and Salinsky, 1997; National Committee for Quality Assurance, 1997; Starr, 1997; U.S. Department of Health and Human Services, 1997; Public Health Foundation, 1998). This final chapter reviews the panel's essential conclusions regarding performance partnership agreements, outlines its vision for a national health information network to facilitate performance measurement for publicly funded health programs, and recommends steps that can and should be taken to realize that vision. Although much in these conclusions and recommendations is applicable in principle to the broadest spectrum of health services in the public and private sectors, the panel has anchored its positions in the context of its discussions of

OCR for page 132
--> publicly funded health programs, including those in mental health, substance abuse, and various areas of public health. Conclusions Regarding Performance Partnership Agreements Examination of performance measurement in the context of the proposal for Performance Partnership Grants (PPGS) for health programs has led the panel to a set of general principles that it believes should guide further performance partnership efforts. First, the panel concludes that those who are affected by decisions resulting from the application of performance measures must share fully in the creation and selection of those measures, and, where possible, the measurement process. Representatives from federal, state, and local health and health-related agencies, and often segments of the general public, various private-sector interests, and health care provider groups, should all have a significant voice in the development, selection, assessment, interpretation, and use of performance measures. Second, the panel concludes that state- and local-level data are essential to the achievement of federal goals for performance measurement and that support of data collection mechanisms to produce those data is important to the success of the system. Most national surveys, for example, are not able to produce state-specific estimates that can be used to compare state-level performance. The development of performance measures and the data systems on which they depend should be approached with a broad national perspective that considers the interdependencies across governmental levels. Third, the panel believes that performance measurement information, resources, and processes should be organized so that states and communities can avoid unnecessary duplication of effort in developing new or enhanced data systems by using, to the extent possible, existing data systems that already serve other purposes and by working together to learn from each other. Collaborative efforts within and among states can reduce the unnecessary inconsistencies and incompatibilities that tend to arise in independently developed monitoring programs or data systems. The challenge is to foster this collaboration and cooperation while preserving the flexibility needed by individual states and communities to accommodate diversity in their programs and goals. Finally, the panel stresses that performance measurement should focus on the overall goals of an activity, not seek to measure primarily the impact of a particular source of funding for that activity. Generally speaking, program goals are best represented by outcome measures, and most health outcomes are influenced by many more factors than those that might be linked to a single funding source, making inferences concerning individual funding sources problematic.

OCR for page 132
--> A National Information Network for Health-Related Data Having considered both the general features of performance measurement and the specific context of publicly funded health-related programs, the panel has concluded that over the long-term, performance measurement for such programs will be made easier and more effective by the development of a broadly based national health information network that can promote a collaborative and coordinated approach across the local, state, and federal levels and can help in meeting diverse needs for a variety of health-related information, including performance data. This approach does not require the creation of an entirely new data system or a federally managed system. Instead, the panel envisions building on existing data systems operated by agencies at all levels of government, as well as looking to data systems in the private sector, to produce the information required for performance measurement. The challenge is to develop a reasonably efficient and effective network from the current diversity of data sources and information needs. A collaborative approach, consistent with the partnership element of the PPG proposal, will be essential for success. Collaboration is needed to accommodate the missions of both population-based and personal health services and to facilitate harmonization of parallel activities that are conducted independently by individual states or communities (e.g., vital records systems, the Behavioral Risk Factor Surveillance System [BRFSS]). Among the most important tenets guiding the development of the envisioned information network should be the requirements that it meet real managerial and accountability needs; that it reflect important interdependencies and relationships across governmental and programmatic lines; and that it recognize that multiple, specialized data systems may no longer be affordable or consistent with other critical priorities. In the current technological environment, an information network that can facilitate the transmission and aggregation of data from multiple sources without requiring the use of specific equipment or software is more feasible than ever before. Cooperation and collaboration are required, however, to establish agreement on such matters as electronic interfaces and data definitions. Effective use of technology also requires investments both in the development of staff expertise and in hardware and software. (See the discussion of this issue in Chapter 4.) Such investments have, however, been beyond the reach of many state and local health agencies. The panel recognizes that developing a truly comprehensive information network of national scope is a massive, potentially overwhelming endeavor. The substantial overlap that the panel found in potential performance measures for the substantive areas it considered in its first report clearly argues against a narrow, disease-specific approach to data collection and analysis. However, the larger issues involved in developing a national information system of relevance to a

OCR for page 132
--> broad range of health and health care concerns will be much more difficult and complicated, both technically and politically, than the further development of disease-specific systems. This much broader undertaking will require the involvement of many DHHS agencies (e.g., Centers for Disease Control and Prevention [CDC], Health Resources and Services Administration [HRSA], Substance Abuse and Mental Health Services Administration, Food and Drug Administration, Health Care Financing Administration, National Institutes of Health), other federal agencies (e.g., Department of Agriculture, Environmental Protection Agency, Department of Education, Department of Housing and Urban Development, Department of Justice, Department of Transportation), and a similar array of state and local agencies, plus the private sector. Establishing and maintaining the network envisioned by the panel will require a strong and continuing commitment by health agencies at the local, state, and federal levels. Even with such a commitment, this undertaking will almost certainly require an incremental approach. Many states do not yet have the capability to produce the information that would be required in any evaluation framework, and other states with more advanced data systems will have to make some changes to achieve compatibility with new standards that will evolve for a broad national network. Initial efforts might focus on enhancing existing information systems that serve either broad surveillance purposes or specific programmatic areas (e.g., children's health, substance abuse, mental health, chronic disease) or perhaps on conducting state or local demonstration projects to test the coordination of data systems across programmatic lines. Many of these public-sector programs must also rely on currently inadequate clinical data systems. Various observers (e.g., Institute of Medicine, 1997a; National Committee for Quality Assurance, 1997; Starr, 1997) have addressed the need for a commitment by health care organizations to make incremental improvements toward building a comprehensive information framework that can increase the health care industry's capacity to measure and improve performance. Successful development of information systems for performance measurement in the public or private sectors will require both time and financial investment from a variety of sources, as well as commitment and persistence. Essential Features of a Health Information Network to Serve Different Levels of Decision Making Given the complexity and scope of the task of developing the envisioned health information network, the panel believes that a clear vision of the intended long-term goal is required if the effort is to stay on course amid the vast amount of detail, variation, and difficult choices involved. Thus, the panel's vision for a cohesive, national health information network includes the following features. National Collaboration The network should be developed through collabo-

OCR for page 132
--> ration among local, state, federal, and private efforts. A collaborative approach, in contrast to a more hierarchical or centralized model, requires the recognition of all participants as partners who can contribute to the success of the information network and the performance measurement activities it is intended to support, and whose information needs and program priorities must be taken into consideration. Participants should expect both to gain benefits from access to the information network and to assume responsibility for contributing to its effective operation, including ensuring the availability of adequate resources (e.g., funding, staff, data, information technology and expertise, commitment of policy makers). With multiple participants at each level, a collaborative approach is necessary to promote efficient and effective data collection and use, as well as agreement on appropriate performance measures. The aim is to achieve a network that is national in scope without being specifically a federal enterprise. The voluntary state-based National Vital Statistics System (National Center for Health Statistics, 1998) serves as an example of a mature nationally collaborative network. Linkage but not Consolidation The information network should be based on compatible structure and architecture to promote linkage of comparable data and sharing of information within and across the local, state, and national levels in a manner that is consistent with appropriate protections for personal privacy and the confidentiality and security of health-related data. The concept of a comprehensive network implies neither the desirability nor the need for a single national health database or information system. Ability to Meet the Needs of Varied Users The network should, in the aggregate, support multiple purposes, including monitoring for performance-based accountability of population-based and personal health services, operation of health programs, and delivery and management of clinical care (including payment for services). To do so, it must encompass the full range of health-related services, from population-based and clinical preventive services through treatment and remediation. The network should be able to serve the information needs of managers, planners, health care providers, evaluators, policy makers, and the public at the national, state, and local levels. For example, data systems operated by states and communities can provide essential geographic detail and flexibility in data collection and analysis that are often not available from federal data systems, whereas federal data systems provide the broad national information that guides federal policy making and can serve as a reference point for assessing progress by individual states and communities. For performance measurement specifically, the information network must be able to accommodate the differing data and information needs at various oversight levels. Some measures and data are important primarily or exclusively at the operating level, while other, often more limited information is appropriate for a more distant accountability audience. Plans for performance measurement should distinguish among these information needs and match the selection of measures accordingly.

OCR for page 132
--> Standardization of Data and Measures Critical to such a comprehensive network is the establishment of clear and common definitions of data elements, measures, and coding systems, and standard approaches to data collection so that information can be aggregated across multiple populations and regions and so that comparisons, where appropriate, can be made among populations and geographic areas. The BRFSS, for example, illustrates the development of a standard survey questionnaire that is used independently in each state. Currently, this survey provides the primary means by which state-specific estimates of key behavioral risks are generated, but variations in survey methods among the states can limit the comparability of the data collected. To meet the full range of performance measurement requirements, new or modified data collection instruments with demonstrated reliability and validity or more standardized protocols may be needed to maximize the value of the information collected. Appropriate Performance Measures For performance measurement, an information network should focus on measures of health outcomes and risk factors and on measures of processes and capacity that are widely recognized as linked to important health outcomes. For many program areas, the measures should be selected to represent multiple perspectives, specifically including that of the consumer (e.g., satisfaction with access to services). The range of measures must also be applicable to varied types of services (e.g., individual and population-based, inpatient and outpatient), to different age groups (e.g., youth, adult, and older adult), and to people of differing ethnic and cultural backgrounds. Efficient and Effective Use of Resources The envisioned national health information network will require sufficient resources to operate effectively and support a wide range of users, but it should be organized efficiently to minimize its consumption of resources and any competition for resources with health programs and services. One consideration is efficient data collection. Certain types of data that can be used for performance measurement can be collected using various methods, such as consumer surveys, medical record reviews, or compilation of data from administrative records. Likewise, data might be collected at varying intervals (e.g., continuously, annually, every 2 years). In setting performance measurement requirements, careful judgments should be made about the intensity of the information required in terms of frequency of data collection, level of detail, and completeness of coverage (e.g., sample or census data). Coordination and integration of information systems will often prove more efficient than the development and operation of program-specific systems. An information network should also take advantage of appropriate information available from sources that are not primarily health-related. For example, mental health programs may want information from corrections department databases on the number of people with serious mental illness in youth and adult corrections systems. Training and technical assistance must be an integral part of the frame-

OCR for page 132
--> work for an information network so that staff at all levels have the skills to manage data systems effectively and use the information that they produce. Adaptability to Change The national health information network should be established on the premise that it is a developing tool that must be able to adapt easily and in a timely fashion to changes and enhancement as information sources grow, knowledge expands, or requirements change. One can expect performance measures to evolve and improve as the focus of interventions changes (e.g., the decline in lead poisoning as the inventory of homes with lead-based paint drops) and as new public health threats are identified. With this framework for a multilevel, user-oriented national information network in mind, the discussion now turns to steps that can be taken to promote the development of such a network. In discussing these steps, the panel presents recommendations both to help overcome barriers and to take advantage of opportunities. These recommendations address four broad concerns: policy actions to promote a collaborative approach to health-related performance measurement activities, operational principles to guide performance measurement, required investment in data systems and in training and technical assistance, and a national research agenda to improve the knowledge base for performance measurement. Major Policy Actions Needed National Collaboration The promotion of effective collaboration among multiple partners in the development of plans for performance measurement, the assessment of data needs, and the design (or redesign) of data systems to facilitate performance measurement must be a top priority. For example, focusing only on data to satisfy states' federal reporting requirements risks neglecting states' need for related but more detailed performance data that can be used in managing their program activities. Similarly, focusing unduly on individual categorical programs risks encouraging duplication of effort across programs, incompatibility across programs that rely on a program-specific data collection process, or lack of attention to essential functions (e.g., elements of the public health infrastructure) that are not specifically linked to individual programs. Although the proposal to transform significant portions of DHHS grant funding to states into PPGs specifically addressed a federal-state relationship, performance measurement requires a much broader collaborative partnership across federal, state, and local governments if it is to succeed. The panel sees opportunities to strengthen health-related performance measurement efforts through collaborative efforts that would enable all parties to learn from each other and would promote consensus regarding health outcome, process, and capacity measures that are appropriate at the federal, state, and local levels. These efforts must

OCR for page 132
--> entail a sharing of responsibilities, as well as opportunities for participation in decision making. Collaboration is needed not only across levels of government but also within each level. For example, more coordination among federal agencies in determining data needs would reduce duplication of effort at both the federal and state levels. Similarly, the more that states work together—in multistate regions, for example—and the more that agencies within the same state collaborate, the more likely they will be to learn from each other, share data, and develop comparable measures and definitions for data related to health outcomes and program activities. Although the panel has focused primarily on the public-sector participants in performance monitoring, many private-sector players should also be considered partners in the process to one degree or another. For example, organizations that represent state and local public health officials and related professionals are playing an important role in planning and implementing performance monitoring. Health care institutions and organizations have a stake in decisions that affect data reporting requirements. Managed care plans should have a strong interest in performance goals that focus on reducing acute and chronic disease risks in populations they serve (e.g., increasing the percentage of older adults who receive influenza vaccinations; increasing the percentage of persons with diabetes who have had their blood glucose control checked within the past 12 months). Attention should be given to including these private-sector partners, as appropriate, in the collaborative performance measurement and monitoring process. One model might be the broad community-level public-private partnership for performance monitoring described by the Institute of Medicine (1997b). To achieve the collaboration that the panel views as essential for effective performance measurement, the federal, state, and local partners in this effort should take the following recommended steps. 1. Federal, state, and local governments should commit to a common and national strategic goal of incorporating performance measurement into the practices of publicly funded health programs. Achieving the goals of performance measurement requires a public and collaborative commitment by partners at the federal, state, and local levels and a strategic planning process for implementation. The performance measurement concept will also require ongoing advocacy and promotion. All levels of government should identify and use opportunities for dissemination of data that will encourage the use of the data for performance measurement. Performance measures and support for the development of integrated data systems that can facilitate the application of those measures should be incorporated in strategic planning documents with national scope, such as the Healthy People 2010 report

OCR for page 132
--> currently in preparation. Where appropriate, these public-sector efforts should interact with related activities in the private sector. 2. Federal, state, and local governments, with input from private partner organizations, should plan and implement all steps of the performance measurement process in full collaboration with one another. A collaborative decision making process will help ensure subsequent support for processes and investment in outcomes at all levels of government. The PPG proposal explicitly recognized states as partners in the performance measurement process, but a stronger basis for a broader, ongoing, and longer-term dialogue is needed. Collaborative efforts must extend across the federal, state, and local levels and will require the participation of a multiplicity of stakeholders from each level to ensure that all important issues are addressed. Discussions at the national level among organizations representing the range of officials who will need to participate in performance monitoring efforts can help promote a shared national vision of and commitment to performance monitoring. The participants in such discussions should represent a mix of policy, program, and technical perspectives. Some of the relevant organizations are listed in Box 5-1. An example of this type of collaboration is the new National Public Health Performance Standards Program (see Halverson et al., 1998). This partnership involves the CDC, the National Association of County and City Health Officials (NACCHO), the Association of State and Territorial Health Officials (ASTHO), the National Association of Local Boards of Health, the Public Health Foundation, and the American Public Health Association. The Public Health Performance Standards Program is an effort to define and measure core public health performance at the local and state levels. Measures of local public health performance will be incorporated into a new version of the Assessment Protocol for Excellence in Public Health (APEX/PH) that NACCHO is developing to assist strategic planning efforts by local public health systems throughout the country. To facilitate such discussions and appropriate participation in decision making, a process is needed to bring together appropriate stakeholders in performance measurement. To accomplish this, the panel makes the following recommendation. 3. DHHS should work in partnership with members of the relevant groups representing policy, program, and technical officials of states and local entities to establish a process for developing policies and procedures that can facilitate the implementation of performance measurement efforts in health-related areas.

OCR for page 132
--> Box 5-1 Examples of National Organizations That Might Represent State and Local Perspectives in Discussions on Performance Measurement Listed below are examples of national organizations representing state and local policy, program, and technical perspectives that should participate in discussions regarding health-related performance measurement activities. The panel emphasizes that this is an illustrative list and should not be considered comprehensive or definitive. American Public Human Services Association Association of Maternal and Child Health Programs Association of Public Health Laboratories Association of State and Territorial Health Officials Association of State and Territorial Chronic Disease Program Directors Council of State and Territorial Epidemiologists Environmental Council of the States National Alliance of State and Territorial AIDS Directors National Association for Public Health Statistics and Information Systems National Association of Counties National Association of County and City Health Officials National Association of Health Data Organizations National Association of Local Boards of Health National Association of State Alcohol and Drug Abuse Directors National Association of State Emergency Medical Services Directors National Association of State Medicaid Directors National Association of State Mental Health Program Directors National Conference of State Legislators National Governors' Association State and Territorial Injury Prevention Directors Association Steps must be taken to initiate the consensus-building discussions that are needed to further the development of performance measurement efforts. A process for continuing these discussions must also be established. DHHS will be a key participant and may be an essential catalyst for this process, but must act as a partner with state and local stakeholders. Because many points of view must be considered fully and fairly, one approach might be to identify an interested party without a direct stake in the outcomes (e.g., a foundation, a university, a unit of the National Academy of Sciences) that can convene local, state, and national stakeholders in a neutral setting. Well-defined mechanisms should be established

OCR for page 132
--> for designating representatives of interested stakeholder groups to ensure that the views presented are authoritative and that channels exist for communicating with the stakeholder communities. A forum such as this might become a nationally recognized body through which many of this panel's recommendations might be advanced. Consultation among federal, state, and local program representatives can help ensure that performance measures are consistent with priorities for health programs across levels of government. For example, federal partners should gain a better understanding of attributes of data systems that are critical to states and others. Without this input, such systems are not likely to address state-specific issues and concerns, and opportunities to link or compare national surveys with related state surveys may be missed. The National Health Interview Survey, for example, produces national estimates but has not routinely had input from state representatives. Thus while this survey meets the needs of many federal programs, it is currently of little direct use to state health programs. The federal partners can also make important contributions to states and local entities in the planning and maintenance of performance monitoring systems. From their national vantage point, federal partners are often able to showcase information about a wider range of state activities than individual states could ascertain on their own. By serving as a clearinghouse for such information, the federal partners can disseminate problem-solving strategies developed in one state to other participants who might benefit from the information. Federal partners are also in a position to provide expertise and technical support to states (as was the case with the federal-state cooperation that contributed to the development of the national vital statistics system), and by serving as coordinating agents, they can promote comparability and quality through standardization of measures and data. A process that allows states and communities to learn from each other can be expected to aid them in making more effective use of the resources available to support performance measurement. As an extension of a broad national discussion, a similar process should be established to help key executive agencies at the state and local levels (e.g., health, human services, education, natural resources) share the vision on which performance measurement is based, develop the administrative procedures needed to ensure appropriate and timely reporting, and provide the leadership required to encourage appropriate information sharing and data integration. 4. Federal, state, and local governments should accept explicit responsibilities, determined in collaboration with other stakeholders, in return for their share in the governance of and benefits from broader efforts to improve performance monitoring.

OCR for page 132
--> 12. DHHS should work in partnership with state and local stakeholders to promote the development and adoption of standard definitions for performance measures and standards for associated data collection and data quality in performance measurement systems. Comparability of Data and Data Collection Methods While specific applications of performance measurement may vary, common definitions are critical for any communication and comparison of results. Similarly, while various data collection methods may be used to obtain performance data, standards for each method should be promoted in all cases to enhance data quality and promote comparability. For performance measures, agreement is needed on the definitions of the measures and on the data elements to be used to construct the measures. For a measure of adolescent tobacco use, for example, the definitions must cover age (e.g., 14–17 years, 13–18 years, 12–17 years), the scope of tobacco use (e.g., smoking cigarettes, all smoked tobacco, smoked and chewed tobacco), and the time reference (e.g., the past week, the past month, the past year). The choices made should reflect considerations of scientific and policy relevance and of practical factors such as data availability. Consideration should also be given to specifying the most appropriate method of data collection. Many data collection methods can be used to produce performance data. Potential methods include sample surveys, which can employ techniques such as mailed questionnaires, telephone interviews, and in-person interviews; clinical or administrative records from service encounters; reports to surveillance systems; and environmental monitoring. The data collection process cannot be specified in complete detail, but essential guidelines can be established so that differences in data collection methods will not undermine the comparability gained through the use of standard definitions for performance measures. If standards for performance measures and data collection are to be developed and accepted, a broad range of stakeholders must have a means of achieving consensus and harmonizing the implementation of their performance measurement activities. For example, DHHS has worked with the states to reach agreement on common definitions and practices for vital registration that permit the valid compilation of state data to produce national figures. Similar collaborative efforts in other areas should be able to produce sufficient comparability across state data systems so that greater reliance could be placed on aggregating state data to produce national measures, rather than requiring separate data collection systems at the federal level. Consensus-building mechanisms also exist in the private sector, as illustrated by the participation of many organizations and individuals in voluntary standards-setting activities, such as efforts under the auspices of the American Society for Testing and Materials (ASTM) to develop data and systems standards for certain types of clinical information. Another example is the Performance Measurement

OCR for page 132
--> Coordinating Council, formed in 1998 among the American Medical Accreditation Program, the Joint Commission on Accreditation of Healthcare Organizations, and the National Committee for Quality Assurance to coordinate performance measurement activities across the health care system (Joint Commission on Accreditation of Healthcare Organizations, 1998). The American National Standards Institute (ANSI) is another useful mechanism. It accredits a variety of consensus-based voluntary standards-setting activities and helps promulgate and maintain the standards that are adopted through those activities. If more jurisdictions used common measures for performance monitoring, greater numbers of valid comparisons could be made among those jurisdictions. Use of the same measures and data definitions would also encourage cost efficiencies by reducing the need to redesign data collection instruments, electronic processing protocols, and similar infrastructure elements. The advantages of this harmonization of measures and data collection practices for comparability and efficiency should not, however, be allowed to obscure the need for continued critical assessment of their appropriateness in general or for specific applications. Data Quality Data systems that support performance measurement must ensure adequate quality and appropriate handling of the data that are collected. Data quality can be compromised in many ways, such as inaccurate or incomplete reporting, poorly designed survey samples (including nonresponse), errors introduced in data processing procedures, and inaccurate calculation of measures. In addition to consideration of such problems within individual data systems, a broader information network such as that envisioned by the panel would require consideration of the differences in data quality across information systems and their impact on comparisons of performance measures. No data system or data set is ever perfect, and costs tend to rise rapidly as residual errors are further reduced. An appropriate balance must be found between the desired degree of data quality and the cost of achieving it. The requirements for data quality should be judged largely on the basis of the intended and anticipated uses of the data, but with some consideration of future uses not yet foreseen. At the same time, data systems should not be overdesigned in an attempt to anticipate all possible future uses. For example, efforts by cancer registries to collect accurate data on treatment have often been scaled back because the necessary quality and uniformity could not be attained on a population-wide basis. The effects on data quality of bias (e.g., survey response rates that differ across population groups) and random variation should be considered separately because these two factors have largely different sources and different implications. For example, bias that is constant over time may have little impact if policy considerations focus on time trends. Similarly, in situations where bias dominates random variation, there may be little profit in further reducing the random component of uncertainty. Increasing the size of a survey's sample, for example,

OCR for page 132
--> would tend to reduce random error but would not overcome the bias introduced by a factor such as widespread underreporting of tobacco use. Standards for data quality and the methods adopted to meet those standards should be based on informed assessments of how the data are to be used and the degree of accuracy and precision needed to serve those uses. The panel also advises consideration of policies and practices that will help achieve and maintain the desired level of data quality. Many observers agree that making data useful to those who produce them is one of the stronger incentives for ensuring that the data are of high-quality. Required Investment for an Information Network Performance measurement activities can be expected to impose new demands (e.g., additional data collection, data system development, and analysis of performance data) on those whose performance is being assessed. The panel sees a need to ensure that adequate resources, including funding, trained staff, and technology, are available to meet those demands, as well as to maintain the effective elements of current data systems. Moreover, because resources are often so limited, the panel is concerned that information system development will be forced to compete for funds with program services. To respond to these concerns, the panel recommends investments both in data systems and in training and technical assistance for health agency personnel, who will be expected to assume responsibility for planning and implementing performance measurement. Data Systems Existing health data systems provide a strong base for performance measurement, but because they have generally not been developed for this purpose, information gaps will exist. To address these gaps, the panel makes the following recommendation. 13. DHHS and state and local users of performance measurement data should each commit resources to reduce gaps in the supporting information systems. Adequate resources are needed to maintain key information systems that provide essential data for performance measurement and to enhance or develop new systems for data that cannot currently be produced. DHHS acknowledges that obtaining better data for performance measurement, especially state-level data, will require investments in data systems at the federal and state levels (U.S. Department of Health and Human Services, 1997). The panel urges specific attention to the need for resources to ensure and improve the availability of data for state- and local-level performance measures.

OCR for page 132
--> For example, many of the measures identified in the panel's first report depend on data produced by the BRFSS, for which a core survey is fielded in every state. Overall, grants from CDC's BRFSS program support about half of this collaborative state-CDC data system. Recent instability in CDC funding is of concern to the panel. Funding of $3.5 million for the 1996–1997 grant cycle was reduced by nearly one-quarter, to $2.7 million, for the 1997–1998 cycle (D. Nelson, Centers for Disease Control and Prevention, personal communication, 1998). An increase to $3.9 million for the 1998–1999 grants is encouraging, but the adequacy of the level and stability of federal support for this program should be assessed. A positive sign is the Maternal and Child Health Bureau's State Systems Development Initiative, which specifically allows states to apply for grants of up to $100,000 that can be used for such purposes as the development of integrated information systems or the collection of data for performance measures for the Maternal and Child Health Services Block Grant (Maternal and Child Health Bureau, 1998). Other important sources of state and local data that require adequate support include public health surveillance systems, such as vital records, notifiable disease systems, and registries for cancers, immunizations, and birth defects. Administrative and clinical data systems such as those that track client services in mental health and substance abuse programs should be assessed to determine their potential to contribute data at this level of geographic detail. The panel specifically noted in its first report that the mental health and substance abuse fields will have to develop ongoing information systems to provide most of the data necessary for performance measurement. SAMHSA's proposed Data Infrastructure Development funding should be helpful in this regard. The federal and state agencies that are establishing performance measurement requirements should support the development and operation of the information systems needed to produce the required performance data. Because the health information network envisioned by the panel would have the potential to produce data of value to a variety of audiences in the public and private sectors, those data users might provide additional resources. At the same time, the panel recognizes that new funding for strengthening and maintaining a health information network may be difficult to obtain and does not want to see funding for services compromised to support information systems. Therefore, innovative ways of using existing resources are needed. Some resources currently invested in data collection and analysis activities at the federal, state, and local levels could perhaps be redirected to produce more useful data. To this end, the panel recommends the following step. 14. DHHS should sponsor a review of the current array of federal, state, and local data collection and analysis activities to begin an assessment of how existing resources might be used most effectively to meet performance measurement and other needs for health data.

OCR for page 132
--> This review must include participation by appropriate state and local representatives. DHHS efforts to review and better coordinate some of its major national health surveys (U.S. Department of Health and Human Services, 1995) are a good start, but a broader approach will be needed. The recommended review of data collection and analysis activities must take into account data needs, interests, and resources at the local, state, and federal levels, and may benefit from private-sector participation. Duplication in current data collection and data systems should be examined carefully to determine whether information needs can be met more efficiently, potentially freeing resources to improve or expand weaker data systems. All participants in this effort will have some stake in existing information systems and may be reluctant to accept changes in those systems. However, successful development of a broader national health information network that can support performance measurement and other operational, managerial, and analytic activities will require coordination and compromise, as well as a broad, long-term commitment by the participants. The Environmental Protection Agency's (1998) One Stop Reporting Program, through which the agency is working with states to improve all aspects of environmental reporting, might serve as a model for a comprehensive assessment of health information systems. The panel notes that this program also allows states that are investing in environmental information system improvements to apply for demonstration grants of $500,000 to support such efforts. Training and Technical Assistance To have an effective information system for performance measurement, it will be essential to invest not only in the processes and equipment needed for data collection and analysis but also in enhancement of the knowledge and expertise of the information system staff and others who will produce and use performance data. The panel makes the following recommendations with regard to training and technical assistance. 15. To ensure the success of performance measurement, all stakeholders, with substantial leadership from DHHS, should contribute ongoing technical assistance, training, and resources to enhance state and local data systems and analytic capacity. To make effective performance measurement a reality, DHHS must actively support efforts to increase competency in analytic, programmatic, and leadership skills at the federal, state, and local levels. Resources must be allocated to fund and train staff at various levels, not just in the collection and analysis of perfor-

OCR for page 132
--> mance data but also in use of the results to bring about improvements in health programs. Throughout the nation, states and communities are assuming more responsibility for their own health through priority setting, development and implementation of programs, and performance measurement. As a consequence, there is a growing need for state and community capacity for data collection and analysis. Staff vary tremendously, however, in their knowledge of the relevant disciplines and methodologies (e.g., epidemiology, statistics, social science research) and in their experience with the use of data to plan, evaluate, and revise community programs. Moreover, state and local health agencies are often understaffed, making it difficult to take on the additional tasks in data collection and analysis that performance measurement may entail. A variety of approaches might help in capacity building. A useful first step might be facilitating access to the funds and expertise required to assess specific capacity needs. One community or state might have a pressing need for data processing resources (e.g., expertise and equipment), while another might need expertise in survey design or data analysis. Because of these differing needs, varying means of meeting them will be necessary. CDC's Public Health Training Network might be one mechanism, or model, through which training materials could be developed and delivered to state and local audiences. Distance-based learning programs, which reduce the need to travel by establishing video and audio connections to multiple sites, should be considered as a way of reaching this broad audience. Additional funding specifically for travel to major conferences and training programs could also be an effective investment. Direct interaction with colleagues provides learning opportunities that are not available with other approaches, such as use of self-guided instruction or distance-based learning. Access to scholarships and dissertation grants could assist staff in obtaining more advanced academic training. Other opportunities for state and local training and capacity building should also be explored. For example, university medical centers and schools of public health could develop collaborative programs that would provide state and local health departments with access to training and academic expertise in data collection and analysis while providing faculty and students with opportunities for field experience and research. Foundations, health plans, businesses, and others with interest or experience in performance measurement might become partners with state and local health agencies that are working to improve their skills. 16. DHHS should develop and maintain information technology expertise to assist states and communities as they use new technologies to improve the quality of and capacity for data collection, analysis, and dissemination.

OCR for page 132
--> Effective use of information technology requires considerable expertise in matters such as hardware, software, systems design and integration, and applications development. To the extent that DHHS can serve as a central and authoritative source for information and guidance on these technology matters, state and local health agencies will be able to make more rapid use of a broader range of expertise than they would be likely to assemble on their own. In the current highly competitive market for information technology personnel, many state and local agencies are at a disadvantage because their salaries tend to be relatively low. One example of work DHHS is already doing along these lines is the preparation of an investment analysis guide for states that want to combine categorical grant funds for the development of integrated health information systems (Centers for Disease Control and Prevention and Health Resources and Services Administration, 1998). Another specific area for support is exploration of the technical issues involved in merging data from diverse systems. Among the activities DHHS might support are pilot projects to test the use of specific equipment or processes. DHHS might also promote the development of a national clearinghouse or similar repository for information system applications and modular programming components (i.e., software objects). Access to such resources can make the systems development process simpler, faster, and less costly. A National Research Agenda The panel emphasizes that a multidisciplinary research program must be an integral part of any ongoing performance measurement activity for health-related programs. Because experience with performance measurement is still limited, studies are needed to improve understanding of what measures and methods of data collection are appropriate. Also crucial is further research to establish evidence regarding causal links between program interventions and desired outcomes. This evidence, which is currently limited in many fields, is essential for selecting demonstrably meaningful capacity, process, and risk status measures. For example, recent studies have shown that the identification and treatment of co-occurring mental health problems are an important predictor of positive outcomes in substance abuse treatment (Harrison and Asche, in press). Specifically, the panel makes the following recommendation. 17. Federal agencies, foundations, and other private-sector groups should develop and fund a research agenda to support performance measurement activities, including the testing of intervention effectiveness, the investigation of the links between program capacity and processes and program outcomes, the development of measures, the

OCR for page 132
--> refinement of data collection and information system technologies, and the use of performance measurement systems and performance-based decision making. Most users of performance measures will need to rely on the research of others to identify effective health interventions and measures of outcomes for these interventions, as well as to determine the effective use of performance measurement as a management and oversight tool. Therefore, the agendas of federal agencies and private organizations that fund health-related programs should include support for research intended to inform performance measurement in these areas. The range of studies must be broad enough to ensure that they are informative for a variety of settings, including the local, state, and national levels. Initiatives in the Agency for Health Care Policy and Research (AHCPR) under the designations of Strengthen Quality Measurement and Improvement and Support Improvements in Health Outcomes are models for the types of federal research agendas that could serve well for performance measurement research in other health areas. However, carrying out the research is not, in itself, sufficient for informing and improving the performance monitoring process. Significant findings must be communicated to the participating partners before they can be applied. Therefore, the panel offers this further recommendation. 18. DHHS, foundations and other private organizations, and other partners involved in performance measurement activities should contribute in an appropriate manner to a process of information gathering and dissemination to support the use of evidence-based performance measures. Resources are needed to support comprehensive reviews and rigorous analyses of the relevant scientific evidence and to produce evidence-based reports describing the scientific foundation applicable to the development of performance measures for use in publicly funded health programs. AHCPR currently sponsors a program to develop, use, and evaluate evidence-based tools and information related to clinical health care. A similar program that would apply to the areas of public health, mental health, and substance abuse would be immensely valuable for developing and refining performance measures and improving the performance monitoring system. Although states and communities generally have limited funds for conducting research or disseminating research results, they can make an important nonmonetary contribution to these efforts by facilitating studies undertaken by others and sharing information about their experiences in the use of performance measurement.

OCR for page 132
--> Final Observations The development and implementation of a broad national health information network that can support performance measurement will require a concerted effort. The leadership of DHHS and its agencies is critical, but these federal agencies must be part of a collaborative undertaking that includes the participation of stakeholders at the state and local levels, in the public and private sectors, and across a variety of program areas. The participants must be partners in seeking benefits from and sharing responsibility for the effective management and operation of a broader health information network. The information network envisioned by the panel should enhance the capacity of federal, state, and local health programs to meet performance measurement obligations and to use performance data and other information to achieve desired health outcomes. Several significant challenges lie ahead. Appropriate performance measures must be developed, and the data needed to use those measures must be available. Greater consensus must be achieved regarding standards for measures and data that will promote comparability in performance measurement. Necessary technical and analytic skills must be developed and applied to the creation and use of performance data. Sufficient financial and nonfinancial resources must be obtained to support both near-term efforts to introduce performance measurement activities and the longer-term data collection and analysis and research needed to sustain those efforts. This may appear to be a difficult set of challenges, but the panel believes that the need to address them is great and that the current commitment to performance measurement creates an opportunity to make significant progress toward meeting this need in the near future. References Broome, C.V., and C.E. Fox 1998. CDC/HRSA Grant Funding Flexibility for Integrated Health Information Systems. Grant funding transmittal letter. April 1, 1998. U.S. Department of Health and Human Services. http://www.hrsa.dhhs.gov/policy.htm (also at http://www.cdc.gov/funds/policy.htm) (April 21, 1998). Centers for Disease Control and Prevention 1997. Case definitions for infectious conditions under public health surveillance. MMWR 46(RR-10). Centers for Disease Control and Prevention and Health Resources and Services Administration 1998. Integrated Health Information Systems Investment Analysis Guide. http://www.hrsa.dhhs.gov/investment.htm#iv (also at http://www.cdc.gov/funds/invest7.htm) (April 21, 1998). Environmental Protection Agency 1998. One Stop Program Strategy and Grant Award Criteria. http://www.epa.gov/reinvent/onestop/strategy.htm (April 28, 1998). Gold, M. 1995. Miss or Match: How Well Do State Data Systems Meet State Health Policy Needs? Washington, D.C.: Mathematica Policy Research.

OCR for page 132
--> Halverson, P., R.M. Nicola, and E.L. Baker 1998. Performance measurement and accreditation of public health organizations: A call to action. Journal of Public Health Management and Practice 4(4):5–7. Harrison, P.A., and S. Asche in press. Comparison of substance abuse treatment outcomes for inpatients and outpatients. Journal of Substance Abuse Treatment. Institute of Medicine 1997a. The Computer-Based Patient Record: An Essential Technology for Health Care, Revised ed. R.S. Dick, E.B. Steen, and D.E. Detmer, eds. Committee on Improving the Patient Record. Washington, D.C.: National Academy Press. 1997b. Improving Health in the Community: A Role for Performance Monitoring. J.S. Durch, L.A. Bailey, and M.A. Stoto, eds. Committee on Using Performance Monitoring to Improve Community Health. Washington, D.C.: National Academy Press. Joint Commission on Accreditation of Healthcare Organizations 1998. Nation's Three Leading Health Care Quality Oversight Bodies to Coordinate Measurement Activities. Press release. May 19, 1998. http://www.jcaho.org/news/nb.htm (June 5, 1998). Lasker, R.D., B.L. Humphreys, and W.R. Braithwaite 1995. Making a Powerful Connection: The Health of the Public and the National Information Infrastructure. Report of the U.S. Public Health Service Public Health Data Policy Coordinating Committee. Washington, D.C. http://www.nlm.nih.gov/pubs/staffpubs/lo/makingpd.html (August 11, 1998). Maternal and Child Health Bureau 1998. State Systems Development Initiative (SSDI) Grant Application Guidance for FY98. U.S. Department of Health and Human Services, Health Resources and Services Administration. http://www.hhs.gov:80/hrsa/mchb/guidance.htm (June 4, 1998). Mendelson, D.N., and E.M. Salinsky 1997. Health information systems and the role of state government. Health Affairs 16(3):106–119. National Center for Health Statistics 1998. National Vital Statistics System. U.S. Department of Health and Human Services, Centers for Disease Control and Prevention. http://www.cdc.gov/nchswww/about/major/nvss/nvss.htm (July 6, 1998). National Committee for Quality Assurance 1997. HEDIS 3.0/1998. Vol. 4, A Roadmap for Information Systems: Evolving Systems to Support Performance Measurement. Washington, D.C.: National Committee for Quality Assurance. National Research Council 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. E.B. Perrin and J.J. Koshel, eds. Panel on Performance Measures and Data for Public Health Performance Partnership Grants, Committee on National Statistics. Washington, D.C.: National Academy Press. Perrin, E.B., L.G. Hart, S.M. Skillman, B. Paul, M.A. Hanken, and J. Hummel 1996. Health Information Systems and Their Role in Rural Health Services: Issues and Recommendations. Report to the Agency for Health Care Policy and Research (Contract #282-93-0036-4). Seattle: University of Washington. Public Health Foundation 1998. Measuring Health Objectives and Indicators: 1997 State and Local Capacity Survey. Washington, D.C.: Public Health Foundation. Roos, N.P., C.D. Black, N. Frohlich, C. Decoster, M.M. Cohen, D.J. Tataryn, C.A. Mustard, F. Toll, K.C. Carriere, C.A. Burchill, et al. 1995. A population-based health information system. Medical Care 33(12 suppl.):DS13–DS20.

OCR for page 132
--> Starr, P. 1997. Smart technology, stunted policy: Developing health information networks. Health Affairs 16(3):91–105. Thacker, S.B., and D.F. Stroup 1994. Future directions for comprehensive public health surveillance and health information systems in the United States. American Journal of Epidemiology 140:383–397. U.S. Department of Health and Human Services 1995. HHS Plan for Consolidation of Surveys. April 11, 1995. Washington, D.C. http://aspe.os.dhhs.gov/datacncl/srvyrptl.htm (January 12, 1998). 1997. 1997 Strategic Plan. September 30, 1997. Washington, D.C. 1998. Registry of State-Level Efforts to Integrate Health Information. http://aspe.os.dhhs.gov/statereg/index.htm (February 9, 1998).