Strategies for Supporting Performance Measurement Through a National Health Information Network
As the previous chapters have shown, further advances in performance measurement for publicly funded health programs will require thoughtful and continuing attention to a varied set of policy, programmatic, and data and information system issues. The current focus on performance-based accountability has helped highlight the limitations of existing health-related data and data sources. Despite their diversity and breadth, the data suitable for performance measurement are limited. In fact, the U.S. Department of Health and Human Services (DHHS) (1997:10) has acknowledged that lack of suitable data is a ''critical limiting factor" in developing departmental performance objectives as required under the Government Performance and Results Act (GPRA). The need for stronger and more coherent data systems for public health surveillance and personal health services is a concern at the federal, state, and local levels and in the private sector as well (e.g., Thacker and Stroup, 1994; Gold, 1995; Lasker et al., 1995; Mendelson and Salinsky, 1997; National Committee for Quality Assurance, 1997; Starr, 1997; U.S. Department of Health and Human Services, 1997; Public Health Foundation, 1998).
This final chapter reviews the panel's essential conclusions regarding performance partnership agreements, outlines its vision for a national health information network to facilitate performance measurement for publicly funded health programs, and recommends steps that can and should be taken to realize that vision. Although much in these conclusions and recommendations is applicable in principle to the broadest spectrum of health services in the public and private sectors, the panel has anchored its positions in the context of its discussions of
publicly funded health programs, including those in mental health, substance abuse, and various areas of public health.
Conclusions Regarding Performance Partnership Agreements
Examination of performance measurement in the context of the proposal for Performance Partnership Grants (PPGS) for health programs has led the panel to a set of general principles that it believes should guide further performance partnership efforts. First, the panel concludes that those who are affected by decisions resulting from the application of performance measures must share fully in the creation and selection of those measures, and, where possible, the measurement process. Representatives from federal, state, and local health and health-related agencies, and often segments of the general public, various private-sector interests, and health care provider groups, should all have a significant voice in the development, selection, assessment, interpretation, and use of performance measures.
Second, the panel concludes that state- and local-level data are essential to the achievement of federal goals for performance measurement and that support of data collection mechanisms to produce those data is important to the success of the system. Most national surveys, for example, are not able to produce state-specific estimates that can be used to compare state-level performance. The development of performance measures and the data systems on which they depend should be approached with a broad national perspective that considers the interdependencies across governmental levels.
Third, the panel believes that performance measurement information, resources, and processes should be organized so that states and communities can avoid unnecessary duplication of effort in developing new or enhanced data systems by using, to the extent possible, existing data systems that already serve other purposes and by working together to learn from each other. Collaborative efforts within and among states can reduce the unnecessary inconsistencies and incompatibilities that tend to arise in independently developed monitoring programs or data systems. The challenge is to foster this collaboration and cooperation while preserving the flexibility needed by individual states and communities to accommodate diversity in their programs and goals.
Finally, the panel stresses that performance measurement should focus on the overall goals of an activity, not seek to measure primarily the impact of a particular source of funding for that activity. Generally speaking, program goals are best represented by outcome measures, and most health outcomes are influenced by many more factors than those that might be linked to a single funding source, making inferences concerning individual funding sources problematic.
A National Information Network for Health-Related Data
Having considered both the general features of performance measurement and the specific context of publicly funded health-related programs, the panel has concluded that over the long-term, performance measurement for such programs will be made easier and more effective by the development of a broadly based national health information network that can promote a collaborative and coordinated approach across the local, state, and federal levels and can help in meeting diverse needs for a variety of health-related information, including performance data. This approach does not require the creation of an entirely new data system or a federally managed system. Instead, the panel envisions building on existing data systems operated by agencies at all levels of government, as well as looking to data systems in the private sector, to produce the information required for performance measurement.
The challenge is to develop a reasonably efficient and effective network from the current diversity of data sources and information needs. A collaborative approach, consistent with the partnership element of the PPG proposal, will be essential for success. Collaboration is needed to accommodate the missions of both population-based and personal health services and to facilitate harmonization of parallel activities that are conducted independently by individual states or communities (e.g., vital records systems, the Behavioral Risk Factor Surveillance System [BRFSS]). Among the most important tenets guiding the development of the envisioned information network should be the requirements that it meet real managerial and accountability needs; that it reflect important interdependencies and relationships across governmental and programmatic lines; and that it recognize that multiple, specialized data systems may no longer be affordable or consistent with other critical priorities.
In the current technological environment, an information network that can facilitate the transmission and aggregation of data from multiple sources without requiring the use of specific equipment or software is more feasible than ever before. Cooperation and collaboration are required, however, to establish agreement on such matters as electronic interfaces and data definitions. Effective use of technology also requires investments both in the development of staff expertise and in hardware and software. (See the discussion of this issue in Chapter 4.) Such investments have, however, been beyond the reach of many state and local health agencies.
The panel recognizes that developing a truly comprehensive information network of national scope is a massive, potentially overwhelming endeavor. The substantial overlap that the panel found in potential performance measures for the substantive areas it considered in its first report clearly argues against a narrow, disease-specific approach to data collection and analysis. However, the larger issues involved in developing a national information system of relevance to a
broad range of health and health care concerns will be much more difficult and complicated, both technically and politically, than the further development of disease-specific systems. This much broader undertaking will require the involvement of many DHHS agencies (e.g., Centers for Disease Control and Prevention [CDC], Health Resources and Services Administration [HRSA], Substance Abuse and Mental Health Services Administration, Food and Drug Administration, Health Care Financing Administration, National Institutes of Health), other federal agencies (e.g., Department of Agriculture, Environmental Protection Agency, Department of Education, Department of Housing and Urban Development, Department of Justice, Department of Transportation), and a similar array of state and local agencies, plus the private sector.
Establishing and maintaining the network envisioned by the panel will require a strong and continuing commitment by health agencies at the local, state, and federal levels. Even with such a commitment, this undertaking will almost certainly require an incremental approach. Many states do not yet have the capability to produce the information that would be required in any evaluation framework, and other states with more advanced data systems will have to make some changes to achieve compatibility with new standards that will evolve for a broad national network. Initial efforts might focus on enhancing existing information systems that serve either broad surveillance purposes or specific programmatic areas (e.g., children's health, substance abuse, mental health, chronic disease) or perhaps on conducting state or local demonstration projects to test the coordination of data systems across programmatic lines. Many of these public-sector programs must also rely on currently inadequate clinical data systems. Various observers (e.g., Institute of Medicine, 1997a; National Committee for Quality Assurance, 1997; Starr, 1997) have addressed the need for a commitment by health care organizations to make incremental improvements toward building a comprehensive information framework that can increase the health care industry's capacity to measure and improve performance. Successful development of information systems for performance measurement in the public or private sectors will require both time and financial investment from a variety of sources, as well as commitment and persistence.
Essential Features of a Health Information Network to Serve Different Levels of Decision Making
Given the complexity and scope of the task of developing the envisioned health information network, the panel believes that a clear vision of the intended long-term goal is required if the effort is to stay on course amid the vast amount of detail, variation, and difficult choices involved. Thus, the panel's vision for a cohesive, national health information network includes the following features.
National Collaboration The network should be developed through collabo-
ration among local, state, federal, and private efforts. A collaborative approach, in contrast to a more hierarchical or centralized model, requires the recognition of all participants as partners who can contribute to the success of the information network and the performance measurement activities it is intended to support, and whose information needs and program priorities must be taken into consideration. Participants should expect both to gain benefits from access to the information network and to assume responsibility for contributing to its effective operation, including ensuring the availability of adequate resources (e.g., funding, staff, data, information technology and expertise, commitment of policy makers).
With multiple participants at each level, a collaborative approach is necessary to promote efficient and effective data collection and use, as well as agreement on appropriate performance measures. The aim is to achieve a network that is national in scope without being specifically a federal enterprise. The voluntary state-based National Vital Statistics System (National Center for Health Statistics, 1998) serves as an example of a mature nationally collaborative network.
Linkage but not Consolidation The information network should be based on compatible structure and architecture to promote linkage of comparable data and sharing of information within and across the local, state, and national levels in a manner that is consistent with appropriate protections for personal privacy and the confidentiality and security of health-related data. The concept of a comprehensive network implies neither the desirability nor the need for a single national health database or information system.
Ability to Meet the Needs of Varied Users The network should, in the aggregate, support multiple purposes, including monitoring for performance-based accountability of population-based and personal health services, operation of health programs, and delivery and management of clinical care (including payment for services). To do so, it must encompass the full range of health-related services, from population-based and clinical preventive services through treatment and remediation. The network should be able to serve the information needs of managers, planners, health care providers, evaluators, policy makers, and the public at the national, state, and local levels. For example, data systems operated by states and communities can provide essential geographic detail and flexibility in data collection and analysis that are often not available from federal data systems, whereas federal data systems provide the broad national information that guides federal policy making and can serve as a reference point for assessing progress by individual states and communities.
For performance measurement specifically, the information network must be able to accommodate the differing data and information needs at various oversight levels. Some measures and data are important primarily or exclusively at the operating level, while other, often more limited information is appropriate for a more distant accountability audience. Plans for performance measurement should distinguish among these information needs and match the selection of measures accordingly.
Standardization of Data and Measures Critical to such a comprehensive network is the establishment of clear and common definitions of data elements, measures, and coding systems, and standard approaches to data collection so that information can be aggregated across multiple populations and regions and so that comparisons, where appropriate, can be made among populations and geographic areas.
The BRFSS, for example, illustrates the development of a standard survey questionnaire that is used independently in each state. Currently, this survey provides the primary means by which state-specific estimates of key behavioral risks are generated, but variations in survey methods among the states can limit the comparability of the data collected. To meet the full range of performance measurement requirements, new or modified data collection instruments with demonstrated reliability and validity or more standardized protocols may be needed to maximize the value of the information collected.
Appropriate Performance Measures For performance measurement, an information network should focus on measures of health outcomes and risk factors and on measures of processes and capacity that are widely recognized as linked to important health outcomes. For many program areas, the measures should be selected to represent multiple perspectives, specifically including that of the consumer (e.g., satisfaction with access to services). The range of measures must also be applicable to varied types of services (e.g., individual and population-based, inpatient and outpatient), to different age groups (e.g., youth, adult, and older adult), and to people of differing ethnic and cultural backgrounds.
Efficient and Effective Use of Resources The envisioned national health information network will require sufficient resources to operate effectively and support a wide range of users, but it should be organized efficiently to minimize its consumption of resources and any competition for resources with health programs and services. One consideration is efficient data collection. Certain types of data that can be used for performance measurement can be collected using various methods, such as consumer surveys, medical record reviews, or compilation of data from administrative records. Likewise, data might be collected at varying intervals (e.g., continuously, annually, every 2 years). In setting performance measurement requirements, careful judgments should be made about the intensity of the information required in terms of frequency of data collection, level of detail, and completeness of coverage (e.g., sample or census data).
Coordination and integration of information systems will often prove more efficient than the development and operation of program-specific systems. An information network should also take advantage of appropriate information available from sources that are not primarily health-related. For example, mental health programs may want information from corrections department databases on the number of people with serious mental illness in youth and adult corrections systems. Training and technical assistance must be an integral part of the frame-
work for an information network so that staff at all levels have the skills to manage data systems effectively and use the information that they produce.
Adaptability to Change The national health information network should be established on the premise that it is a developing tool that must be able to adapt easily and in a timely fashion to changes and enhancement as information sources grow, knowledge expands, or requirements change. One can expect performance measures to evolve and improve as the focus of interventions changes (e.g., the decline in lead poisoning as the inventory of homes with lead-based paint drops) and as new public health threats are identified.
With this framework for a multilevel, user-oriented national information network in mind, the discussion now turns to steps that can be taken to promote the development of such a network. In discussing these steps, the panel presents recommendations both to help overcome barriers and to take advantage of opportunities. These recommendations address four broad concerns: policy actions to promote a collaborative approach to health-related performance measurement activities, operational principles to guide performance measurement, required investment in data systems and in training and technical assistance, and a national research agenda to improve the knowledge base for performance measurement.
Major Policy Actions Needed
The promotion of effective collaboration among multiple partners in the development of plans for performance measurement, the assessment of data needs, and the design (or redesign) of data systems to facilitate performance measurement must be a top priority. For example, focusing only on data to satisfy states' federal reporting requirements risks neglecting states' need for related but more detailed performance data that can be used in managing their program activities. Similarly, focusing unduly on individual categorical programs risks encouraging duplication of effort across programs, incompatibility across programs that rely on a program-specific data collection process, or lack of attention to essential functions (e.g., elements of the public health infrastructure) that are not specifically linked to individual programs.
Although the proposal to transform significant portions of DHHS grant funding to states into PPGs specifically addressed a federal-state relationship, performance measurement requires a much broader collaborative partnership across federal, state, and local governments if it is to succeed. The panel sees opportunities to strengthen health-related performance measurement efforts through collaborative efforts that would enable all parties to learn from each other and would promote consensus regarding health outcome, process, and capacity measures that are appropriate at the federal, state, and local levels. These efforts must
entail a sharing of responsibilities, as well as opportunities for participation in decision making.
Collaboration is needed not only across levels of government but also within each level. For example, more coordination among federal agencies in determining data needs would reduce duplication of effort at both the federal and state levels. Similarly, the more that states work together—in multistate regions, for example—and the more that agencies within the same state collaborate, the more likely they will be to learn from each other, share data, and develop comparable measures and definitions for data related to health outcomes and program activities.
Although the panel has focused primarily on the public-sector participants in performance monitoring, many private-sector players should also be considered partners in the process to one degree or another. For example, organizations that represent state and local public health officials and related professionals are playing an important role in planning and implementing performance monitoring. Health care institutions and organizations have a stake in decisions that affect data reporting requirements. Managed care plans should have a strong interest in performance goals that focus on reducing acute and chronic disease risks in populations they serve (e.g., increasing the percentage of older adults who receive influenza vaccinations; increasing the percentage of persons with diabetes who have had their blood glucose control checked within the past 12 months). Attention should be given to including these private-sector partners, as appropriate, in the collaborative performance measurement and monitoring process. One model might be the broad community-level public-private partnership for performance monitoring described by the Institute of Medicine (1997b).
To achieve the collaboration that the panel views as essential for effective performance measurement, the federal, state, and local partners in this effort should take the following recommended steps.
1. Federal, state, and local governments should commit to a common and national strategic goal of incorporating performance measurement into the practices of publicly funded health programs.
Achieving the goals of performance measurement requires a public and collaborative commitment by partners at the federal, state, and local levels and a strategic planning process for implementation. The performance measurement concept will also require ongoing advocacy and promotion. All levels of government should identify and use opportunities for dissemination of data that will encourage the use of the data for performance measurement. Performance measures and support for the development of integrated data systems that can facilitate the application of those measures should be incorporated in strategic planning documents with national scope, such as the Healthy People 2010 report
currently in preparation. Where appropriate, these public-sector efforts should interact with related activities in the private sector.
2. Federal, state, and local governments, with input from private partner organizations, should plan and implement all steps of the performance measurement process in full collaboration with one another.
A collaborative decision making process will help ensure subsequent support for processes and investment in outcomes at all levels of government. The PPG proposal explicitly recognized states as partners in the performance measurement process, but a stronger basis for a broader, ongoing, and longer-term dialogue is needed. Collaborative efforts must extend across the federal, state, and local levels and will require the participation of a multiplicity of stakeholders from each level to ensure that all important issues are addressed.
Discussions at the national level among organizations representing the range of officials who will need to participate in performance monitoring efforts can help promote a shared national vision of and commitment to performance monitoring. The participants in such discussions should represent a mix of policy, program, and technical perspectives. Some of the relevant organizations are listed in Box 5-1.
An example of this type of collaboration is the new National Public Health Performance Standards Program (see Halverson et al., 1998). This partnership involves the CDC, the National Association of County and City Health Officials (NACCHO), the Association of State and Territorial Health Officials (ASTHO), the National Association of Local Boards of Health, the Public Health Foundation, and the American Public Health Association. The Public Health Performance Standards Program is an effort to define and measure core public health performance at the local and state levels. Measures of local public health performance will be incorporated into a new version of the Assessment Protocol for Excellence in Public Health (APEX/PH) that NACCHO is developing to assist strategic planning efforts by local public health systems throughout the country.
To facilitate such discussions and appropriate participation in decision making, a process is needed to bring together appropriate stakeholders in performance measurement. To accomplish this, the panel makes the following recommendation.
3. DHHS should work in partnership with members of the relevant groups representing policy, program, and technical officials of states and local entities to establish a process for developing policies and procedures that can facilitate the implementation of performance measurement efforts in health-related areas.
Box 5-1 Examples of National Organizations That Might Represent State and Local Perspectives in Discussions on Performance Measurement
Listed below are examples of national organizations representing state and local policy, program, and technical perspectives that should participate in discussions regarding health-related performance measurement activities. The panel emphasizes that this is an illustrative list and should not be considered comprehensive or definitive.
American Public Human Services Association
Association of Maternal and Child Health Programs
Association of Public Health Laboratories
Association of State and Territorial Health Officials
Association of State and Territorial Chronic Disease Program Directors
Council of State and Territorial Epidemiologists
Environmental Council of the States
National Alliance of State and Territorial AIDS Directors
National Association for Public Health Statistics and Information Systems
National Association of Counties
National Association of County and City Health Officials
National Association of Health Data Organizations
National Association of Local Boards of Health
National Association of State Alcohol and Drug Abuse Directors
National Association of State Emergency Medical Services Directors
National Association of State Medicaid Directors
National Association of State Mental Health Program Directors
National Conference of State Legislators
National Governors' Association
State and Territorial Injury Prevention Directors Association
Steps must be taken to initiate the consensus-building discussions that are needed to further the development of performance measurement efforts. A process for continuing these discussions must also be established. DHHS will be a key participant and may be an essential catalyst for this process, but must act as a partner with state and local stakeholders. Because many points of view must be considered fully and fairly, one approach might be to identify an interested party without a direct stake in the outcomes (e.g., a foundation, a university, a unit of the National Academy of Sciences) that can convene local, state, and national stakeholders in a neutral setting. Well-defined mechanisms should be established
for designating representatives of interested stakeholder groups to ensure that the views presented are authoritative and that channels exist for communicating with the stakeholder communities. A forum such as this might become a nationally recognized body through which many of this panel's recommendations might be advanced.
Consultation among federal, state, and local program representatives can help ensure that performance measures are consistent with priorities for health programs across levels of government. For example, federal partners should gain a better understanding of attributes of data systems that are critical to states and others. Without this input, such systems are not likely to address state-specific issues and concerns, and opportunities to link or compare national surveys with related state surveys may be missed. The National Health Interview Survey, for example, produces national estimates but has not routinely had input from state representatives. Thus while this survey meets the needs of many federal programs, it is currently of little direct use to state health programs.
The federal partners can also make important contributions to states and local entities in the planning and maintenance of performance monitoring systems. From their national vantage point, federal partners are often able to showcase information about a wider range of state activities than individual states could ascertain on their own. By serving as a clearinghouse for such information, the federal partners can disseminate problem-solving strategies developed in one state to other participants who might benefit from the information. Federal partners are also in a position to provide expertise and technical support to states (as was the case with the federal-state cooperation that contributed to the development of the national vital statistics system), and by serving as coordinating agents, they can promote comparability and quality through standardization of measures and data.
A process that allows states and communities to learn from each other can be expected to aid them in making more effective use of the resources available to support performance measurement. As an extension of a broad national discussion, a similar process should be established to help key executive agencies at the state and local levels (e.g., health, human services, education, natural resources) share the vision on which performance measurement is based, develop the administrative procedures needed to ensure appropriate and timely reporting, and provide the leadership required to encourage appropriate information sharing and data integration.
4. Federal, state, and local governments should accept explicit responsibilities, determined in collaboration with other stakeholders, in return for their share in the governance of and benefits from broader efforts to improve performance monitoring.
A well-designed and effectively operating performance monitoring system offers benefits for all of its participants. To achieve those benefits, however, participants must also accept an appropriate share of responsibility for the design and maintenance of the system and for an investment of resources so that sufficient capacity is available for the system to be used effectively. Among these responsibilities is the provision of adequate staff, facilities, and technical competence (e.g., in epidemiology, computer programming) to meet the objectives and standards of the data system. Moreover, participants at all levels must expect to bear a fair share of the financial cost in proportion to the degree to which the system meets their specific needs. All participants have a responsibility to work for compromise solutions in such matters as uniformity in definitions and procedures, choice of data items and data collection methods, and timeliness and format of the data forwarded to other collaborators.
Integration of Data Systems
Some states are beginning to pursue a more integrated approach to health data (U.S. Department of Health and Human Services, 1998), especially data related to health care services. However, the categorical nature of much of the federal funding for state and local health-related programs has often encouraged both a fragmented approach to health problems and the development of program-specific data systems and reporting requirements. Even though programmatic funding streams are likely to remain a prominent feature of federal funding, additional opportunities are needed at the state and local levels to improve data systems by coordinating and integrating a broader array of health data. Specifically, the panel makes the following recommendations.
5. DHHS should lead efforts to integrate data systems across categorical health program lines.
Coordination and integration of data systems across program areas promises increased quality, efficiency, timeliness, and usefulness for performance measurement and other purposes. As noted above, however, the categorical nature of federal funding streams and related reporting requirements often limits the opportunity for and value of such approaches. DHHS should develop specific incentives to encourage programs with separate categorical funding streams to develop integrated data systems. For example, federal requirements for the collection and transmission of data on HIV, tuberculosis, sexually transmitted diseases, and vaccine-preventable diseases should be mutually compatible and sufficiently flexible to ensure that states can easily collect and transmit the data using a single notifiable disease reporting system.
There is evidence that some federal agencies are recognizing the limitations
of the categorical approach that has to date shaped information system funding and development. CDC and HRSA have endorsed the use of their categorical grant funds in the development of integrated health information systems to derive the benefits of more comprehensive measures of health status, greater efficiency in managing programs, and more timely information for decision making (Broome and Fox, 1998). ASTHO and NACCHO have also endorsed this policy. The panel encourages all DHHS agencies that fund program activities at the state and local levels to promote this broader perspective in planning information system changes and to facilitate the use of funds from federal programs to implement those changes.
CDC and HRSA note, however, that their new policy does not supersede current legal restrictions limiting the use of grant funds for planning and evaluation or administrative expenses. The Mental Health Block Grant, which is administered by the Substance Abuse and Mental Health Services Administration (SAMHSA), limits states to using no more than 5 percent of these funds for such administrative functions as information system development and operation. To facilitate the implementation of recommendation 5 above, the panel also makes the following recommendation.
6. DHHS, in collaboration with state and local partners, should review restrictions on the use of grant funds to determine whether they represent a significant barrier to progress in the development of integrated health information systems. If so, DHHS should pursue changes in the terms of those grant programs that would permit greater flexibility in the use of the funds.
The review of any grant program should involve representative stakeholders from the national, state, and local levels.
The rapidly evolving information technologies that will enhance the feasibility of performance measurement and the health information network envisioned by the panel (see Chapter 4) are also creating a need for standards that will facilitate the transmission and aggregation of data from multiple sources without requiring the use of specific equipment or operating systems and software. The development of capabilities for linking and merging electronic data from disparate sources (e.g., administrative records, patient records, vital statistics records, disease registries, disease surveillance systems, periodic surveys, needs assessment studies, social indicator systems) will remain important. The Internet and the World Wide Web offer increasing opportunities to transmit and use data easily and rapidly (see Chapter 4). Furthermore, as the cost of desktop computers drops and their capabilities increase, users can perform more complex analyses,
although many may have limited training to ensure that those analyses are appropriate (see the discussion of training needs below).
While the panel sees great opportunities in these technologies, it also sees a need for explicit efforts to oversee and guide their application to performance measurement to ensure their appropriate and effective use. Specifically, the panel makes the following recommendation.
7. DHHS should provide leadership in the development and use of data transmission standards and of new information technologies to collect, analyze, and disseminate health-related data.
The information revolution is proceeding at such a rapid pace that many state and local health agencies cannot remain current. In addition, standardization is a key feature of successful integration and interoperability of data systems. DHHS should serve as a catalyst for consensus building on standards for data collection and transmission and as a technical assistance resource for jurisdictions seeking help and guidance on the appropriate application of new information technologies for a broad range of health-related data. The standards development process mandated by the Health Insurance Portability and Accountability Act (HIPAA) is achieving substantial progress for administrative transactions of health plans; however, these standards are not likely to be sufficient for performance measurement because they are not intended to address data and data systems for the population-based services provided by most state and local health departments. As standards are developed, attention should also be given to opportunities to facilitate the linkage of data sets. Moreover, all activities related to the development and use of information technologies must address the protection of individual privacy and assurance of the confidentiality and security of health-related data (see Chapter 4).
In addition to considering policy actions required for the implementation of an effective performance measurement information system, the panel has developed recommendations regarding three operational principles that it believes are important for the success of such a system: involve a broad range of agencies, build on existing information systems, and provide for ongoing review and standardization of performance measures and data systems.
Involve a Broad Range of Public Agencies
8. As states and communities work to implement performance monitoring systems for health-related programs, they should ensure that
all relevant public agencies, including those outside traditional health areas, have the opportunity to participate.
The development of a performance monitoring system requires a commitment to understanding the range of substantive factors that influence desired outcomes for health and well-being, as well as the administrative, analytic, and technical resources needed to collect and use performance measures. The process must ensure participation not only by state and local health agencies traditionally responsible for the health of the public, but also those with programmatic responsibilities in relevant nonhealth areas (e.g., criminal justice, housing, transportation) and those that collect and manage data on basic socioeconomic characteristics of the population and the state or community (e.g., population estimates, economic development data). A community health improvement process embracing this broad involvement in performance measurement has been described by the Institute of Medicine (1997b), and the implications of such an approach for population information systems are explored by Roos and colleagues (1995).
Build on Existing Information Systems
A health information network should facilitate access to all available information on populations that could be used to track health risks and promote the health of communities. Substantial amounts of health-related data are currently being collected through existing data systems at the federal, state, and local levels. Many states and localities have the capability for and are already invested in the collection of performance data because these data comprise a subset of the information needed to operate efficient and effective programs. Although few existing data systems have been designed specifically for performance measurement, they nonetheless provide an essential base from which to build an information network that can meet a broad range of health information needs, including performance data. Use of existing data systems to the extent feasible would provide an important efficiency in the establishment of a performance measurement system. Thus, the panel makes the following recommendation.
9. When possible, partners should obtain performance measurement information from existing or enhanced federal, state, and local information systems.
Instead of creating an entirely new data system for the purpose of performance measurement, data needs should be met to the extent possible through the use of existing systems. Systems such as vital records, the notifiable disease systems, adult and youth behavioral risk factor surveillance, cancer registries, and immunization registries can provide data for national, state, and local estimates regarding health risks, disease incidence and prevalence, treatment, and
health outcomes and should serve as the foundation for performance measurement.
When existing data systems cannot meet information needs for performance measurement, investments must be made to enhance those systems or, in cases where enhancement is not feasible or sufficient, to develop new data systems. These investments can potentially yield multiple benefits across programs and across national, state, and local lines, but modifications should reflect a careful balancing of costs and benefits. For example, enhanced technical assistance and improved standardization of BRFSS methodology may involve costs and loss of some flexibility by states, but it may also produce more directly comparable state specific estimates and data that could be aggregated to yield meaningful national estimates. The panel supports efforts to make the data systems in a health information network as efficient and effective as possible.
As noted in conjunction with recommendation 8, an information network should also provide access to data systems that are beyond the purview of health agencies but contain data valuable for tracking health risks and outcomes. For example, law enforcement and transportation department data on traffic accident locations and times have been used to identify high-risk intersections and highways. By linking these data with emergency medical services records and hospital emergency room data, a health department may be able to distinguish physical risks in the community (e.g., poorly marked curves and intersections) from behavioral risks (e.g., times of day when alcohol-related injuries are more likely to occur) (Perrin et al., 1996). Similarly, access to law enforcement data on drug-related crimes is crucial for monitoring the effectiveness of some substance abuse treatment services.
Performance measures themselves require further development and standardization. For some program areas, a basic set of outcome measures is not yet recognized. For others, the evidence base for capacity, process, and risk status measures is still developing. In most program areas, understanding of the relationship between measures that make sense at the state or local level and national measures is still limited. Furthermore, health needs and program priorities can be expected to change over time. Thus, the selection and review of performance measures and the development of related standards must be a continuing effort that takes into account the evolving knowledge base on which such measures rest, changing needs for and opportunities to obtain data, and the changing program environment in which performance measures will be used.
Once suitable measures have been identified, the adoption of standard definitions and procedures for data collection will enhance the comparability of performance data over time and across states and localities. One of the challenges for the health information network envisioned by the panel will be balancing flexibil-
ity against standardization. Flexibility is needed to accommodate a broad range of measures as they continue to evolve. Standardization is required to produce data and measures that permit comparisons among communities, states, and regions, and that allow data to be tracked over time or pooled for regional and national estimates. To promote the continued development and improved comparability of performance measures, the panel recommends the following steps.
10. DHHS, in partnership with state and local stakeholders, should lead the implementation of a process for ongoing development and review of performance measures to be used in conjunction with state and local health programs.
The recommended process should be a collaborative effort that includes participation by federal, state, and local health agencies, plus representatives of consumers and relevant private-sector interests (e.g., health plans, clinicians, insurers, businesses, foundations, patient advocates, grass roots organizations, and the general public). Involvement of a broad range of stakeholders is recommended to ensure that performance measures are consistent with state and local public health priorities and that policy, programmatic, and technical perspectives are all represented. The ongoing collaboration between CDC and the Council of State and Territorial Epidemiologists for periodic review and revision of the case definitions of specific infectious diseases might serve as a model for these efforts (Centers for Disease Control and Prevention, 1997).
This process must also take into account the differing stages of development among various health fields. For example, the mental health field should first emphasize the development of a standardized framework for the evaluation of mental health services that reflects an understanding of the relationship of outcomes to program capacity and processes. In other fields, such as immunization, the framework for assessing program activities is more fully developed. For those fields in which greater progress has been made in defining outcome measures, the process should focus on reviewing and refining existing performance measures to ensure that they reflect current knowledge regarding causal relationships between outcomes and processes and capacity.
In some fields (e.g., substance abuse and mental health), the development and review of performance measures must be preceded by efforts to build consensus on a framework for assessing health outcomes. The panel's earlier efforts to identify performance measures in these fields were hindered by the current diversity of evaluation systems among states, provider organizations, and accrediting bodies and by the resulting inconsistencies in measurement and data collection. A broadly based effort will be needed to bring together relevant federal, state, and local agencies, professional groups, and consumer interests to seek consensus on the dimensions of health outcomes, processes, and capacities that should serve as the basis for performance assessment in these fields. Chapter 3
notes some of the work that has been initiated in the fields of mental health and substance abuse.
The recommended review process should also examine both the positive and negative effects that the use of these performance measures in a performance monitoring system might have (or might be expected to have) on program activities and outcomes. Positive effects would be anticipated, and should be verified, for those measures that encourage programs to implement evidence-based best practices. Other measures, however, might conceivably result in a negative impact if, for example, they encouraged restrictions in program services for hard-to-reach or high-risk groups to achieve the appearance of better performance.
11. DHHS, in partnership with state and local stakeholders, should lead a process for assembling and evaluating sets of performance measures from which users can identify and agree upon those appropriate for specific applications.
Because of data limitations, differing health problems, and differing priorities, users of performance measures need to select a suitable subset of health outcome measures from a larger array of measures that have been found to meet the basic tests of validity, reliability, responsiveness, and data adequacy. States and localities may reasonably pursue many different strategies to target a single health outcome, so users should have an even larger number of process and capacity measures associated with these outcomes from which to choose. Efforts by private nonprofit groups to identify measures suitable for outcomes research in health care (e.g., the Medical Outcomes Trust) might serve as models for a process for assembling sets of performance measures for publicly funded health programs.
The panel's first report (National Research Council, 1997) provided examples of measures of health outcomes, program processes, and capacities that could be used to monitor performance in specific program areas. The panel concluded that it could not propose a definitive list of measures because few states are likely to have the data necessary to support every measure, and individual agencies are likely to have priorities in addition to (or different from) those reflected in the measures selected by the panel.
Instead, one of the panel's principal goals in its first report was to provide an analytic framework that could be used to assess the appropriateness of specific outcome, process, and capacity measures proposed for performance monitoring purposes, and could be applied over time to modify or replace measures as new or better ones were identified. Using this framework, it would be possible to develop a menu from which suitable measures could be chosen. An effort should be made, however, to associate particular program goals and strategies with specific outcome, risk status, process, and capacity measures so that identical activities related to those goals and strategies can be monitored using the same measures.
12. DHHS should work in partnership with state and local stakeholders to promote the development and adoption of standard definitions for performance measures and standards for associated data collection and data quality in performance measurement systems.
Comparability of Data and Data Collection Methods While specific applications of performance measurement may vary, common definitions are critical for any communication and comparison of results. Similarly, while various data collection methods may be used to obtain performance data, standards for each method should be promoted in all cases to enhance data quality and promote comparability.
For performance measures, agreement is needed on the definitions of the measures and on the data elements to be used to construct the measures. For a measure of adolescent tobacco use, for example, the definitions must cover age (e.g., 14–17 years, 13–18 years, 12–17 years), the scope of tobacco use (e.g., smoking cigarettes, all smoked tobacco, smoked and chewed tobacco), and the time reference (e.g., the past week, the past month, the past year). The choices made should reflect considerations of scientific and policy relevance and of practical factors such as data availability.
Consideration should also be given to specifying the most appropriate method of data collection. Many data collection methods can be used to produce performance data. Potential methods include sample surveys, which can employ techniques such as mailed questionnaires, telephone interviews, and in-person interviews; clinical or administrative records from service encounters; reports to surveillance systems; and environmental monitoring. The data collection process cannot be specified in complete detail, but essential guidelines can be established so that differences in data collection methods will not undermine the comparability gained through the use of standard definitions for performance measures.
If standards for performance measures and data collection are to be developed and accepted, a broad range of stakeholders must have a means of achieving consensus and harmonizing the implementation of their performance measurement activities. For example, DHHS has worked with the states to reach agreement on common definitions and practices for vital registration that permit the valid compilation of state data to produce national figures. Similar collaborative efforts in other areas should be able to produce sufficient comparability across state data systems so that greater reliance could be placed on aggregating state data to produce national measures, rather than requiring separate data collection systems at the federal level.
Consensus-building mechanisms also exist in the private sector, as illustrated by the participation of many organizations and individuals in voluntary standards-setting activities, such as efforts under the auspices of the American Society for Testing and Materials (ASTM) to develop data and systems standards for certain types of clinical information. Another example is the Performance Measurement
Coordinating Council, formed in 1998 among the American Medical Accreditation Program, the Joint Commission on Accreditation of Healthcare Organizations, and the National Committee for Quality Assurance to coordinate performance measurement activities across the health care system (Joint Commission on Accreditation of Healthcare Organizations, 1998). The American National Standards Institute (ANSI) is another useful mechanism. It accredits a variety of consensus-based voluntary standards-setting activities and helps promulgate and maintain the standards that are adopted through those activities.
If more jurisdictions used common measures for performance monitoring, greater numbers of valid comparisons could be made among those jurisdictions. Use of the same measures and data definitions would also encourage cost efficiencies by reducing the need to redesign data collection instruments, electronic processing protocols, and similar infrastructure elements. The advantages of this harmonization of measures and data collection practices for comparability and efficiency should not, however, be allowed to obscure the need for continued critical assessment of their appropriateness in general or for specific applications.
Data Quality Data systems that support performance measurement must ensure adequate quality and appropriate handling of the data that are collected. Data quality can be compromised in many ways, such as inaccurate or incomplete reporting, poorly designed survey samples (including nonresponse), errors introduced in data processing procedures, and inaccurate calculation of measures. In addition to consideration of such problems within individual data systems, a broader information network such as that envisioned by the panel would require consideration of the differences in data quality across information systems and their impact on comparisons of performance measures.
No data system or data set is ever perfect, and costs tend to rise rapidly as residual errors are further reduced. An appropriate balance must be found between the desired degree of data quality and the cost of achieving it. The requirements for data quality should be judged largely on the basis of the intended and anticipated uses of the data, but with some consideration of future uses not yet foreseen. At the same time, data systems should not be overdesigned in an attempt to anticipate all possible future uses. For example, efforts by cancer registries to collect accurate data on treatment have often been scaled back because the necessary quality and uniformity could not be attained on a population-wide basis.
The effects on data quality of bias (e.g., survey response rates that differ across population groups) and random variation should be considered separately because these two factors have largely different sources and different implications. For example, bias that is constant over time may have little impact if policy considerations focus on time trends. Similarly, in situations where bias dominates random variation, there may be little profit in further reducing the random component of uncertainty. Increasing the size of a survey's sample, for example,
would tend to reduce random error but would not overcome the bias introduced by a factor such as widespread underreporting of tobacco use.
Standards for data quality and the methods adopted to meet those standards should be based on informed assessments of how the data are to be used and the degree of accuracy and precision needed to serve those uses. The panel also advises consideration of policies and practices that will help achieve and maintain the desired level of data quality. Many observers agree that making data useful to those who produce them is one of the stronger incentives for ensuring that the data are of high-quality.
Required Investment for an Information Network
Performance measurement activities can be expected to impose new demands (e.g., additional data collection, data system development, and analysis of performance data) on those whose performance is being assessed. The panel sees a need to ensure that adequate resources, including funding, trained staff, and technology, are available to meet those demands, as well as to maintain the effective elements of current data systems. Moreover, because resources are often so limited, the panel is concerned that information system development will be forced to compete for funds with program services. To respond to these concerns, the panel recommends investments both in data systems and in training and technical assistance for health agency personnel, who will be expected to assume responsibility for planning and implementing performance measurement.
Existing health data systems provide a strong base for performance measurement, but because they have generally not been developed for this purpose, information gaps will exist. To address these gaps, the panel makes the following recommendation.
13. DHHS and state and local users of performance measurement data should each commit resources to reduce gaps in the supporting information systems.
Adequate resources are needed to maintain key information systems that provide essential data for performance measurement and to enhance or develop new systems for data that cannot currently be produced. DHHS acknowledges that obtaining better data for performance measurement, especially state-level data, will require investments in data systems at the federal and state levels (U.S. Department of Health and Human Services, 1997).
The panel urges specific attention to the need for resources to ensure and improve the availability of data for state- and local-level performance measures.
For example, many of the measures identified in the panel's first report depend on data produced by the BRFSS, for which a core survey is fielded in every state. Overall, grants from CDC's BRFSS program support about half of this collaborative state-CDC data system. Recent instability in CDC funding is of concern to the panel. Funding of $3.5 million for the 1996–1997 grant cycle was reduced by nearly one-quarter, to $2.7 million, for the 1997–1998 cycle (D. Nelson, Centers for Disease Control and Prevention, personal communication, 1998). An increase to $3.9 million for the 1998–1999 grants is encouraging, but the adequacy of the level and stability of federal support for this program should be assessed. A positive sign is the Maternal and Child Health Bureau's State Systems Development Initiative, which specifically allows states to apply for grants of up to $100,000 that can be used for such purposes as the development of integrated information systems or the collection of data for performance measures for the Maternal and Child Health Services Block Grant (Maternal and Child Health Bureau, 1998).
Other important sources of state and local data that require adequate support include public health surveillance systems, such as vital records, notifiable disease systems, and registries for cancers, immunizations, and birth defects. Administrative and clinical data systems such as those that track client services in mental health and substance abuse programs should be assessed to determine their potential to contribute data at this level of geographic detail. The panel specifically noted in its first report that the mental health and substance abuse fields will have to develop ongoing information systems to provide most of the data necessary for performance measurement. SAMHSA's proposed Data Infrastructure Development funding should be helpful in this regard.
The federal and state agencies that are establishing performance measurement requirements should support the development and operation of the information systems needed to produce the required performance data. Because the health information network envisioned by the panel would have the potential to produce data of value to a variety of audiences in the public and private sectors, those data users might provide additional resources.
At the same time, the panel recognizes that new funding for strengthening and maintaining a health information network may be difficult to obtain and does not want to see funding for services compromised to support information systems. Therefore, innovative ways of using existing resources are needed. Some resources currently invested in data collection and analysis activities at the federal, state, and local levels could perhaps be redirected to produce more useful data. To this end, the panel recommends the following step.
14. DHHS should sponsor a review of the current array of federal, state, and local data collection and analysis activities to begin an assessment of how existing resources might be used most effectively to meet performance measurement and other needs for health data.
This review must include participation by appropriate state and local representatives.
DHHS efforts to review and better coordinate some of its major national health surveys (U.S. Department of Health and Human Services, 1995) are a good start, but a broader approach will be needed. The recommended review of data collection and analysis activities must take into account data needs, interests, and resources at the local, state, and federal levels, and may benefit from private-sector participation. Duplication in current data collection and data systems should be examined carefully to determine whether information needs can be met more efficiently, potentially freeing resources to improve or expand weaker data systems. All participants in this effort will have some stake in existing information systems and may be reluctant to accept changes in those systems. However, successful development of a broader national health information network that can support performance measurement and other operational, managerial, and analytic activities will require coordination and compromise, as well as a broad, long-term commitment by the participants.
The Environmental Protection Agency's (1998) One Stop Reporting Program, through which the agency is working with states to improve all aspects of environmental reporting, might serve as a model for a comprehensive assessment of health information systems. The panel notes that this program also allows states that are investing in environmental information system improvements to apply for demonstration grants of $500,000 to support such efforts.
Training and Technical Assistance
To have an effective information system for performance measurement, it will be essential to invest not only in the processes and equipment needed for data collection and analysis but also in enhancement of the knowledge and expertise of the information system staff and others who will produce and use performance data. The panel makes the following recommendations with regard to training and technical assistance.
15. To ensure the success of performance measurement, all stakeholders, with substantial leadership from DHHS, should contribute ongoing technical assistance, training, and resources to enhance state and local data systems and analytic capacity.
To make effective performance measurement a reality, DHHS must actively support efforts to increase competency in analytic, programmatic, and leadership skills at the federal, state, and local levels. Resources must be allocated to fund and train staff at various levels, not just in the collection and analysis of perfor-
mance data but also in use of the results to bring about improvements in health programs.
Throughout the nation, states and communities are assuming more responsibility for their own health through priority setting, development and implementation of programs, and performance measurement. As a consequence, there is a growing need for state and community capacity for data collection and analysis. Staff vary tremendously, however, in their knowledge of the relevant disciplines and methodologies (e.g., epidemiology, statistics, social science research) and in their experience with the use of data to plan, evaluate, and revise community programs. Moreover, state and local health agencies are often understaffed, making it difficult to take on the additional tasks in data collection and analysis that performance measurement may entail.
A variety of approaches might help in capacity building. A useful first step might be facilitating access to the funds and expertise required to assess specific capacity needs. One community or state might have a pressing need for data processing resources (e.g., expertise and equipment), while another might need expertise in survey design or data analysis. Because of these differing needs, varying means of meeting them will be necessary.
CDC's Public Health Training Network might be one mechanism, or model, through which training materials could be developed and delivered to state and local audiences. Distance-based learning programs, which reduce the need to travel by establishing video and audio connections to multiple sites, should be considered as a way of reaching this broad audience. Additional funding specifically for travel to major conferences and training programs could also be an effective investment. Direct interaction with colleagues provides learning opportunities that are not available with other approaches, such as use of self-guided instruction or distance-based learning. Access to scholarships and dissertation grants could assist staff in obtaining more advanced academic training.
Other opportunities for state and local training and capacity building should also be explored. For example, university medical centers and schools of public health could develop collaborative programs that would provide state and local health departments with access to training and academic expertise in data collection and analysis while providing faculty and students with opportunities for field experience and research. Foundations, health plans, businesses, and others with interest or experience in performance measurement might become partners with state and local health agencies that are working to improve their skills.
16. DHHS should develop and maintain information technology expertise to assist states and communities as they use new technologies to improve the quality of and capacity for data collection, analysis, and dissemination.
Effective use of information technology requires considerable expertise in matters such as hardware, software, systems design and integration, and applications development. To the extent that DHHS can serve as a central and authoritative source for information and guidance on these technology matters, state and local health agencies will be able to make more rapid use of a broader range of expertise than they would be likely to assemble on their own. In the current highly competitive market for information technology personnel, many state and local agencies are at a disadvantage because their salaries tend to be relatively low.
One example of work DHHS is already doing along these lines is the preparation of an investment analysis guide for states that want to combine categorical grant funds for the development of integrated health information systems (Centers for Disease Control and Prevention and Health Resources and Services Administration, 1998). Another specific area for support is exploration of the technical issues involved in merging data from diverse systems. Among the activities DHHS might support are pilot projects to test the use of specific equipment or processes. DHHS might also promote the development of a national clearinghouse or similar repository for information system applications and modular programming components (i.e., software objects). Access to such resources can make the systems development process simpler, faster, and less costly.
A National Research Agenda
The panel emphasizes that a multidisciplinary research program must be an integral part of any ongoing performance measurement activity for health-related programs. Because experience with performance measurement is still limited, studies are needed to improve understanding of what measures and methods of data collection are appropriate. Also crucial is further research to establish evidence regarding causal links between program interventions and desired outcomes. This evidence, which is currently limited in many fields, is essential for selecting demonstrably meaningful capacity, process, and risk status measures. For example, recent studies have shown that the identification and treatment of co-occurring mental health problems are an important predictor of positive outcomes in substance abuse treatment (Harrison and Asche, in press). Specifically, the panel makes the following recommendation.
17. Federal agencies, foundations, and other private-sector groups should develop and fund a research agenda to support performance measurement activities, including the testing of intervention effectiveness, the investigation of the links between program capacity and processes and program outcomes, the development of measures, the
refinement of data collection and information system technologies, and the use of performance measurement systems and performance-based decision making.
Most users of performance measures will need to rely on the research of others to identify effective health interventions and measures of outcomes for these interventions, as well as to determine the effective use of performance measurement as a management and oversight tool. Therefore, the agendas of federal agencies and private organizations that fund health-related programs should include support for research intended to inform performance measurement in these areas. The range of studies must be broad enough to ensure that they are informative for a variety of settings, including the local, state, and national levels. Initiatives in the Agency for Health Care Policy and Research (AHCPR) under the designations of Strengthen Quality Measurement and Improvement and Support Improvements in Health Outcomes are models for the types of federal research agendas that could serve well for performance measurement research in other health areas.
However, carrying out the research is not, in itself, sufficient for informing and improving the performance monitoring process. Significant findings must be communicated to the participating partners before they can be applied. Therefore, the panel offers this further recommendation.
18. DHHS, foundations and other private organizations, and other partners involved in performance measurement activities should contribute in an appropriate manner to a process of information gathering and dissemination to support the use of evidence-based performance measures.
Resources are needed to support comprehensive reviews and rigorous analyses of the relevant scientific evidence and to produce evidence-based reports describing the scientific foundation applicable to the development of performance measures for use in publicly funded health programs. AHCPR currently sponsors a program to develop, use, and evaluate evidence-based tools and information related to clinical health care. A similar program that would apply to the areas of public health, mental health, and substance abuse would be immensely valuable for developing and refining performance measures and improving the performance monitoring system. Although states and communities generally have limited funds for conducting research or disseminating research results, they can make an important nonmonetary contribution to these efforts by facilitating studies undertaken by others and sharing information about their experiences in the use of performance measurement.
The development and implementation of a broad national health information network that can support performance measurement will require a concerted effort. The leadership of DHHS and its agencies is critical, but these federal agencies must be part of a collaborative undertaking that includes the participation of stakeholders at the state and local levels, in the public and private sectors, and across a variety of program areas. The participants must be partners in seeking benefits from and sharing responsibility for the effective management and operation of a broader health information network.
The information network envisioned by the panel should enhance the capacity of federal, state, and local health programs to meet performance measurement obligations and to use performance data and other information to achieve desired health outcomes. Several significant challenges lie ahead. Appropriate performance measures must be developed, and the data needed to use those measures must be available. Greater consensus must be achieved regarding standards for measures and data that will promote comparability in performance measurement. Necessary technical and analytic skills must be developed and applied to the creation and use of performance data. Sufficient financial and nonfinancial resources must be obtained to support both near-term efforts to introduce performance measurement activities and the longer-term data collection and analysis and research needed to sustain those efforts. This may appear to be a difficult set of challenges, but the panel believes that the need to address them is great and that the current commitment to performance measurement creates an opportunity to make significant progress toward meeting this need in the near future.
Broome, C.V., and C.E. Fox 1998. CDC/HRSA Grant Funding Flexibility for Integrated Health Information Systems. Grant funding transmittal letter. April 1, 1998. U.S. Department of Health and Human Services. http://www.hrsa.dhhs.gov/policy.htm (also at http://www.cdc.gov/funds/policy.htm) (April 21, 1998).
Centers for Disease Control and Prevention 1997. Case definitions for infectious conditions under public health surveillance. MMWR 46(RR-10).
Centers for Disease Control and Prevention and Health Resources and Services Administration 1998. Integrated Health Information Systems Investment Analysis Guide. http://www.hrsa.dhhs.gov/investment.htm#iv (also at http://www.cdc.gov/funds/invest7.htm) (April 21, 1998).
Environmental Protection Agency 1998. One Stop Program Strategy and Grant Award Criteria. http://www.epa.gov/reinvent/onestop/strategy.htm (April 28, 1998).
Gold, M. 1995. Miss or Match: How Well Do State Data Systems Meet State Health Policy Needs? Washington, D.C.: Mathematica Policy Research.
Halverson, P., R.M. Nicola, and E.L. Baker 1998. Performance measurement and accreditation of public health organizations: A call to action. Journal of Public Health Management and Practice 4(4):5–7.
Harrison, P.A., and S. Asche in press. Comparison of substance abuse treatment outcomes for inpatients and outpatients. Journal of Substance Abuse Treatment.
Institute of Medicine 1997a. The Computer-Based Patient Record: An Essential Technology for Health Care, Revised ed. R.S. Dick, E.B. Steen, and D.E. Detmer, eds. Committee on Improving the Patient Record. Washington, D.C.: National Academy Press.
1997b. Improving Health in the Community: A Role for Performance Monitoring. J.S. Durch, L.A. Bailey, and M.A. Stoto, eds. Committee on Using Performance Monitoring to Improve Community Health. Washington, D.C.: National Academy Press.
Joint Commission on Accreditation of Healthcare Organizations 1998. Nation's Three Leading Health Care Quality Oversight Bodies to Coordinate Measurement Activities. Press release. May 19, 1998. http://www.jcaho.org/news/nb.htm (June 5, 1998).
Lasker, R.D., B.L. Humphreys, and W.R. Braithwaite 1995. Making a Powerful Connection: The Health of the Public and the National Information Infrastructure. Report of the U.S. Public Health Service Public Health Data Policy Coordinating Committee. Washington, D.C. http://www.nlm.nih.gov/pubs/staffpubs/lo/makingpd.html (August 11, 1998).
Maternal and Child Health Bureau 1998. State Systems Development Initiative (SSDI) Grant Application Guidance for FY98. U.S. Department of Health and Human Services, Health Resources and Services Administration. http://www.hhs.gov:80/hrsa/mchb/guidance.htm (June 4, 1998).
Mendelson, D.N., and E.M. Salinsky 1997. Health information systems and the role of state government. Health Affairs 16(3):106–119.
National Center for Health Statistics 1998. National Vital Statistics System. U.S. Department of Health and Human Services, Centers for Disease Control and Prevention. http://www.cdc.gov/nchswww/about/major/nvss/nvss.htm (July 6, 1998).
National Committee for Quality Assurance 1997. HEDIS 3.0/1998. Vol. 4, A Roadmap for Information Systems: Evolving Systems to Support Performance Measurement. Washington, D.C.: National Committee for Quality Assurance.
National Research Council 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. E.B. Perrin and J.J. Koshel, eds. Panel on Performance Measures and Data for Public Health Performance Partnership Grants, Committee on National Statistics. Washington, D.C.: National Academy Press.
Perrin, E.B., L.G. Hart, S.M. Skillman, B. Paul, M.A. Hanken, and J. Hummel 1996. Health Information Systems and Their Role in Rural Health Services: Issues and Recommendations. Report to the Agency for Health Care Policy and Research (Contract #282-93-0036-4). Seattle: University of Washington.
Public Health Foundation 1998. Measuring Health Objectives and Indicators: 1997 State and Local Capacity Survey. Washington, D.C.: Public Health Foundation.
Roos, N.P., C.D. Black, N. Frohlich, C. Decoster, M.M. Cohen, D.J. Tataryn, C.A. Mustard, F. Toll, K.C. Carriere, C.A. Burchill, et al. 1995. A population-based health information system. Medical Care 33(12 suppl.):DS13–DS20.
Starr, P. 1997. Smart technology, stunted policy: Developing health information networks. Health Affairs 16(3):91–105.
Thacker, S.B., and D.F. Stroup 1994. Future directions for comprehensive public health surveillance and health information systems in the United States. American Journal of Epidemiology 140:383–397.
U.S. Department of Health and Human Services 1995. HHS Plan for Consolidation of Surveys. April 11, 1995. Washington, D.C. http://aspe.os.dhhs.gov/datacncl/srvyrptl.htm (January 12, 1998).
1997. 1997 Strategic Plan. September 30, 1997. Washington, D.C.
1998. Registry of State-Level Efforts to Integrate Health Information. http://aspe.os.dhhs.gov/statereg/index.htm (February 9, 1998).