4
Data and Information Systems: Issues for Performance Measurement

A performance measurement program must begin by identifying outcome goals, and then using those goals to guide the selection of suitable measures of desired outcomes and related processes and capacities. Once those steps have been completed, operationalizing performance measurement requires access to appropriate data and analytic resources. In its first report, the panel observed that many types of data useful for monitoring the performance of publicly funded health programs are collected and assembled across the country, but that few data sources are ideal for this purpose. For the most part, data systems have not been created specifically for performance measurement, so they may currently be narrower, less timely, or less comparable to other data systems than is optimal.

Despite such shortcomings, there are a number of reasons why the panel favors enhancing this extensive and often strong information base rather than establishing wholly new and specialized data systems for performance measurement. Although current health data collection processes and the resulting data sets often are not well coordinated with each other (Thacker and Stroup, 1994), the panel is hopeful that the current interest in performance measurement, reflected in reports such as this one, will encourage policy makers and health professionals at the federal, state, and local levels to transform the many different existing data sources into a more efficient and effective health information system with the capability of responding to varied information needs.

Collecting and assembling data is expensive, and expanding data collection efforts carries the risk of reducing the resources available for program services. Building on existing data systems for purposes of performance measurement would still require a substantial commitment of resources, but should be expected



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 83
--> 4 Data and Information Systems: Issues for Performance Measurement A performance measurement program must begin by identifying outcome goals, and then using those goals to guide the selection of suitable measures of desired outcomes and related processes and capacities. Once those steps have been completed, operationalizing performance measurement requires access to appropriate data and analytic resources. In its first report, the panel observed that many types of data useful for monitoring the performance of publicly funded health programs are collected and assembled across the country, but that few data sources are ideal for this purpose. For the most part, data systems have not been created specifically for performance measurement, so they may currently be narrower, less timely, or less comparable to other data systems than is optimal. Despite such shortcomings, there are a number of reasons why the panel favors enhancing this extensive and often strong information base rather than establishing wholly new and specialized data systems for performance measurement. Although current health data collection processes and the resulting data sets often are not well coordinated with each other (Thacker and Stroup, 1994), the panel is hopeful that the current interest in performance measurement, reflected in reports such as this one, will encourage policy makers and health professionals at the federal, state, and local levels to transform the many different existing data sources into a more efficient and effective health information system with the capability of responding to varied information needs. Collecting and assembling data is expensive, and expanding data collection efforts carries the risk of reducing the resources available for program services. Building on existing data systems for purposes of performance measurement would still require a substantial commitment of resources, but should be expected

OCR for page 83
--> to promote more efficient and effective use of those systems, and to improve their value for other applications as well. In relying on data collected for other primary purposes, however, those who develop and use performance measures must have a good understanding of the nature and limitations of those data. This chapter begins by reviewing various health data resources. It then examines analytic and operational challenges involved in using those data, including assuring the quality of data and data analysis; developing and implementing standards for data and data systems; enhancing performance measurement through advances in information technology; and protecting the privacy, confidentiality, and security of health data. The chapter then outlines steps that can be taken to strengthen the data and data systems used to support performance measurement, in particular by investing in health data and data systems and by taking a collaborative approach to their development. Health Data Resources Diverse health-related data are required to monitor and better understand the health of the population, including the incidence and prevalence of disease, morbidity and mortality associated with acute and chronic illness, behavioral risk factors, disability, and the quality of life. Data are also needed to plan, implement, and evaluate health policies, programs, and services. The data to meet these needs are produced and used in both the public and private sectors and, increasingly, by public-private partnerships. The Performance Partnership Grants (PPG) proposal that was the impetus for the work of this panel focused attention specifically on data for performance measures to be used in the context of state reporting requirements for federal grants. The panel emphasizes, however, that if performance measurement activities are to succeed, they should fit into a broader agenda for collecting and using health data to protect the health of the public, as well as for guiding the development and implementation of health policies at the local, state, and federal levels. Although the panel did not attempt to address measurement of the quality and performance of individual health care providers or health plans, it should be noted that these activities are generating similar concerns about such matters as the selection of suitable performance measures, the limitations of administrative data sets for assessing health outcomes, the need for greater standardization of measures and data and for methods to improve data quality, and broader use of new information technologies (see, e.g., Iezzoni, 1997a; National Committee for Quality Assurance, 1997; Palmer, 1997; Foundation for Accountability, 1998; and Joint Commission on Accreditation of Healthcare Organizations, 1998). Major changes in social welfare programs are also prompting a reexamination of the adequacy of data resources for monitoring those programs, especially at the state and local levels (e.g., Joint Center for Poverty Research, 1998; National Research Council, 1998).

OCR for page 83
--> Data for performance measurement can be drawn from a variety of sources, such as reports to disease surveillance or vital statistics systems, environmental monitoring systems, population surveys, and clinical or administrative records from service encounters. Considering only the program areas covered by the original PPG proposal, the panel identified 48 data systems that might provide data for performance measurement (National Research Council, 1997a). Most states and communities can be expected to have a similarly large number of systems from which to draw data for performance measurement. Four basic types of data resources are available: (1) registries, often referred to as census data systems, that attempt to capture information about all events of interest on such matters as health status (e.g., births, deaths, cases of disease) or risk factors (e.g., immunizations, environmental contaminants); (2) surveys that obtain data through the systematic collection of information from a representative sample of a population of interest; (3) patient records that contain clinical information obtained in the course of providing health care; and (4) administrative data, such as billing records, that are collected as part of the operation of a program (although these records may include data on health status or clinical care, that is not their primary purpose). Each type of data has a place in performance measurement, but each also has limitations that must be taken into account. Linking data over time or across data sets can potentially overcome some of those limitations and result in more useful information than is obtainable using a single data set or data for a single point in time. The basic types of health data and some of the issues related to linkage of data sets are reviewed in this section. Registries Registries are census-like data systems designed to compile information on all events of a specified type, such as births, deaths, specific injuries and environmental or infectious diseases, cancers, immunizations, hospital discharges, and birth defects. Vital records and disease surveillance registries are some of the most long-standing examples of these health data systems. Reporting systems also compile information on air and water quality, work-related injuries, and motor vehicle crashes resulting in deaths. Registries rely on reports of specific information to a designated authority. Some registries collect data through direct reporting of the events of interest (e.g., births, cases of reportable diseases), whereas others rely on assembling information originally collected in whole or in part for other purposes (e.g., work-related injuries). Some of these systems operate locally, while others are connected to a state-or nationwide data system. For example, hospitals file reports on births with local or state registrars, and states then transmit these records to the National Center for Health Statistics (NCHS), where national vital statistics data are compiled. The rules governing which data are collected and how they are reported are developed and maintained through a federal-state collaborative system. In

OCR for page 83
--> contrast, immunization registries are being developed by some states and communities to capture reports on all immunizations administered to children (and also to serve as an information resource for health care providers on the immunization status of children under their care), but there is no national registry of immunization reports. Registry systems benefit from standardized reporting practices. For example, NCHS and the states work together to develop standard birth and death certificates and guidelines for completing them. Systems differ substantially in their completeness, however. For example, virtually all births are reported, but reporting of fetal deaths is much less complete. Data on some reportable but often clinically mild or asymptomatic diseases (e.g., chlamydia, hepatitis C) are often incomplete because cases may not receive medical care or may not be diagnosed. The quality of the reported data also varies. Birth certificate data on birth weight, for example, are generally more reliable than some of the accompanying information, such as reports of birth defects or the mother's use of tobacco during pregnancy. The significance of such limitations in these data depends on how the data are to be used. Estimation of reliable incidence and prevalence rates, for example, requires nearly complete reporting, whereas monitoring of trends depends more (within limits) on consistency of reporting than on completeness. For example, consistent and essentially complete reporting of births and deaths is the basis for calculation of comparable birth and death rates at the local, state, and national levels. In contrast, reportable disease data compiled at the national level are useful for monitoring disease trends even if they are not complete; however, these data are appropriate for more precise assessments of incidence rates only for those conditions for which reporting is essentially complete. And any variation in reporting practices from state to state means the resulting data will not be appropriate for assessing small differences in incidence rates across states. Surveys Surveys are an essential resource for population-based performance measurement data. Well-designed surveys produce information about an entire population by collecting data from a representative sample of that population. The population of interest in a survey is often defined by residence in a geographic area, such as a state or county, but may also be defined by other characteristics, such as age, place of employment, enrollment in a public assistance program (e.g., Medicaid), or use of a specific clinic. Continuing survey programs that have a defined schedule (e.g., the National Immunization Survey, the Behavioral Risk Factor Survey) can combine a stable core of questions, yielding results that can be compared over time, with changing sets of questions that can address topics of special interest. One-time surveys or surveys repeated on an irregular schedule have less value for performance measurement because they provide at

OCR for page 83
--> best a limited basis for comparisons over time. The use of surveys requires special expertise in such matters as questionnaire and sample design. Surveys are particularly well suited to obtaining data for many measures of health status, functioning, and risk that depend on reports of behaviors, perceptions, and attitudes. They also are good tools for collecting information on general activities and events. Survey data are, however, vulnerable to misreporting and can be adversely affected by nonresponse. Respondents may misreport unintentionally because of recall errors or lack of knowledge (e.g., date of last illness or hypertension status), may refuse to answer certain questions, or may intentionally alter their responses on sensitive topics (e.g., drug use or even exercise habits). Careful questionnaire design can help reduce some forms of misreporting. Nonresponse is a concern because individuals who are missed may differ from the respondents in important ways (e.g., older or younger, lower or higher income, sicker or healthier) that cannot be determined with certainty. Despite such limitations, surveys may be the best or only option for obtaining data on key topics of interest. The cost of surveys is a major constraint on their use. In contrast to data collection that occurs as a byproduct of other activities, such as restaurant inspections or health care visits, surveys require a set of specialized activities, including developing a sampling frame, selecting the sample, locating the eligible respondents, and gathering the survey data. For each of these activities, choices can be made that affect costs, but those choices may also affect the quality of the survey results. For example, telephone interviews tend to be less costly than in-person interviews, but cannot reach people who do not have a telephone. Such cost trade-offs should be weighed carefully. For some purposes, a factor such as telephone access may have little impact on the quality of the data, readily justifying the use of a less costly method of data collection. An analysis of National Health Interview Survey data, which were obtained through in-person interviews, found little difference in results between respondents who had telephone access and the overall responses, even when the analysis was restricted to persons below the poverty level (Anderson et al., 1998). Similarly, studies of the Behavioral Risk Factor Surveillance System suggest that its telephone-based methods are sufficiently reliable to justify continued use of this less expensive method (e.g., Arday et al., 1997). In contrast, a study focusing on health insurance coverage suggests that reliance on telephone interviews alone may not be adequate for some analyses (Strouse et al., 1997). It may, however, be possible to use baseline data from in-person interviews to adjust estimates based on data collected by telephone in subsequent rounds of a study. Patient Records and Related Clinical Encounter Data The detailed clinical records maintained by physicians, hospitals, health plans, and most other health care providers on each patient they treat are reposito-

OCR for page 83
--> ries for an array of data such as patient-reported health status and risk factors, clinical observations, diagnoses, procedures performed, medications prescribed, and results of laboratory tests. Access to clinical data from medical records would improve the analytic strength of many health survey and administrative data sets. However, these records have important limitations. Most patient records are still maintained in paper form, which makes it difficult to aggregate and analyze the data or integrate them into broader health data systems. Extracting data from paper records requires time-consuming and costly review of individual files. Research studies that require specific clinical data often review samples of records, but even that approach is likely to be too costly and time-consuming to be practical for the periodic reporting required for performance measurement. Furthermore, the completeness and consistency of records may differ across records or within a single record over time, and may vary more for certain types of information than for others. For example, numerical data, such as blood pressure readings, are more readily recorded in a consistent manner than are notes describing clinical observations. There is widespread support for the development of computer-based patient records (CPRs), and considerable progress has been made in this area in recent years (Institute of Medicine, 1997). The CPR holds the promise that documentation of the process and outcomes of care will become a byproduct of the use of such an information system in the delivery of care, and that patient records will become a more practical source of data for performance measurement for both the health care industry and health agencies at the federal, state, and local levels. Major advances are needed in at least three areas, however, if more extensive use is to be made of clinical data in computerized form: standards defining the structure and content of electronic clinical records must be established, technology for converting natural medical language into standardized coding systems must be developed, and privacy concerns must be resolved. Despite progress, there are still substantial differences and incompatibilities among the CPR systems now in use. Standards for the data elements included in patient records, the codes and vocabulary used to represent clinical data, and the format of electronic records are still evolving. Additional research and testing are also needed to move beyond prototype systems for converting natural medical language into medical procedure and diagnosis codes. Among the groups working on these CPR issues are federal agencies such as the National Library of Medicine and the Agency for Health Care Policy and Research, private organizations such as the Computerized Patient Record Institute, and various private companies. Progress toward a CPR should also result from the Health Insurance Portability and Accountability Act of 1996 (HIPAA) (Public Law 104-191), which directs the Secretary of Health and Human Services to promulgate guidelines for computerized medical records by August 2000. HIPAA also calls for establishing policies to protect the security and privacy of electronic health data transactions. Privacy concerns are an issue for all health-related data, but are

OCR for page 83
--> particularly acute for information contained in medical records. (Other provisions of HIPAA are reviewed elsewhere in this chapter.) Computerization per se will not, however, overcome certain limitations inherent in patient records. For example, as used in most health care settings, patient records are not well suited to capturing information on patients' views about the care they receive. Clinical records can also be incomplete when people receive health care services from several sources, each of which maintains a separate record. For some of its performance measures, the Health Plan Employer Data and Information Set (HEDIS), version 3.0, compensates for such factors by requiring health plans to use data from a member survey rather than from administrative or medical records (National Committee for Quality Assurance, 1996; see also Chapter 2). For example, the rate of influenza vaccination among older adults is to be tracked with survey data because health plan members may receive these shots through community programs instead of their health plan. Administrative Data The operation of health programs typically generates substantial amounts of nonclinical administrative data that can be useful for performance measurement. Some of this information describes program resources or characteristics of program operation, such as numbers and qualifications of staff members or features of facilities used to provide services (e.g., number of laboratories meeting quality standards). Administrative records on population-based services can produce such information as the number and results of restaurant inspections, the number of immunizations administered at special immunization events, or the number of health education programs offered. Programs that provide services to specific individuals (e.g., substance abuse treatment or prenatal care) generate administrative records that contain information about those individuals and the services they receive. Administrative data produced by various other activities that are not specifically health-related can also provide useful information for health programs. For example, traffic safety records can provide data on motor vehicle crashes resulting in injuries, and state corrections records can provide information on incarcerated adults with serious mental illness. In addition, administrative records can sometimes be used to identify a population for a separate survey-based data collection activity. Most administrative data sets are created to serve operational purposes rather than the needs of performance measurement or other analytic tasks. Even so, they are a valuable resource with some advantages over other types of data. A recent assessment of the utility of administrative data for policy studies of public assistance programs provides useful insights for the health-related programs of interest to this panel (Joint Center for Poverty Research, 1998). Administrative data sets can offer detailed and generally accurate program information, large enough numbers of records to permit analyses of subgroups of participants, greater

OCR for page 83
--> state and local specificity and applicability than many national data sources, longitudinal information on programs and program participants, and low marginal costs for data collection. At the same time, these data sets have important limitations for secondary uses such as performance monitoring. They generally cover a selected set of people and activities and are not necessarily representative of the population as a whole. In the case of health services, for example, such data sets have no information on individuals in a community who might be in need of those services but have not sought care. The data sets also may lack useful descriptive information on the economic and demographic characteristics of the individuals who are included. Measures such as program participation rates require that administrative data (the numerator) be supplemented by population data (the denominator) from another source. Information on outcomes and events that occur outside the framework of the program are rarely available. For example, the records of a substance abuse treatment program can produce data such as the number of participants who complete treatment, but will not directly capture the drug-related arrests of program drop-outs or the subsequent employment history of people who have successfully completed treatment. Similarly, records of a water treatment facility can provide data on observed levels of bacterial contaminants, but will not reflect outbreaks of waterborne disease. Linkages to other data sets (discussed below) can overcome some of these limitations, but the linkage process poses its own technical and policy challenges. These issues are discussed elsewhere in this chapter. Use of program-specific data definitions can hinder or prevent valid comparisons across data sets. For example, one health program may define adolescents as young people between the ages of 12 and 18, while another may use ages 13 to 17. Greater coordination and collaboration and the development of standard measures may overcome some definitional differences. Yet other differences in data definitions reflect true variations in program features; if comparisons are necessary, those variations must be taken into account. Operational and design factors may also affect the usefulness of administrative data sets for purposes such as performance monitoring. Programs that serve families may not identify each family member separately, making it difficult to distinguish who received what services. If closed or inactive cases are dropped, the data set cannot provide a complete record of services or participants. And the installation of new or upgraded information systems (either equipment or programs) may result in lost or limited access to records created with the previous system. Claims Data A specialized administrative data resource that bridges the public and private sectors in health care is insurance and other third-party claims for payment for health services. An enormous quantity of data is produced from the billing and

OCR for page 83
--> payment of health insurance claims. In accordance with the administrative simplification provisions of HIPAA, standards for the format and content of electronic claims transactions are being established. Claims data have been used to study the effectiveness and outcomes of health care and may also have a place in performance measurement. As with the other data sources discussed in this section, however, their limitations must be kept in mind. Claims data generally include only a minimal amount of clinical information (e.g., diagnosis, procedure performed) to document the fact that a covered service was provided and payment is owed. Moreover, medical conditions and treatments can be characterized in varying ways in insurance claims. This factor can reduce the consistency and comparability of claims records. Incentives such as higher reimbursement rates for certain types of care can encourage more deliberate changes over time in the content of claims data. The timeliness of these data can also be a concern. Greater use of electronic data interchange (EDI) allows faster claims processing, but delays of several months may still occur in filing and settling claims. Another limitation of claims data is that they may not provide a complete record of services received by an individual or by a population in a given community or state because claims are submitted only for covered services and only for the individuals served by a specific insurer. Typically, a defined geographic area (a state or a community) is served by several insurers, each of which may offer many different insurance products that vary in scope and terms of coverage. In addition, Medicaid claims records may be managed separately by state agencies, and prepaid managed care plans generally do not generate claims records. With nearly universal participation in Medicare among those aged 65 and older, Medicare claims files have been more complete than other claims databases and therefore often more useful for state and local analyses. However, claims records are generally not available for Medicare services provided through prepaid managed care plans. The experience of the State of Maryland in using Medicaid claims data in conjunction with public health initiatives illustrates both the strengths and limitations of such data when used for a purpose other than that for which they were originally collected (see Box 4-1). Although these data are a promising means of monitoring health care services for a vulnerable population, they do not capture all of the information that may be needed for some purposes. Linkage of Data Sets Data linkage involves matching records on specific individuals to other records for those individuals in the same or other data sets. This panel believes that in many cases, better performance measurement data could be obtained if selected data sets could be linked. As noted earlier, such linkages can overcome some of the limitations of specific data sets. This is especially true in efforts to

OCR for page 83
--> Box 4-1 Development of a Health Care Services Database in Maryland In 1985, Maryland began developing person-based analytic files from Medicaid data. These files were used to conduct analyses that provided a basis for statewide public health initiatives such as diabetes management (Stuart, 1994) and the Maryland Access to Care program, which increased access to primary care for women and children participating in Medicaid (Stuart et al., 1990). The Medicaid data files were also used in research studies, alone or supplemented by data from medical records, to assess the quality or effectiveness of care (Starfield et al., 1994; Powe et al., 1996; Svikis et al., 1998). In another study, the claims records were used as a sampling frame for a survey that provided detailed information on a vulnerable population (Rubin et al., 1994). The state's use of claims data was expanded in 1993 with the establishment of the Maryland Health Care Access and Cost Commission. The commission's tasks included the development of a medical care database to provide statewide information on health services rendered in all nonhospital settings, including health maintenance organizations and pharmacies. This mandate has been carried out using the computerized claims payment history files from third-party insurers in Maryland, including Medicaid, Medicare, and selected private payers (Stuart, 1995). The commission's work has also highlighted the limitations of this type of database for public health applications such as tracking immunizations. In particular, it was anticipated that suitable data could be obtained from claims filed using standard billing forms (HCFA-1500 and UB-82/92). Private physicians, noting the requirement to complete multiple forms every time they immunized a child, pleaded for a unified system that required only one submission of data. Public health administrators pointed out that the HCFA-1500 form does not capture information on vaccine lot numbers, and standard procedure codes have not been established for administering some vaccines. In addition, since instructions for using the forms and codes for this purpose have not been standardized, recorded information could be subject to inaccurate interpretation (Stuart, 1995). While the Maryland claims database cannot yet replace the public health immunization registry, the potential for using computerized claims files to reduce administrative costs and improve timely access to data that are useful for public health has been recognized.

OCR for page 83
--> relate health outcomes to services provided. For example, linking data from a community survey to administrative records from a prenatal care program could help identify eligible mothers who did not participate in the program and therefore do not appear in the administrative data system. Alternatively, program records could compensate for survey respondents' recall errors about numbers of visits or timing of specific services. Another approach to linking data sets is taken with some immunization registries: birth records are used to create an initial entry in the registry to which subsequent immunization reports are linked. In health care studies, efforts have been made to link multiple insurance claims for an individual to construct a more coherent picture of care for an episode of illness. A particularly broad pilot project on data linkage that is relevant to the panel's earlier work on performance measures for emergency medical services was initiated by the National Highway Traffic Safety Administration (1996) of the U.S. Department of Transportation. The Crash Outcome Data Evaluation System (CODES), originally tested in seven states, is designed to link data on motor vehicle crashes, emergency medical services, emergency department care, hospital and outpatient care, rehabilitation and long-term care, death certificates, and insurance claims. Using these linked data, states have been able to explore such factors as populations at increased risk for injury (e.g., on the basis of age, alcohol use, or failure to use seatbelts), the consequences of specific types of crashes or injuries (e.g, collisions with pedestrians, abdominal versus head injuries), and the effects of delayed prehospital care. Attempting to match records from separate data systems poses significant technical challenges. Reasonably successful techniques have been developed that rely on combinations of information, such as name, address, and date of birth, to establish highly probable matches. Use of unique personal identifiers might simplify the process of establishing exact matches, but such identifiers have not been uniformly employed. Provisions of HIPAA now call for adoption of these identifiers, especially for use in electronic health care data transactions, but there is serious concern that stronger privacy protections must be enacted before unique personal identifiers can be used with confidence or comfort (see National Committee on Vital and Health Statistics, 1997b). Even without the use of personal identifiers, the linkage of data sets must be undertaken only with firm assurance that personal privacy and the confidentiality of the data will be protected. (See the discussion of these issues later in this chapter.) Steps Toward Integration of Data Sets In the public sector, many states are working to enhance the integration and accessibility of health data (Mendelson and Salinsky, 1997; U.S. Department of Health and Human Services, 1998b). For example, Georgia has provided a single Internet access point to county- and state-level information from several data

OCR for page 83
--> of computer systems, has placed a high premium on information technology skills. Given these constraints, access to training programs that can enhance the skills of existing staff and to technical assistance that draws on the expertise of others becomes especially important. For staff to obtain the needed training, suitable materials and programs are required, as well as time and funds to support the staff members' participation. Training opportunities may take many forms, including formal academic programs (e.g., graduate programs in schools of public health) and specialized courses and training sessions offered by federal agencies (e.g., the CDC Public Health Training Network), academic institutions, or others in the private sector. Funding for scholarships and dissertation grants could assist staff in obtaining advanced academic training. Support for other training opportunities is also needed. Teleconferencing, self-guided instruction, and other forms of distance-based learning can bring a variety of training to large audiences and can compensate in part for constraints on funding for travel to attend courses and conferences. However, supplementing distance-based training with attendance at off-site programs may give staff valuable opportunities to learn through direct interaction with colleagues from other states or communities. The panel was informed that even though CDC provides funds specifically to allow staff from each state to attend the annual BRFSS conference, these funds generally cover participation by the data managers who oversee the collection and maintenance of state BRFSS data sets, but are not adequate to support the attendance of most users of BRFSS data.3 Technical assistance can make a large reservoir of expertise available to meet diverse needs. The assistance can take many forms, including publications, information clearinghouses, conferences, and consultations with experts. In a recent activity of particular relevance to the interests of this panel, CDC and HRSA worked with the Association of State and Territorial Health Officials and the National Association of County and City Health Officials to develop an "investment guide" to assist states in planning and developing integrated health information systems (Centers for Disease Control and Prevention and Health Resources and Services Administration, 1998). A review of technical assistance activities in DHHS led to the conclusion that these activities could be enhanced by greater coordination and evaluation of the effectiveness of current forms of assistance (U.S. Department of Health and Human Services, 1997f). It was suggested to the panel that in the area of epidemiologic analysis, for example, states could benefit from greater access to more senior CDC epidemiologists to supplement programs that currently rely primarily 3   This information was reported to the panel in the background paper "Improving Federal-State Data Collection to Monitor Program Performance Measures," which was prepared by the Science and Epidemiology Committee of the Association of State and Territorial Chronic Disease Program Directors and the Council of State and Territorial Epidemiologists.

OCR for page 83
--> on newly trained epidemiologists.4 Because of their national perspective and their influential role as funders of many health programs, federal agencies are well placed to serve as a focal point for technical assistance. User groups that draw participants from local, state, and federal health agencies could open other channels for obtaining technical assistance and learning about a broader range of health data issues. States and communities might also look to academic institutions and others in the private sector, particularly in a rapidly evolving area such as information technology. Foundations or other nonprofit groups might be able to serve as intermediaries in sponsoring such public-private collaborations. Taking A Collaborative Approach to the Development of Health Data and Information Systems The panel's deliberations regarding performance measurement have led to the conclusion that much greater collaboration and coordination are an essential foundation for further development of the nation's health data and data systems. It appears that by adopting a broadly based approach to health data needs and resources, it will be possible to make more effective use of available data and information systems for performance measurement, as well as for other purposes, including monitoring health status in the population, managing health programs, and informing policy makers and the public. For publicly funded health programs, it is essential that information needs at the federal, state, and local levels all be taken into account. The DHHS strategic plan recognizes the need for accurate and timely data at all these levels for assessing changes in health status and managing health programs (U.S. Department of Health and Human Services, 1997a). States are responding to these concerns with initiatives aimed at strengthening their health data infrastructure by improving data quality; developing standards for data definitions, information system configurations, and electronic transmission of data; and linking data systems (see the earlier discussion in this chapter).5 The panel is encouraged to see states taking these steps and believes there is additional value in promoting a national approach to these matters. State-specific solutions may limit the comparability of data across states, and states may miss opportunities to collaborate or to adopt successful strategies developed elsewhere. Likewise, the panel applauds the advances that HIPAA is expected to bring to standards for electronic health care transactions, but also urges support for efforts that will encourage the development of standards for an even broader range of health data elements, such as those likely to be used in performance measures for a variety of publicly funded health programs. 4   This suggestion was also made to the panel in the background paper "Improving Federal-State Data Collection to Monitor Program Performance Measures." 5   A summary of state efforts to integrate health information was compiled by DHHS and The Lewin Group. Information on activities in each state can be found at <http://aspe.os.dhhs.gov/statereg/>.

OCR for page 83
--> Meeting the Needs of Many Data Users Many federal health data systems have been designed to provide national-level data for an overall assessment of health status to help guide the planning and implementation of national health policies and programs. At the state and local levels—the "front line" for service delivery—the perspective is somewhat different. Detailed local data are needed to guide planning and program operations, and they have more immediate value than national estimates. Even summary state-level data may lack sufficient detail to be useful for understanding health needs and program outcomes at the local level. For example, data for Illinois as a whole are not likely to provide a satisfactory picture of health status and program activities in either Chicago or a rural county in southern Illinois. Developing a more efficient and effective approach to information systems used to support performance measurement may depend on finding a way to accommodate differing perspectives on several issues. One concern is the tension between the program-specific perspective that is often the basis for funding and oversight of publicly funded health programs and a more functional perspective on the operation of data systems that focuses on the commonalities among the data collection and management tasks to be performed for many program areas. Categorical grant programs help ensure that funds are directed to specific needs, but they may hinder both a broad view of health and the efficient organization of data systems at the state and local levels. At the federal level, the programmatic perspective often dominates. The various categorical funding programs often have specialized reporting requirements, and some require the use of independent, customized systems to file those reports (e.g., for HIV/AIDS cases as noted earlier). In contrast, an approach that consolidates data collection systems across program areas can be beneficial at the state and local levels, where limited staff and operational funding can be used more efficiently if similar tasks can be combined. For example, a single ongoing survey such as a state's Behavioral Risk Factor Survey can collect data on such topics as smoking habits, alcohol use, disabling conditions, and mammography use without requiring each program to operate a separate survey. The specialized data systems developed to meet categorical program requirements tend to have a limited scope and may be costly to maintain. They may require duplication of data collection and management tasks, and if their reporting requirements are incompatible, they may preclude use of a single, more efficient data collection method at the state or local level. Cooperation across programs can provide an opportunity to combine resources from diverse program areas to support similar tasks in data collection and analysis. For example, a single system of notifiable disease surveillance could accommodate reports on AIDS cases and pesticide exposures, or a single ongoing telephone survey of adults could integrate questions about domestic violence and mammography use. For this approach to work, compromises may be needed to balance the interests of diverse program areas. If a single survey that addresses both domestic violence

OCR for page 83
--> and mammography use is to remain short enough to be practical, it may have to collect less detail on each topic than would be gathered by separate surveys. State and local officials and health planners are also concerned about the flexibility and timeliness of data collection and reporting. They require access to current information about the specific populations they serve for effective program implementation and management. Often, data systems managed at the federal level have not been able to respond to these needs. For example, the National Health Interview Survey, National Health and Nutrition Examination Survey, and National Hospital Discharge Survey produce valuable national data, but are not designed to produce state- or local-level estimates. Moreover, state and local health departments and other health-related agencies have had little organized opportunity to participate in shaping the design and content of many federally operated data systems. Without this input, such systems are less likely to be relevant to state and local concerns, and opportunities to improve comparability or coordination across federal, state, and local data systems may be missed. Also, data managed at the national level have often been produced more slowly than is useful for state and local purposes. New computer and communications technologies are reducing the time needed to collect and process data and produce reports, but they may require expertise and equipment that are not yet available in some states and communities. To the extent that federal data systems will be relied upon to meet the need for state and local data for performance measurement, those systems will require the capacity and flexibility to respond in a timely way to state or local information needs. They will also have to ensure that data processing and reporting proceed as expeditiously as possible. The need for timely data of state or local relevance should not, however, undermine the quality of the data in terms of validity, reliability, completeness, or accuracy. For example, new survey questions or modules must be validated, and survey staff must be trained to administer them. Concerns at the federal level about the quality and comparability of data produced by states have tended to encourage federal centralization of data systems rather than aggregation of state-level data. Although states acknowledge shortcomings in some areas, they are committed to producing high-quality data. Federal-state collaborations in areas such as vital records data and AIDS case reporting have achieved good quality and comparability in state-based data systems. These collaborations stand as examples for efforts that could be undertaken in other areas, such as enhancing the comparability of states' behavioral risk factor data. Another source of tension is the burden associated with the reporting requirements for the federal block grants (e.g., the Preventive Health and Health Services Block Grant or the Substance Abuse Prevention and Treatment Block Grant) that provide a portion of the funds used to support state and local health programs. In the past, the reporting requirements associated with these grants have imposed a significant burden because some of the required information is not readily available and is often expensive or time-consuming to obtain. In

OCR for page 83
--> addition, the reporting requirements of different federal grants are not always consistent across program areas. Constraints on the use of grant funds have also tended to prevent the consolidation of funding to support the development of integrated data systems. The performance partnership concept represents an effort to reduce this burden by making the states partners in a negotiation with the granting agency that leads to the selection of some of the measures to be reported. Plans should be made to assess the impact of this approach on states' reporting burden. Collaboration in the Design and Implementation of Data Systems The panel has concluded that a more collaborative approach to the planning, design, and operation of health data systems would better serve the needs of all parties at the federal, state, and local levels. This conclusion is consistent with the views of the Council of State and Territorial Epidemiologists (1997a,b), as reflected in that organization's recommendations in support of a National Public Health Surveillance System and for enhanced usefulness of state and local data collection by the National Center for Health Statistics. Those recommendations included improving access to surveillance data through better coordination of data systems, and planning surveillance and other data collection activities at the state and local levels in a standardized but collaborative fashion that includes local, state, and federal partners from relevant organizations. The panel's position is also consistent with the follow-up steps proposed as a result of the 1997 review of progress toward the Healthy People 2000 objectives on surveillance and data systems (U.S. Department of Health and Human Services, 1997e). Those proposals included involving state and local governments at every stage of national data collection, analysis, and dissemination; providing easier access to national data sets, including additional geocoding to facilitate subnational analyses; improving coordination of data resources within DHHS and between census and health program data; and giving greater attention to state and local priorities in the development of health objectives for Healthy People 2010. Collaborative efforts are complicated by the multiplicity of stakeholders across the federal, state, and local levels. No single voice at any of these levels can speak to all of the issues that need to be addressed, and no established framework is currently available for selecting representatives and involving them in deliberations about data system issues (e.g., survey design, question selection). At the federal level, DHHS has a critical leadership role to play in these activities, but it must function as a partner with other stakeholders. Mechanisms are needed for designating recognized representatives of key stakeholder groups and for supporting their participation in formal and informal efforts to improve coordination and collaboration. Currently, opportunities for state officials to meet with their federal counterparts may be lost because funding constraints prevent out-of-state

OCR for page 83
--> travel. Similarly, states must work in partnership with community-level stakeholders, as well as with relevant federal and private-sector groups, to ensure that community information needs are addressed. For state and local government, the stakeholders include both staff with policy and programmatic responsibilities who use health data and staff with technical expertise in data collection and analysis who produce and manage health data. Collaboration must be pursued not only in an intergovernmental framework, but also intragovernmentally. Better coordination among federal agencies, both within DHHS and between DHHS and other departments, could contribute to more effective use of available data and data collection systems and help reduce duplication of reporting effort for states and communities. Similarly, greater collaboration among states and communities increases the likelihood that they will be able to learn from each other and develop comparable measures, definitions, and data collection methods for monitoring health programs. Conclusions Health-related data are needed for the formulation of health policies and for the optimal targeting of resources to address priority health issues. Recent interest in performance measurement and performance-based accountability has brought renewed and broader attention to many long-standing concerns about these data and the data systems through which they are produced and used. The panel is convinced that this interest could and should be translated into the sustained commitment of time and resources needed to develop a more comprehensive and coherent approach to health data and health data systems that would build effectively on existing data resources and be capable of meeting health information needs at the federal, state, and local levels. The panel has focused primarily on the public-sector perspective, but recognizes that there are closely related private-sector interests and developments that must not be overlooked. Attention must be given both to operational concerns and to policy issues. On the operational side, one of the most fundamental requirements must be ensuring that good-quality data are available and used in appropriate analyses. To make health data more useful in a broader context, greater consistency and comparability are needed. Key to achieving this objective will be the variety of activities under way to establish standards for the methods used to collect the data; the content and format of data files; the formats for exchanging data electronically; the protection of data privacy, confidentiality, and security; and the measures used to assess performance. Advances in computer technology and electronic data transmission could speed the collection and analysis of data and facilitate access to a broader range of health-related data for many more users. The fundamental need is for a collaborative partnership across the local, state, and federal levels as a basis for strengthening and better coordinating the health data and information systems needed to support performance measurement.

OCR for page 83
--> Mechanisms must allow stakeholders to participate in an ongoing process that encourages them to contribute to policy determinations about what information is to be collected and how it is to be used. Because health issues affect everyone and are addressed in a variety of ways, the panel supports a national approach that recognizes a broad range of interests. DHHS has an important leadership role to play in furthering these efforts, but all of the participants must share responsibility for ensuring that health data and data systems receive the support they need to operate efficiently and effectively. An investment must be made in the data collection programs and information technology that are at the core of these information systems and in the necessary training and technical assistance for the people who produce and use health data. References Anderson, J.E., D.E. Nelson, and R.W. Wilson 1998. Telephone coverage and measurement of health risk indicators: Data from the National Health Interview Survey. American Journal of Public Health 88:1392–1395. Arday, D.R., S.L. Tomar, D.E. Nelson, R.K. Merritt, M.W. Schooley, and P. Mowery 1997. State smoking prevalence estimates: A comparison of the behavioral risk factor surveillance system and current population surveys. American Journal of Public Health 87:1665–1669. Bailar, J.C., and F. Mosteller, eds. 1992. Medical Uses of Statistics, 2nd ed. Boston: NEJM Books. Broome, C.V., and C.E. Fox 1998. CDC/HRSA Grant Funding Flexibility for Integrated Health Information Systems. Grant funding transmittal letter. April 1, 1998. U.S. Department of Health and Human Services. http://www.hrsa.dhhs.gov/policy.htm (also at http://www.cdc.gov/funds/policy.htm) (April 21, 1998). Centers for Disease Control and Prevention 1997. Case definitions for infectious conditions under public health surveillance. MMWR 46(RR-10). Centers for Disease Control and Prevention and Health Resources and Services Administration 1998. Integrated Health Information Systems Investment Analysis Guide. http://www.hrsa. dhhs.gov/investment.htm#iv (also at http://www.cdc.gov/funds/invest7.htm) (April 21, 1998). Clarke, K.C., S.L. McLafferty, and B.J. Tempalski 1996. On epidemiology and geographic information systems: A review and discussion of future directions. Emerging Infectious Diseases 2(2):85–92. Council of State and Territorial Epidemiologists 1997a. Implementation of National Public Health Surveillance System. Position statement #EC-1. Adopted June 19, 1997. http://www.cste.org/page9.html (September 28, 1998). 1997b. NCHS-State Data Coordination. Position statement #CD-2. Adopted June 19, 1997. http://www.cste.org/page58.html (September 28, 1998). Environmental Protection Agency 1998. One Stop Program Strategy and Grant Award Criteria. http://www.epa.gov/reinvent/onestop/strategy.htm (April 21, 1998).

OCR for page 83
--> Foundation for Accountability 1998. About FACCT. http://www.facct.org/about.html (April 15, 1998). Gordis, L. 1996. Epidemiology. Philadelphia: Saunders. Gostin, L., Z. Lazzarini, V.S. Neslund, and M.T. Osterholm 1996. The public health information infrastructure: A national review of the law on health information privacy. JAMA 275:1921–1927. Hoaglin, D.C., R.J. Light, B. McPeek, F. Mosteller, and M.A. Stoto 1982. Data for Decisions: Information Strategies for Policymakers. Cambridge, Mass.: Abt Books. Iezzoni, L.I., ed. 1994. Risk Adjustment for Measuring Health Care Outcomes. Ann Arbor, Mich.: Health Administration Press. Iezzoni, L.I. 1997a. Assessing quality using administrative data. Annals of Internal Medicine 127:666–674. 1997b. The risks of risk adjustment. JAMA 278:1600–1607. Institute of Medicine 1994. Health Data in the Information Age: Use, Disclosure and Privacy. M.S. Donaldson and K.N. Lohr, eds. Committee on Regional Health Data Networks. Washington, D.C.: National Academy Press. 1997. The Computer-Based Patient Record: An Essential Technology for Health Care, Revised ed. R.S. Dick, E.B. Steen, and D.E. Detmer, eds. Committee on Improving the Patient Record. Washington, D.C.: National Academy Press. Joint Center for Poverty Research 1998. Administrative Data for Policy Relevant Research: Assessment of Current Utility and Recommendations for Development. V.J. Hotz, R. Goerge, J.D. Balzekas, and F. Margolin, eds. Report of the Advisory Panel on Research Uses of Administrative Data. Evanston, Ill.: Northwestern University/University of Chicago Joint Center for Poverty Research. Joint Commission on Accreditation of Healthcare Organizations 1998. Performance Measurement. http://www.jcaho.org/perfmeas/perfmsmt.htm (July 24, 1998). Kim, I., and K.G. Keppel 1997. Priority Data Needs: Sources of National, State, and Local-Level Data and Data Collection Systems. Healthy People 2000 Statistical Notes, No. 15. Hyattsville, Md.: National Center for Health Statistics. Landon, B., L.I. Iezzoni, A.S. Ash, M. Shwartz, J. Daley, J.S. Hughes, and Y.D. Mackiernan 1996. Judging hospitals by severity-adjusted mortality rates: The case of CABG surgery. Inquiry 33:155–166. Lasker, R.D., B.L. Humphreys, and W.R. Braithwaite 1995. Making a Powerful Connection: The Health of the Public and the National Information Infrastructure. Report of the U.S. Public Health Service Public Health Data Policy Coordinating Committee. Washington, D.C. http://www.nlm.nih.gov/pubs/staffpubs/lo/makingpd.html (August 11, 1998). Luft, H.S., and P.S. Romano 1993. Chance, continuity, and change in hospital mortality rates. Coronary artery bypass graft patients in California hospitals, 1983 to 1989. JAMA 270:331–337. Maternal and Child Health Bureau 1998. State Systems Development Initiative (SSDI) Grant Application Guidance for FY98. U.S. Department of Health and Human Services, Health Resources and Services Administration. http://www.hhs.gov:80/hrsa/mchb/guidance.htm (June 4, 1998).

OCR for page 83
--> McLellan, A.T., H. Kushner, D. Metzger, R. Peters, I. Smith, G. Grisson, H. Pettinati, and M. Argeriou 1992. The fifth edition of the Addiction Severity Index. Journal of Substance Abuse Treatment 9:199–213. Mendelson, D.N., and E.M. Salinsky 1997. Health information systems and the role of state government. Health Affairs 16(3):106–119. Munro, N. 1996. Infotech reshapes health care marketplace. Washington Technology 11(August 8):1–3. National Association of State Mental Health Program Directors 1998. State Mental Health Directors Adopt a Framework of Performance Indicators for Mental Health Systems. Press announcement. May 1998. Alexandria, Va. National Association of State Mental Health Program Directors Research Institute 1998. Five State Feasibility Study on State Mental Health Agency Performance Measures. Final Report. Alexandria, Va.: National Association of State Mental Health Program Directors Research Institute. National Committee for Quality Assurance 1996. NCQA Issues Final Technical Specifications for HEDIS 3.0. Press release. October 31, 1996. http://www.ncqa.org/news/techre12.htm (April 7, 1998). 1997. HEDIS 3.0/1998. Vol. 4, A Roadmap for Information Systems: Evolving Systems to Support Performance Measurement. Washington, D.C.: National Committee for Quality Assurance. National Committee on Vital and Health Statistics 1997a. Health Privacy and Confidentiality Recommendations. June 25, 1997. Washington, D.C. http://aspe.os.dhhs.gov/ncvhs/privrecs.htm (June 11, 1998). 1997b. Letter to Secretary Shalala on Unique Health Identifiers. September 9, 1997. Washington, D.C. http://aspe.os.dhhs.gov/ncvhs/uhid.htm (June 11, 1998). National Highway Traffic Safety Administration 1996. Why Data Linkage? The Importance of CODES. Washington, D.C.: U.S. Department of Transportation, National Highway Traffic Safety Administration. Available at http://www.nhtsa.dot.gov/people/ncsa/codes/whylink.html (April 28, 1998). National Library of Medicine 1998. Fact Sheet: Unified Medical Language System. National Institutes of Health. February 23, 1998. http://www.nlm.nih.gov/pubs/factsheets/umls.html (July 10, 1998). National Research Council 1993. Private Lives and Public Policies: Confidentiality and Accessibility of Government Statistics. G.T. Duncan, T.B. Jabine, and V.A. de Wolf, eds. Panel on Confidentiality and Data Access, Committee on National Statistics. Washington, D.C.: National Academy Press. 1997a. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. E.B. Perrin and J.J. Koshel, eds. Panel on Performance Measures and Data for Public Health Performance Partnership Grants, Committee on National Statistics. Washington, D.C.: National Academy Press. 1997b. For the Record: Protecting Electronic Health Information. Committee on Maintaining Privacy and Security in Health Care Applications of the National Information Infrastructure, Computer Science and Telecommunications Board. Washington, D.C.: National Academy Press. 1998. Providing National Statistics on Health and Social Welfare Programs in an Era of Change. Summary of a workshop. C.F. Citro, C.F. Manski, and J. Pepper, eds. Committee on National Statistics. Washington, D.C.: National Academy Press.

OCR for page 83
--> Office of the Secretary, U.S. Department of Health and Human Services 1998. National standard health care provider identifier. Proposed rule. May 7. Federal Register 63(88):25320–25357. Palmer, R.H. 1997. Process-based measures of quality: The need for detailed clinical data in large health care databases. Annals of Internal Medicine 127:733–738. Powe, N., J. Weiner, B. Starfield, M. Stuart, A. Baker, and D. Steinwachs 1996. The development and testing of a claims data-based approach for evaluating the care of patients with chronic illnesses. Medical Care 34:798–810. Public Health Foundation 1998. Measuring Health Objectives and Indicators: 1997 State and Local Capacity Survey. Washington, D.C.: Public Health Foundation. Ries, L.A.G., C.L. Kosary, B.F. Hankey, A. Harras, and B.K. Edwards, eds. 1997. SEER Cancer Statistics Review, 1973–1994. NIH Pub. No. 97-2789. Bethesda, Md.: National Cancer Institute. Rothman, K.J. 1986. Stratified analysis. Pp. 176–236 in Modern Epidemiology. Boston: Little, Brown. Rubin, H., M.W. Jenckes, M. Stuart, and M. Wickham 1994. Maryland Medicaid Recipients' Ratings of Prenatal and Pediatric Care by HMOs and Fee-for-Service Providers. Paper presented at 122nd meeting of the American Public Health Association (abstract, p. 324). Washington, D.C. Seitz, F., and B. Jonas 1998. Operational Definitions for Year 2000 Objectives: Priority Area 6, Mental Health and Mental Disorders. Healthy People 2000 Statistical Notes, No. 16. Hyattsville, Md.: National Center for Health Statistics. Starfield, B., N. Powe, J. Weiner, M. Stuart, D. Steinwachs, S. Scholle, and A. Gerstenberger 1994. Costs vs. quality in different types of primary care settings. JAMA 272(24):1903–1908. Starr, P. 1997. Smart technology, stunted policy: Developing health information networks. Health Affairs 16(3):91–105. Starr, P., and S. Starr 1995. Reinventing vital statistics: The impact of changes in information technology, welfare policy, and health care. Public Health Reports 110:534–544. Strouse, R., J. Hall, B.L. Carlson, and J. Cheng 1997. Impact of the Nontelephone Sample on Family Health Insurance Survey Estimates. Report submitted by Mathematica Policy Research, Inc., to the Robert Wood Johnson Foundation. March 3, 1997. Princeton, N.J. Stuart, M. 1994. Redefining boundaries in the financing and care of diabetes: The Maryland experience. Milbank Quarterly 72(4):679–694. 1995. Public health issues in the development of centralized health care databases. Journal of Public Health Management and Practice 1(2):31–38. Stuart, M., D. Steinwachs, J. Harlow, and M. Fox 1990. Ambulatory practice variation in Maryland: Implications for Medicaid cost management. Health Care Financing Review (Annual) December:57–67. Svikis, D., M. McCaul, T. Feng, M. Stuart, M. Fox, and E. Stokes 1998. Drug dependence during pregnancy: Effect of an on-site support group. Journal of Reproductive Medicine 43:799–805. Thacker, S.B., and D.F. Stroup 1994. Future directions for comprehensive public health surveillance and health information systems in the United States. American Journal of Epidemiology 140:383–397.

OCR for page 83
--> Turner, C.F., L. Ku, S.M. Rogers, L.D. Lindberg, J.H Pleck, and F.L. Sonenstein 1998. Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science 280(5365):867–873. U.S. Department of Health and Human Services 1991. Healthy People 2000: National Health Promotion and Disease Prevention Objectives. DHHS Pub. No. (PHS) 91-50212. Washington, D.C.: Office of the Assistant Secretary for Health. 1997a. 1997 Strategic Plan. September 30, 1997. Washington, D.C. 1997b. Confidentiality of Individually-Identifiable Health Information. Recommendations of the Secretary of Health and Human Services, pursuant to Section 264 of the Health Insurance Portability and Accountability Act of 1996. September 11, 1997. Washington, D.C. 1997c. Health Insurance Portability and Accountability Act of 1996 Administrative Simplification. Fact sheet. February 13, 1997. http://www.hhs.gov/news/press/1997pres/970213.html (June 29, 1998). 1997d. Major Provisions of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (P.L. 104-193). April 28, 1997. http://www.acf.dhhs.gov/news/welfare/aspesum.htm (July 8, 1998). 1997e. Progress review: Surveillance and data systems. Prevention Report 12(1). 1997f. Technical Assistance in the U.S. Department of Health and Human Services. Report of the Technical Assistance and Training Liaison Work Group. July 1997. Washington, D.C. http://aspe.os.dhhs.gov/progsys/ta/hhs-ta.htm (April 13, 1998)). 1998a. Leading Indicators for Healthy People 2000. Report from the HHS Working Group on Sentinel Objectives. Washington, D.C.: U.S. Department of Health and Human Services. 1998b. Registry of State-Level Efforts to Integrate Health Information. http://aspe.os.dhhs.gov/statreg/index.htm (February 9, 1998). U.S. General Accounting Office 1996. Executive Guide: Effectively Implementing the Government Performance and Results Act. GAO/GGD-96-118. Washington, D.C.: U.S. Government Printing Office. 1997. Managing for Results: Analytic Challenges in Measuring Performance. GAO/HEHS/GGD-97-138. Washington, D.C.: U.S. Government Printing Office. U.S. Public Health Service 1979. Healthy People: Surgeon General's Report on Health Promotion and Disease Prevention. DHEW (PHS) Pub. No. 79-55071. Washington, D.C.: U.S. Department of Health, Education, and Welfare.