Measuring the Quality of Health Care (1999)

In 1994, the Institute of Medicine's (IOM's) Council issued a white paper, America's Health in Transition: Protecting and Improving Quality (IOM, 1994a). That white paper was the start of a special initiative on quality of health care that included the formation of the IOM's National Roundtable on Health Care Quality, which has issued this statement. In the preface to the white paper, Kenneth Shine, M.D., president of the IOM stated that,

In January 1997, the presidents of the National Academy of Sciences, the National Academy of Engineering and the IOM issued a paper that synthesized, summarized, and highlighted principal conclusions and recommendations from recent studies (NAS, 1997).

This policy paper extends the IOM's efforts to inform policymakers, provider organizations and clinicians, purchasers, and consumers about the measurement of health care quality—its uses, methods, promise and current challenges. It is based on a conference held at the IOM in September 1996, “Measuring Quality of Care: State of The Art" and the conclusions of the members of the National Roundtable on Health Care Quality.

Viewed most broadly, the purpose of quality measurement is to secure for Americans the most health care value for society's very large investment. Knowledge about the state of quality is essential if policymakers are to understand the effects of health of services that are provided and how these effects may differ for different patient populations, health conditions, and settings of care. Such knowledge is also needed to understand whether the organization, delivery, and financing of health care is affecting quality of care, and if so how these health services have affected individual and population levels of physical, mental and social functioning. Furthermore, effectively functioning health care markets require that patients, employers, and other consumers have good information for decisionmaking, including knowledge about health plan, organization, and clinician performance and the efficacy, effectiveness, and cost-effectiveness of health services—both for new services and for those that are well established. In particular, measurement of health care quality serves a range of objectives, including the following:

• providing data to inform quality improvement efforts;
• inspecting and certifying that a facility or individual meets previously established standards;
• comparing groups for a variety of purposes, including selective contracting by purchasers and choice of providers and practitioners by individuals;
• informing patients, families, and employees about the health care decisions and choices they face;
• identifying and possibly eliminating substandard performers—those whose performance is so far below an acceptable level that immediate actions are needed;
• highlighting, rewarding, and disseminating best practices;
• monitoring and reporting information about changes in quality over time; and
• addressing the health needs of communities.

The Roundtable emphasizes that although quality measurement has many uses, one of the most important is to provide information that can be used to improve performance. Improving average performance requires excellent measurement of that performance. Measures used by organizations to improve quality as well as those used to compare organizational performance must be detailed, accurate, and timely to be useful. For example, measures for quality improvement must provide a level of clinical detail and site-specificity to allow managers

and clinicians to understand what to change. Measures used for organizational comparisons must include careful sampling and accurate risk- and severity-adjustment to ensure fairness when comparing organizations and individuals with one another or to assess change over time.

The IOM stated in 1990 in Medicare: A Strategy for Quality Assurance that "quality of care is the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge" (IOM, 1990, p. 21). This definition has been widely accepted and has proven to be a robust and useful reference in the formulation of practical approaches to quality assessment and improvement (Blumenthal, 1996). Several ideas in this definition deserve elaboration.

The term health services refers to a wide array of services that affect health, including those for physical and mental illnesses. Furthermore, the definition applies to many types of health care practitioners (physicians, nurses, and various other health professionals) and to all settings of care (from hospitals and nursing homes to physicians' offices, community sites, and even private homes).

The definition emphasizes that high quality care increases the likelihood of good outcomes. It is a reminder that quality is not identical to good outcomes. Poor outcomes occur despite the best possible health care because disease often defeats the best efforts of health care professionals. Conversely, patients may do well despite poor quality care because humans are resilient creatures. The term likelihood recognizes that there is always an unknown aspect of health care, but the services provided are expected to provide more benefit than harm, based on the best available information both about the patient and about the effectiveness of a particular kind of treatment for patients with similar health problems.

The inclusion in the definition of both populations and individuals draws attention to the different perspectives that need to be addressed. On the one hand, there is concern with the quality of care that individual organizations, health plans, and clinicians deliver. On the other hand, attention must be paid to the quality of care across the entire system. In particular, one must ask whether all parts of the population have access to needed and appropriate services, whether services meet or exceed their expectations, and whether their health status is improving. That focus embraces all groups, whether or not they have access to care and whether they are defined by cultural heritage, sociodemographic characteristics, geography (e.g., a state or a region), or diagnosis. It recognizes that such individuals will include the most vulnerable, whether the source of vulnerability is economic, the

rarity or severity of the health problem, physical frailty, or physical or emotional impairment.

The phrase desired health outcomes highlights the crucial link between the care that is provided and its effects on health. Focusing on outcomes requires clinicians to take their patients' preferences and values into account as together clinicians and patients make health care decisions. Determining what is good or poor quality of care requires knowledge of the values that individuals place on various health outcomes and how these may differ among individuals.

Current professional knowledge emphasizes that health professionals must stay abreast of the rapidly expanding and changing knowledge base and use such knowledge appropriately. No matter how good the understanding or measures of quality are today, health care professionals must always be prepared to revise them as new knowledge is generated about what works and what does not work effectively in health care to produce good outcomes for patients. Although the knowledge and practices of individual clinicians are important for high quality care, no health practitioner can stay abreast of this growing body of knowledge without the assistance of good systems of care and good information systems to help ensure that relevant and accurate health information is available when needed. Quality of care can be substantially improved by well-designed systems that prevent and minimize errors and the harms that such errors may cause by coordinating care among settings and among various practitioners.

The question is sometimes asked: why does this definition of quality not include the acknowledgment of constraints on resources? Given that views of good care include managing care to get good value for money, why not include in the definition a phrase related to ensuring that appropriate services are efficiently provided, identifying and implementing appropriate quality standards, protecting people from spending more on health care than its additional benefits warrant or subjecting them to more risks than the added benefit warrants?

Compelling reasons exist not to include resource constraints within a definition of quality itself. Quality of care should not be defined on a sliding scale in which judgments about quality vary according to what can or cannot be afforded. Rather, the useful concept of the value of health care incorporates both quality and cost in the following simple equation: value = quality/cost. This equation is a measure of the efficiency with which care is provided where quality produces more benefit than harm. Responsible parties (individuals, public and private payers, and societal agents) should be able to distinguish quality problems from those arising from resource availability whether they are imposed by budget and coverage constraints or by inefficient delivery of care, or both. If quality of care is deficient as measured by established criteria, we should be able to recognize it and then determine why. Reasons might include not only failures of systems of care, lack of knowledge or skills,

but also factors related to patients such as lack of access, insurance, or failure to adhere to therapeutic advice.

A comprehensive approach to measuring the quality of care requires attention to three different kinds of quality problems: too much care (overuse), too little care (underuse), and misuse (flaws and errors in technical and interpersonal aspects of care).

Examples of overuse include the excessive or unnecessary use of X-ray and other diagnostic tests, unnecessary surgical procedures, and overprescribing antibiotics and some mood-altering drugs (sec Advisory Commission, 1998; Chassin, et al., 1998; Schuster et al., 1998). Those practices may result in still further testing and procedures in a cascade of interventions that might have been avoided and that might make patients vulnerable to harmful side effects. They also waste money and resources that could be put to more effective use.

Many studies have demonstrated the large gap between what is known to be effective care and what patients actually receive, regardless of their ability to pay (see Advisory Commission, 1998; Chassin, et al., 1998; Schuster et al., 1998). For example, screening and preventive services such as mammography and immunizations are not as widely provided as most experts believe is appropriate, and many treatable conditions, including serious depression, are often not diagnosed. Even those individuals with insurance often face geographic, cultural, organizational, or other barriers that limit their abilities to seek or receive care. Others do not receive proper preventive, diagnostic, or therapeutic services if they lack health insurance, do not adhere to recommended therapy, or if they delay seeking care.

Most quality-of-care issues today are brought to light in the context of personal health care services for individuals, but many critical problems relate to the population as a whole. The country must be able to know how changes in the organization, financing, and delivery of care differentially affect certain groups of people, especially vulnerable or disadvantaged people who are most at risk of poor care or inadequate access to care. These tasks call for applying quality measures to all types of providers in both the private and the public sectors and to the extent feasible under all financing mechanisms. To measure underuse, however, requires denominator information—that is, identification of the group for whom services

would be appropriate. Although denominators can be readily identified for assessing many kinds of underuse problems, establishing a denominator in a general population may not be possible, in particular, when no eligible population is defined. Organizations with enrolled populations, such as managed care organizations, can assess underuse of appropriate services such as preventive care, because they have a defined denominator population that is eligible to receive these services.

Inferior care results when the performance of health care professionals or support systems is inadequate or if practitioners lack mastery of their clinical-practice fields, do not adequately explain key aspects of care, or cannot communicate well with their patients. Cases in point include preventable drug interactions and surgical mishaps, failure to monitor or follow up abnormal laboratory-test results, neglect of appropriate education and information for patients, lack of adequate coordination of care, and insensitivity to the ethnic and cultural characteristics of patients (sec Advisory Commission, 1998; Chassin, et al., 1998; Schuster et al., 1998). Inferior care may also result from failure to include patients as appropriate in decisionmaking or disregard of patient preferences regarding care options.

The literature over the last two decades has documented quality problems throughout the health care system—whether from overuse, underuse, or misuse (Schuster et al., 1998; Advisory Commission, 1998). Millions of Americans do not receive proven effective interventions that save lives and prevent disability. Perhaps an equal number suffer needlessly because they are exposed to the harms of unneeded health care services. Large numbers are injured because of preventable harm from medical treatment. These problems exist in managed care and fee-for-service systems, in large and small communities, and in all parts of the country.

Overwhelmingly, individuals are not to blame for these problems (Berwick, 1990; Leape, 1997). These problems tend to result in part from the immense amount of new knowledge about what works to improve health and what does not (Chassin, et al., 1998). Physicians do not have ready access to all the data that would be useful to them as they care for patients. In large part, quality problems result because health practitioners do not have delivery systems that assist in providing error-free care and in bringing to them timely and relevant information about the patients they care for. It should be emphasized that the object of quality measurement should not be to fix blame on organizations or individuals but to find opportunities to improve health and prevent harm.

Given this overview of the definition of quality and the kinds of quality problems that measurement is intended to measure, the remainder of this statement describes major approaches to quality measurement and the challenges.

In a classic formulation of the dimensions of quality of care almost 40 years ago, Avedis Donabedian (1966;1980) described quality as including: structure (viewed as the capacity to provide high quality care), process (now often termed performance), and outcomes.

In general, either processes or outcomes may be valid measures of quality. For an outcome to be a valid measure of quality, it must be closely related to processes of care that can be manipulated to affect the outcome. Likewise, for a process to be a valid measure of quality, it must be closely related to an outcome that people care about. The parts that each of these plays in quality measurement are described briefly below.

Structural measures of quality typically include the characteristics of the resources in the health care system, including individual practitioners, groups of practitioners, organizations and systems of care, geographic location, and accessibility of services. They are measures of the presumed capacity of the practitioner or provider to deliver quality health care. For health care professionals, this may include licensure, specialty board certification, and type of training. For facilities, they include government certification and private accreditation, physical attributes including safety, and policies and procedures.

One example of the use of structural measurement is in assessing nursing home care. Much of the discussion of nursing home quality and regulation for remedying known problems concerns the role of structure in determining quality—the facilities, staffing, and training of those who care for nursing home residents. Many residents of nursing homes have serious disabilities and problems that require skilled nursing care. The nursing home workforce, its training and its availability for patient care require careful review to determine whether quality of care is adequate.

Nowadays, the quality-of-care literature is full of discussions about performance measurement, which is the current terminology related to measuring

the process of care. In terms of clinical quality, such measurement often focuses on the diagnosis and management of disease and may also address preventive care such as screening for disease. The results of such measurement are being given to employees, for example, to help them choose health plans. They are sometimes used to create consumer "report cards" that present the results of a variety of quality measures in a standardized format that allows comparisons among plans.

Measures of performance may include interpersonal aspects of care, service, timeliness, and convenience. They may include such topics as providing patients with information and answering their questions and encouraging patients to share in decisionmaking if at all possible.

Technical aspects of care include the timeliness and accuracy of diagnosis, the appropriateness of therapy, complications, and mishaps during treatment, and coordination of care across delivery settings, episodes of care, and professional disciplines. Errors in carrying out the complex series of steps often involved in patient care may contribute to preventable deaths or failure to help patients return to health. Misuse of medications (e.g., the wrong medication choice or dose) are serious and frequent problems found in many organizations and practices.

In nursing homes, frequently cited problems include inadequate care plans, unsanitary and hazardous environments, and unsanitary food. Other issues revolve around performance such as a failure to maintain the dignity of and respect for the residents and the unnecessary use of restraints.

Large gaps exist between what is known to be efficacious in research settings and how such knowledge is used (if used at all) in usual settings of clinical care (Brook and Lohr, 1985). These failures to provide appropriate care (underuse and overuse) or to provide care without error or failure in the systems of care (misuse) can result in considerable harm to patients, including death. For this reason measures of performance are critical measures of quality.

It is important that process measures take into account patient preferences. That is, a given test or procedure may be indicated but not performed because of a patient's decision, and this does not indicate poor quality.

Health outcomes include the traditional measures of survival (now commonly expressed as risk-adjusted mortality), unintended effects of treatment (e.g., infection), and the relief of symptoms. Such measures may be specific to a given health problem and may focus on biomedical outcomes (e.g., five-year survival, complications from disease, or successful repair and rehabilitation after a knee injury) or more comprehensive assessments of the effect of an intervention.

Outcomes measures also included patient reports about their health (e.g., a scale that asks an individual if his or her health is "excellent, very good, good, fair, or poor"), or they may include detailed measures of function. Functional outcomes measures may center on limitations in performing daily activities such as going to work, attending school, doing housework, as well as physical, social, and mental functioning.

Patient satisfaction measures address various aspects of patient experience in comparison to their expectations. Well-developed instruments for measuring the effects of changes in systems of care are in use in a variety of care settings and are increasingly sophisticated.

Outcomes measurement is in some ways the ultimate form of quality measurement because what interests most people is whether care has improved the patient's health. Nevertheless, the pitfalls are great. They include the rarity of some adverse outcomes, the long time periods required for many outcomes to develop, and the difficulty in identifying the components of health outcomes that are attributable to action taken by the health care system. That is, the effect of health care services may be quite small in comparison to the effects of the social and physical environment, or a patient's genetic makeup or behaviors that affect a patient's health. Furthermore, to be useful for quality improvement, outcomes data need to provide information with a high level of clinical detail and be provided in a sufficiently clear manner that providers and managers can know what processes must be changed. Some experts in quality improvement urge that understanding the rate at which organizations are improving their care are better than using static measures to identify superior and poor performance at a single point in time.

Donald Berwick distinguishes between the now dominant use of "measurement for judgment" compared to "measurement for improvement" (Berwick, 1996). In the former category he includes report cards, benchmark comparisons, the accreditation process, and employer-based performance surveys. Two issues are particularly salient: first, minimizing unintentional unfair comparisons because of the lack of standardized definitions and inadequate risk adjustment and, second, minimizing the intentional gaming of quality measurement. Berwick and others have pointed out that when quality is measured and results are used to judge individuals and organizations (with the accompanying professional and financial implications of such public reporting), the measurement results quickly become subject to denial (in which the measures and data are attacked as deficient or wrong) and to manipulating the results of measurement, that is altering decisions about which patients or members should be included and excluded in measurement or even which patients should be enrolled or treated. In part, this gaming can be addressed by careful specification of measures and by having external parties audit the data. However, he points out that the use of measurement for judgment detracts from a primary reason for measurement which is to help improve care, so that disease and its effects on health can be addressed. A crucial and ongoing challenge in this area is finding

ways to achieve measurement for public reporting that do not undermine measurement for quality improvement.

Some observers question the relationship of structural measures or standards to either process or outcome measures because of little empirical evidence of direct connections. Structural standards may provide a baseline in terms of capacity but compliance does not assure that high quality care is being provided. Nor does their use clearly mean that high quality care cannot be provided unless these standards are met. However, continuing attention has been given to the importance of governance, financial structures, the health care workforce, and the capacity to provide accessible and coordinated care. Such standards have been combined with measures of performance and outcome to assess the quality of care (Donabedian, 1980, 1982, 1985; Koran, 1975a, 1975b).

Measures of the quality of care based on processes are well-developed in comparison to outcomes measures. Nevertheless, they are good measures only if those processes can be linked to outcomes that are important to patients. Similarly, outcomes are good measures of quality of care only to the extent that they can be linked to actions on the part of the health care system that can be changed. The actions that health care managers should take if they are aware of poor health outcomes are not always clear. Indeed, the accountability of individual practitioners and of health care systems for patient and population health outcomes is an issue that can often only partially be addressed by health care professionals as they may more accurately be understood as societal issues.

During the last decade, many in the health care system have begun to apply a model of quality improvement called continuous quality improvement or total quality management. One assumption of this model is that the health care organizations and systems within which professionals practice can always improve. One way to foster this improvement is to set up continuous monitoring systems that alert the organization when performance in some area is slipping or to confirm that efforts at improving care are succeeding, or both. For organizations that have embraced methods of continuous quality improvement, measurement of performance and outcomes is integral to their operations. In such cases the cost of measurement is part of the cost of doing business. Ideally, the collection of information is continuous and detailed, and external reporting of performance uses some of this information.

For more than 40 years, experts have been working to create reliable and valid ways of assessing the quality of health care for a wide range of diagnostic and therapeutic services and for a broad array of health and medical problems (Donaldson and Lohr, 1990). For some purposes, well-understood measurement tools can be put to immediate, widespread use; for others, the science of quality measurement is in an early stage of development. Many advances and refinements in the field of quality measurement have been made. As the acceptance of these quality measures has increased, so has the audience for them. With this has come the need to create a wider domain of measures that indicate the processes and outcomes of care, and address the concerns of consumers. Also needed are measures that have proven validity and reliability.

Examples of different kinds of measures and methods and their use in quality improvement programs were presented at a September 1996 IOM conference entitled Measuring the Quality of Health Care—State of the Art.* Their inclusion here does not mean to imply these are the only or necessarily the best measures. Rather, they are intended to convey the scope of measures. The examples include:

• Automated ways of reminding physicians and other practitioners about the appropriate use of antibiotics and the creation of a database about infectious agents and their treatments.
• The measurement of risk-adjusted mortality and investigation of the science and art of adjusting the measured outcomes of care to take into account the severity of a patient's illness and other risk factors such as the presence of other health conditions.
• Measuring errors that occur in organizations, especially in the administration of medications, so that organizations can pinpoint how such errors occur and how to prevent them.
• The development of patient-reported measures of quality that allow organizations to compare a patient's experience with the patient's expectations.
• Quality measurement in integrated delivery systems that include multiple settings of care. Such measurements seek to assess the performance of whole systems as well as the performance of parts of those systems for defined episodes of care so that quality improvement efforts within the system can be efficiently targeted.
• The translation of well-developed clinical practice guidelines (e.g., those for screening and prevention as well as those for condition-specific treatments) into performance measures for use by purchasers and patients.

These examples are described in more detail below.

Some quality measures began as research projects and have then been further developed for routine use. One example is the New York State Cardiac Surgery Reporting system instituted in the late 1980s, when large differences among hospitals in mortality rates following bypass surgery were being reported. The state health department compared the expected hospital mortality rates with the observed rates (Hannan et al., 1994). When large differences were found, the state health department helped the hospitals focus on developing useful interventions.

Measuring the underuse of effective services is a notoriously difficult approach to measuring quality. Yet, models adapting research methods have been shown to be useful in assessing underuse of certain procedures such as cardiac artery bypass surgery and angioplasty among women, Hispanics, and uninsured individuals in New York City.

As part of an effort to reduce adverse drug events (ADEs), which can increase the risk of hospital deaths two-fold and which can increase the cost of care, a team at Intermountain Healthcare (Salt Lake City) instituted a hospital antibiotic assistant software program that assists physicians in choosing an antibiotic (Pestotnik, 1996; Evans et al., 1997). This effort was intended to improve the use of medications by avoiding errors during the course of treatment rather than focusing more narrowly on tracking prescribing errors and ADEs. The program suggests to the clinician the prescription that best fits the patient's needs on the basis of a variety of clinical criteria. Practitioners can give feedback on the suggestions and can prescribe antibiotics other than those suggested. This program serves to improve and monitor appropriate antibiotic use and has resulted in extensive decreases in ADEs as well as cost savings. For example, in the seven year period in which the program has been used, ADEs related to all antimicrobial agents have decreased by more than 75 percent.

Numerous groups have invested in the translation of clinical guidelines into useful quality measures (Grimshaw and Russell, 1993). The Healthcare Education and Research Foundation in St. Paul, Minnesota, for example, uses national

guidelines for clinical evaluations to provide comparative information for purchasers and consumers of care. Southern California Kaiser Permanente uses evidence-based guidelines that are credible to clinicians and that can be used not only to guide care, but also to assess and improve care. These have included guidelines for cervical cancer screening that have resulted in higher screening and follow-up rates. Similarly, Group Health Cooperative (Puget Sound) has developed a database for guidelines on screening and prevention and for condition-specific treatments that serve as the basis for quality measurement. The Pacific Business Group on Health, a coalition of large health care purchasers, has developed a collaborative reporting effort among health plans in California, the California Collaborative Healthcare Reporting Initiative, which reports performance and outcomes data. Another group, the Foundation for Accountability, has endorsed condition-specific quality measurement sets for several diseases.

It is now increasingly possible to measure systematically a variety of outcomes, including patient-centered measures of health status, ratings, and reports about their care. For example, the Consumer Assessment of Health Plans Study (known as CAHPS), sponsored by the Agency for Health Care Policy and Research, has developed a tool to assess consumer experience in health plans. The CAHPS survey can be used for health plan accreditation and for health plan performance reporting. Version 2.0 has a core set of questions used by purchasers and commercial health plans. Additional questions are added for Medicaid and Medicare beneficiaries. The hope of CAHPS developers and users is that the information can be used along with other measures of process and outcomes to help to inform patients and purchasers and to create a market that responds not just to price, but also to quality of care information.

For hospital care, Cleveland Health Quality Choice, a voluntary collaboration between providers and purchasers, uses surveys to evaluate hospital performance for patients discharged from surgical, medical, and obstetric care units. The Picker Institute in Boston, Massachusetts has also developed a series of survey instruments to help organizations better understand patients' experiences with care and their concerns.

With the increasing complexity of systems of health care, the field of quality measurement has had to respond to the information needs of those involved in quality improvement and measurement within health systems. One example of these efforts is the Consortium for Research on Indicators of System Performance (CRISP). CRISP focuses on the performance of whole systems as well as

on the performance of parts of those systems for defined episodes of care so that quality improvement efforts within the system can be efficiently targeted.

For example, using an algorithm developed by cardiologists, CRISP collaborators have assessed the process of care for acute myocardial infarction from initial chest pain through discharge and follow-up. They created specific measurement points reflecting both the process and the outcome of care. After finding variations among groups, the next step was to identify what these variations meant and at what level of the system the problems were occurring.

Measures have been developed and adopted in the private sector as well as in public health systems. These include measures used by the Department of Defense (DoD) and the Department of Veterans Affairs (DVA) which administers the largest integrated health system in the United States. The DVA measures access to care, the technical quality of care, the cost of care, patient satisfaction with care, and functional status.

DoD's Division of Quality Management oversees the review of care in 14 defined regions covering nine million beneficiaries, including the care provided in 100 hospitals and 500 free-standing clinics. One example of quality measurement included risk-adjusted outcomes of obstetric care (for mother and child) and outcomes scores that reflected clinical outcomes, resource use, functional status, and satisfaction. These scores provided clinical practice profiles that show the relationship between quality and cost and were used to identify ''best-practice hospitals" and to understand how those hospitals achieved what they did.

With the ever increasing number of measures available, it will be increasingly difficult to create a manageable measurement program. As the number of measures continue to increase, so does the need for sources and tools that can be used to choose measures that have proven validity and consistency. Some efforts have been begun to assist in finding and choosing measures. The Computerized Needs-Oriented Quality Evaluation System (known as CONQUEST) is an example. This database was developed with Agency for Health Care Policy and Research support to assist those involved in local quality measurement and improvement efforts to choose and tailor their measurements to fit the needs of their program. Additionally, the Joint Commission on Accreditation of Health Care Organizations (JCAHO) assembled the National Library of Healthcare Indicators, a catalog of measures recommended for use in the JCAHO accreditation programs.

The National Committee for Quality Assurance (NCQA) has had a major influence on quality measurement in ambulatory settings through its standardized set of measures, the Health Plan Employer Data and Information Set HEDIS(version 3.0). It is designed to permit purchasers and consumers to make valid comparisons across health plans. NCQA's Quality Compass product pro-

vides a set of comparative performance reports on health plans. Several states also provide such comparative information. Its reports can be used by employers to determine which plans perform best on HEDIS measures.

The Foundation For Accountability (FACCT) (Portland, OR), a nonprofit organization that includes purchasers, consumer and patient organizations, and government agencies has been developing health care measures for clinical conditions and productivity-related outcomes.

Despite these and other activities being undertaken to help health care organizations measure performance and report data and to help employers interpret those data, a large gap still remains in understanding what quality of care data the consumers would find useful, and a major challenge remains in educating consumers about these issues and presenting them with information in ways that will help them make decisions.

Both public and private organizations have typically been involved, often cooperatively, in work to devise valid, reliable, and practical ways of measuring and comparing the quality of care provided by health plans, institutions, and clinicians. At present, individual health care organizations pay for data collection and analysis of quality. They may be required to do so by third party payers such as public payers or by state regulation. Not all organizations, however, choose to invest in quality measurement and may, indeed find that doing so does not promote their competitive market position. Neither does declining to participate necessarily damage their market position.

Some private-sector organizations have invested in the development of quality measures. For example, JCAHO, NCQA, FACCT, and the American Accreditation HealthCare Commission/URAC, are examples of private-sector organizations involved in developing quality of care standards. Their governing boards are typically composed of multiple parties, including representatives of providers, health plans, corporate health benefits departments, government, and consumers. These organizations, however, have relatively small budgets compared to the budgets needed to work to develop measures and to assemble a cadre of experts to further that development. Further, quality measurement may require collating information from many sources—inpatient and outpatient medical records, laboratory records, and pharmacy records—to retrieve the information that is needed.

The multiplicity of public agencies at the federal and state levels with oversight responsibility and the range of private organizations that accredit health care organizations and review care, as well as the internal quality improvement efforts of health plans, would lead some to believe that assurance of quality is well in

hand. Unfortunately, duplication of effort and gaps in measurement coexist. For example, methods for adjusting health outcomes and performance measures to reflect differences in the age, health status, and other characteristics of health-plan members or other populations are improving but are still inadequate.

The development of quality measures sometimes occurs during research projects. Some are developed by organizations supported by a variety of users such as employers who originally supported development of the HEDIS data sets. Some measures are developed by companies whose products are proprietary. The development, evaluation, and dissemination of the next generation of quality measures will be expensive if they are done well, but such an investment is a public good that is unlikely to be supported and shared by one organization in a competitive marketplace if the benefits are to be gained by all. Thus, those who develop quality measures must look to public and philanthropic sources of funding for the needed investment as is the case for health-services researchers, government agencies, health plans, purchaser coalitions, and others who have done much to improve ways of measuring health outcomes, comparing the outcomes of different health care practices, evaluating the performance of health care providers and practitioners, and developing credible and useful guidance for patients and clinicians in making medical decisions. It is important that U.S. Congress and private organizations continue to support this knowledge-building work with the joint goals of improving average performance and correcting substandard practices.

Outcomes research will provide information about appropriate and effective care. As this information is translated into clinical practice guidelines and criteria for measuring quality, there will be a constant need to update this information and ensure that new information is disseminated to clinicians and the public. With the very rapid increase in the use of Internet resources, for example, individuals and clinicians find enormous amounts of material about the treatment for certain health conditions and, increasingly, assessments of health care quality. It is difficult to assess the validity of much of this material, and attention is turning not only to amassing information but also toward ways (1) of indicating how valid and reliable such information is and (2) of collecting and displaying information at the appropriate level for a variety of users.

Challenges in quality measurement remain. These include the development of data systems that will make the measurement of health care quality as valid and efficient as possible, draw on progress in evidence-based medicine to facilitate the development of valid quality-of-care criteria, establish standard defini-

tions and measurement methods, advance quality measurement that take into consideration differences in severity of illness, ensure accurate public reporting about the quality of care, promote the development of a full range of measures of health care quality that include underprovision and overprovision of care as well as variable provision of effective services for all populations, and incorporate quality measurement into systems of improvement that actually promote and create high quality, cost-effective care delivery.

As the numbers of measures increases, it becomes increasingly important to evaluate them in terms of how valid they are for the purpose intended (do they measure what they are intended to measure?), how reliable they are (do repeated measures give the same results?), and how suitable they are for different populations, such as for patients of different ages and with different health conditions of for those who are very ill. It is also important that users understand the size of the sample needed to obtain meaningful results and to understand how effective these measures are in providing the information needed by clinicians and managers to improve care. McGlynn (1988) has proposed the following criteria for evaluation of quality measures: that they are clinically meaningful, scientifically sound, and interpretable as judged by clinical content detail and specificity and by the intended audience.

The desire to improve the quality and usefulness of health care data is shared by patients, practitioners, administrators, researchers, and policymakers throughout the United States. Much useful work has been and can continue to be done without sophisticated data systems. Yet, primary review of medical records, which are still overwhelmingly paper-based records, is often the only way to collect data with the level of clinical detail needed to assess care. This is extraordinarily labor intensive. For example, the Kaiser Foundation Health Plans have estimated that they performed 150,000 chart reviews for the first year of HEDIS Version 3.0 reporting (Lowry, 1997).

The rapid development of computer applications to health care continues to provide hope that the long awaited era of computer-based patient records is finally approaching. This role of the computer-based patient record in the care process provides the information about a patient needed to support clinical decisions in a more timely manner. It can also be a key information source for quality review and improvement. NCQA has laid out an information framework for successful performance measurement (NCQA, 1997). Nevertheless, the challenges in developing the information systems needed to support a move to computer-

based patient records in community-based practices and other nonhospital settings remain daunting.

Technology can help, but it will not solve all data-related problems of quality measurement. An essential part of quality measurement is determining a rate at which something occurs. To do so in a population requires a denominator based on a defined population in which some event might occur (e.g., the rate of mammography screening for women age 50 or older). The spread of managed care organizations offers the ability to measure such rates because such organizations have defined populations. Many patients are not in managed care organizations, however, and such rates cannot now be measured.

Knowledge about effective clinical care is needed for the development of clinical guidelines for care and for the development of accurate review for measurement of the quality of health care. In the United States and other parts of the world, efforts are now turning to the development of evidence-based centers to assist in managing the explosion of new information about the effectiveness of medical devices, tests, and therapies. These include the Cochrane Centers in the United Kingdom and Canada and the newly established centers of evidence-based health care supported by the Agency for Health Care Policy and Research (U.S. Department of Human Services) in the United States.

Such centers focus on assembling the best available evidence about the diagnosis and treatment of given conditions; on assessing the quality of that evidence regarding the safety, effectiveness, and cost-effectiveness of tests, treatments, services, and health policies; disseminating the results of their findings; and encouraging the rapid incorporation of these findings into clinical practice.

Measures vary in a variety of ways, for example, the definitions of the numerators, denominators, and the time periods specified for data collection. Even what at first appear to be self-evident measures such as immunization rates for children often vary. This means that results based on different measures cannot be compared fairly. The establishment of common definitions and measurement methods is essential if comparisons over time and from one organization to another are to be valid. according to the time frames, ages, and types of immunizations that are measured. Libraries of quality indicators and the establishment of a uniform data sets such as that in HEDIS are first steps in the specification of measures and data elements within those measures. It is not clear that there needs to be single, standardized definitions for measures at this time, but measures

used for comparisons must be standardized for the comparisons in which they are used and be auditable to verify the use of whatever definition is used.

One conspicuous need in the measurement field is the development and application of accepted standards and criteria for quality measurement tools and instruments that take into consideration differences in such factors as patient age and severity of illness. Such risk-adjustment tools are critical, in particular, when making comparisons among individual practitioners and organizations. The paucity of suitable routinely collected data on patient risk characteristics is a key barrier to risk adjustment. Adjustment methods for assessing the outcomes of coronary-artery bypass surgery among patients with various clinical characteristics and conditions are now sufficiently reliable, however, that differences in mortality among institutions can be used as markers of quality (Hannan et al., 1990). However, methods for adjusting health outcomes and performance measures are still imperfect in fields other than cardiac surgery (Iezzoni, 1997).

Another challenge to quality measurement is to ensure the accuracy of data used to provide information about quality. Inaccurate data may result from several sources including: random or inadvertent errors by data collectors, missing data, inconsistent use of definitions and criteria for inclusion, inappropriate aggregation of data, systematic miscoding, for example, of risk factors intended to bias the data in favor of the reporting organization (Chassin, 1996). When data are systematically biased in favor of the reporting organization, they may mislead users both about the magnitude of problems and the relative performance of organizations. When competitive position depends on such performance data, the incentives for systematic bias in reporting clearly increase. One response used, for example by NCQA, has been to require auditing of a data sample by external reviewers. Data collected because of external demands and which has little value for clinical care and management is more likely to be biased than data collected as part of the clinical process itself. The challenge of ensuring that data on quality are both accurate and valued in the marketplace requires careful attention in building information systems and devising market-based incentives.

The development of databases for quality assessment has great potential benefits for health care, but attention to the security of information and to adequate

and effective policies for protecting the confidentiality of health-related information is essential (IOM, 1994b).

Many stakeholders agree on the value of information about quality of care, and there is growing demand for the collection of data that will permit such information to be disseminated. Usually there is no need for person-identifiable data, and aggregate data serves well for performance information. In some, but not all, efforts, it is useful to have person-identified or person-identifiable data that will allow linkage of data from various sources such as laboratory, hospital admission, ambulatory care, and pharmaceutical databases. Such linkage of person-identifiable data creates justifiable concern about the confidentiality of information and harms that might result from disclosure and redisclosure of such information for unauthorized uses.

At the same time, actions to restrict the availability of patient-identified information could severely affect quality measurement and improvement initiatives if such policies do not take into account the appropriate need for access to health care data by those who need access for these purposes. The movement toward proprietary databases is another significant challenge to data acquisition for quality improvement.

Current policy discussions (e.g., about implementation of the Health Insurance Portability and Accountability Act of 1996) center on the form of patient identifier that would be most useful for insurance purposes, research, and quality of care yet could be protected from misuse. The more general issues of data confidentiality, however, include information-use policy (who should be permitted to use these data and for what purposes), and security policies and procedures related to protecting data integrity and preventing unauthorized access. All such issues involve first, well thought out policies and procedures, and second, hardware and software implementation of those policies. Overreaching efforts at data protection can stymie legitimate efforts to learn from patients about their health outcomes, and policymakers will need to consider carefully how best to accomplish both goals.

What is the cost of quality? Does increasing quality cost more? Three issues are of paramount importance. First, what is the cost of disquality (poor quality)? Second, what is the cost of measurement, and how can those costs be justified in comparison to spending on other goals such as outreach to populations or for other non-health care related priorities of society? Third, are the benefits of quality improvement large enough to justify expenditures for quality improvement?

The cost of measuring and improving quality of care cannot be understood without an estimate of the cost of poor quality. The cost savings of disquality resulting from misuse and overuse include direct costs in terms of repeated tests

and procedures, patient visits that might have been avoided, and hospital days and medications that might otherwise have been unnecessary. The burden of poor quality can include work days and school days missed, anxiety, and loss of quality-adjusted life years for patients as well as added anxiety for patients, family members, friend, employers, and society. Improving quality by providing underused services might in some cases increase the direct costs of the care that is provided (e.g., preventive care or a needed procedure), but it is also possible that some of these costs would be offset by the savings from the prevention of a more severe illness later.

Some observers believe that in the long run, even if not in the short-run, good quality is less costly than poor quality. Even if underprovision of services results in greater short-term costs, they assert that in the long run, individual patients and their communities will have more cost effective care if they receive all needed services in a timely manner.

Yet, measuring quality does require resources, and most of these costs are eventually borne by patients and society in general. Costs can also be understood as opportunity costs—that is, how resources might be used for the next best alternative, whether for providing currently underprovided health services or other benefits to society. Measuring quality for public reporting may divert considerable resources that might otherwise be used to provide additional services or to improve services. The well-known HEDIS reporting system, although voluntary for health plans, is only one data set to which health plans are asked to respond. Other data requirements come from employers, state health and insurance departments, and public payers such as Medicaid agencies. To the extent that organizations focus their energies on public reporting and neglect identified issues of clinical importance in their own facilities, these are lost opportunity costs.

Some costs can probably be reduced as data gathering becomes more efficient. As measures evolve and institutions find ways of using the information for internal purposes, these investments become part of quality improvement and should eventually help to improve health status. Other costs may increase as more measures are included and auditing of data for accuracy is implemented. Research is also needed to determine the most efficient ways of achieving these improvements. All such questions are topics requiring evaluation

When addressing quality in managed care organizations or integrated delivery systems, it is not always clear who is responsible for that performance. In many geographic areas, physicians contract with many (more than a dozen) plans. It is difficult to know if differences in performance are the result of differences in plans or in the performance of the medical groups. The answer is likely to vary considerably depending on the amount of control that the managed care

organization exercises and the amount of assistance it provides in meeting expected standards of care.

Quality-of-care concerns have often been set aside to tackle the seemingly more pressing problems of financing and access. Nevertheless, there are compelling reasons to confront quality matters with the same vigor and sophistication as those directed at issues of cost.

The messages of this statement are that (1) that the quality of health care can be measured and improved and (2) that quality of care should be measured with continued and increased vigor. Pursuing this objective means identifying and assessing the risks and opportunities posed by the changes in health care in the United States. It also means describing how health care organizations and clinicians should be accountable to patients and society and, conversely, how individuals can take appropriate responsibility for their own health.

The Roundtable emphasizes that despite cautions about their careful use and interpretation, good measures of quality of care exist. Clinicians, managers, and health plans can use information about quality to guide improvement and consumer choice. Increasingly useful and understandable information is becoming available to public and private purchasers and to patients and their families. Although health care organizations now have some tools for measuring quality, significant additional resources are likely to be needed for their further development. The costs of measurement, however, need to be acknowledged.

Both internal quality improvement efforts and external monitoring have important places in health care. Internal quality improvement efforts aimed at better planning and delivery of care are essential. At the same time, external monitoring of quality of care is necessary to ensure the integrity of the quality-of-care information and so that assessments can be made from a broader population perspective. The necessity of having a dual approach—internal and external quality monitoring and improvement—as a way of understanding the effects of a changing environment on quality of care is not widely understood by the health care community or policymakers. Improving the quality of health and health care requires attention to the processes and outcomes of health services rendered to individuals, with adequate adjustments for the different disease, risk-factor, and personal characteristics of those individuals.

Policymakers can, with the implementation of adequate monitoring of quality come to a better understanding about the effects of changes in health policy, financing, delivery, and market environments on quality to make wise policy decisions. Americans and the health care systems on which they rely urgently need such actions. Actions taken need to made based on the best available evidence. For this reason, the Roundtable believes that it is vitally important to pursue several tasks.

They include the need to inform policymakers and others responsible for determining how quality is to be measured so that they may:

• identify the populations for whom quality of care should be monitored, from individuals and members of health plans to groups with special health needs, communities, and the nation as a whole as part of an effort to monitor the quality of care across the entire system;
• confront the full range of quality-of-care problems, including overuse and underuse of services as well as deficiencies in the technical and interpersonal aspects of health care;
• include valid quality measures whether based on process or outcomes of care;
• consider how to create reliable, uniform data systems and collect consistent data from a variety of sources;
• apply and refine tools and techniques of quality measurement and improvement that will help health care reform succeed;
• devise means of providing valid, useful information on health care quality to consumers, providers, payers, and policymakers; and
• promote communication and education about quality of care and the use of quality measures.
• include quality measurement into systems of improvement that actually promote and create high quality, cost-effective care delivery.