National Academies Press: OpenBook

Performance Measurement: Accelerating Improvement (2006)

Chapter: 4 Moving Forward: What Should Be Measured?

« Previous: 3 Achieving a National System for Performance Measurement and Reporting
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

4
Moving Forward: What Should Be Measured?

CHAPTER SUMMARY

This chapter describes the approach used by the committee to select a starter set of performance measures and identifies significant gaps in the scope of existing measures. In addition to recommending a starter set of measures drawn from earlier work of stakeholder groups, the committee proposes four approaches to address identified gaps in existing measures: comprehensive measurement; longitudinal measurement; patient-level, population-based, and systems-level measurement; and shared accountability.

The committee is convinced that performance measurement is a prerequisite for improving both health and health care in the United States, and that it requires clear stewardship at the federal level. The committee is therefore recommending the establishment of a National Quality Coordination Board (NQCB) housed within the U.S. Department of Health and Human Services to perform this guiding function while working collaboratively with existing stakeholder groups (see Chapter 3).

An important function of the NQCB will be harmonizing current efforts to establish standardized performance measures. Accordingly, this chapter focuses on how the quality of health care services should be measured. The committee performed a comprehensive review of available standardized performance measures for health care services delivered in the ambulatory, acute, long-term care, and in-center hemodialysis settings and evaluated the nature and scope of these measures in light of the 10 design principles articulated in Chapter 2 and Appendix D. Based on this review, the committee identifies critical gaps in existing measures and proposes a starter set of measures that are available for immediate implementation.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

APPROACH

The committee approached the challenge of selecting a starter set of performance measures by first identifying the analytic frameworks for quality assessment that have guided the development of measures in the past. The most important of these were the six aims set forth in the Quality Chasm report (IOM, 2001) and the Foundation for Accountability’s call for assessing care across the lifespan (FACCT, 1997). The committee then identified leading performance measures and measure sets, classifying them within the existing analytic frameworks. A full description of the selection and classification methodology can be found in Appendix E.

The committee recognizes limitations to this approach, as the primary emphasis was on measurement of health care services. This focus constrained the committee’s ability to include measures of other important areas that have a profound impact on health outcomes, such as health behaviors and disparities in care. The committee also acknowledges the difficulty of adopting these measures, particularly for certain providers. For example, rural hospitals will face different barriers to implementation from those faced by community-based hospitals or academic health centers. The committee’s approach did, however, make it possible to identify the major gaps in current performance measure sets and to specify the 10 design principles for a performance measurement and reporting system set forth in Chapter 2. These 10 design principles, in turn, provided an additional lens for the classification of current measures, as well as a basis for recommending next steps.

GAPS IN CURRENT MEASURES AND IMPLICATIONS FOR THE DESIGN OF A PERFORMANCE MEASUREMENT AND REPORTING SYSTEM

The committee reviewed more than 800 measures within the analytic frameworks noted above. As a result of this effort, the committee identified several major gaps in existing measure sets, summarized in Table 4-1. The following sections highlight those areas in which the committee proposes significant changes in direction or new emphasis in performance measurement, as embodied in the following approaches:

  • Comprehensive measurement

  • Longitudinal measurement

  • Individual patient-level, population-based, and systems-level measurement

  • Shared accountability

These approaches represent a change relative to current performance measurement efforts as they provide different frameworks through which

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

TABLE 4-1 Gaps in Current Performance Measure Sets

Gap

Relevant Design Principlesa

Description

Limited scope of measurement: Few measures of patient-centered care, equity, or efficiency. Few measures for children or those at the end of life. Many important conditions unrepresented in measures.

Principle 1: Comprehensive measurement

A performance measurement system should advance the core purpose of the health care system and foster improvements in all six aims identified in the Quality Chasm report (IOM, 2001): safety, effectiveness, patient-centeredness, timeliness, efficiency, and equity.

Narrow time window: Most measures focus on a single point in time and do not assess care across settings.

Principle 3: Longitudinal measurement

Standardized performance measures should characterize health and health care both within and across settings and over time.

A provider-centric focus: Current measures focus on existing silos of care (e.g., physician’s office, hospital)

Principle 7: Individual patient-level, population-based, and systems-level measurement

Measurement and measures should assess the health and health care of both individuals and populations and the many systems within which care is provided.

Narrow focus of accountability: Most measures focus on an individual provider’s actions.

Principle 8: Shared accountability

Measurement should not be constrained by the absence of a current, identifiable, single responsible agent.

aDrawn from Table 2-2 Design Principles for a National System for Performance Measurement and Reporting.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

quality can be measured. The committee believes these approaches are essential to achieving higher-quality health care. Box 4-1 illustrates how the above approaches might be implemented to affect the way care is delivered and yield better health and health care.

BOX 4-1
Illustration of How Approaches to Address Gaps in Performance Measures Might Be Implemented

David is a 67-year-old man living with diabetes mellitus. Over the years, his diabetes has contributed to other conditions, such as heart disease, hypertension, and neuropathy. David sees multiple clinicians, including a primary care physician, cardiovascular specialist, podiatrist, and ophthalmologist. He also takes a total of eight prescription drugs to manage his multiple chronic conditions.


Current Health Care Delivery System

During a recent visit, David’s primary care physician ordered a battery of tests to monitor his condition, including hemoglobin A1c and cholesterol testing. His physician also referred him to an ophthalmologist based on David’s self-report of blurred vision. In addition, David is seeing a cardiologist, who repeated the blood tests ordered by his primary physician as his medical records were not readily available at the time of his visit. His cardiologist also prescribed a cholesterol-lowering drug and high blood pressure medication, which were called in to the pharmacy. Upon checking David’s medication history, the pharmacist noticed that one of the drugs prescribed by the cardiologist was known to have an interaction with another medication he was taking. The pharmacist alerted the cardiologist, who had an incomplete drug history on David, as it was hard for David to remember all the “pills” he was taking and he forgot to bring his prescription bottles to his visit like a friend had recommended. David also had an appointment with a podiatrist as his primary care physician also noted he should have an annual foot exam on his chart. David did not make it to his podiatrist appointment because of transportation issues. Nor did he see the ophthalmologist because he never received the referral paperwork required by his insurance carrier. Upon returning to his primary care physician with complaints of fatigue and “not feeling so good” his physician noticed there were no results in his chart for his eye or foot exam, and the blood work he ordered showed David’s hemoglobin A1c was elevated. His physician makes another referral explaining how important it is for him to get these screenings. He also spoke with David about his diet and monitoring his glucose levels and requested another referral to a dietician.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

Under the current health care system, David’s care is fragmented. Rarely is David asked what he thinks of his care and how well it accommodates his lifestyle. Most of his providers lack a vehicle, such as an electronic health record, with which they can seamlessly communicate patient health information, including treatment plans and laboratory test results. As a result, tests are repeated, histories are retaken, and in some cases, conflicting medications are prescribed.

Not only is David’s care inefficient at the patient level, but it also reflects the waste of resources that characterizes the current health care system. David is not alone, for he serves as an example of how patients are often treated today, augmenting the potential waste created by the many inefficiencies of the health care services system.

Assessing the health care system requires expanding measurement from the individual patient treated by individual physicians to that of the larger community in which David lives. In addition to care delivery services, David’s health is also influenced by other environmental factors in his community. Thus it is important to know how well the community as a whole is performing in regards to the overall health of its diabetics. For example, promotion of preventive services and environmental factors such as having walking paths to promote exercise can impact health in the community.


Future Health Care Delivery System

Through the approaches identified by the committee as leading to better health care through a national system for performance measurement and reporting—comprehensive measurement; longitudinal measurement; individual-patient-level, population-based, and systems-level measurement; and shared accountability—many of the problems David encountered in his care can begin to be addressed.

Comprehensive measurement. Effectiveness measures that adequately document David’s health, especially with respect to his various complex conditions, should be used to assess the quality of care he has received, including safety issues related to drug interactions. Also important to capture in a more inclusive set of measures is David’s perspective on his own care. The array of measures collected by his doctors should make it possible to monitor the course of his disease, as well as all of his health care needs, throughout his lifetime.

Longitudinal measurement. A major barrier to the provision of high-quality care was the lack of communication among David’s providers. The inability to transfer records quickly among all of his physicians not only was inefficient because of duplication of effort, but also posed a threat to his care. With proper attention to care transitions, much of this waste could be avoided. Moreover, further complications, such as David’s

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

blurred vision, could be identified and treated more quickly given assurance that proper follow-up services were available and utilized.

When assessing the quality of the health care delivery system treating David, outcomes and costs should be considered. In a hospital, for example, measurements of the ability of David to perform daily activities and function both physically and mentally at normal levels would be important outcomes. Combined with the costs associated with treating these patients, this information would permit an overall assessment of the longitudinal efficiency of the hospital systems.

Individual-patient-level, population-based, and systems-level measurement. While assessing care at the individual patient level, David’s doctors could measure the comprehensiveness of his care through the use of composites. Composite measures of his diabetes testing for a predetermined bundle of routine disease-specific measures, such as checking for hemoglobin levels, blood pressure management, and eye and foot exams, would provide a complete picture of the evidence-based care David should be receiving. They would also allow David to become a more informed patient, aware of what types of treatment he should, at a minimum, be receiving. As an active participant in his care, David could also collaborate with his physician to ensure that he received all recommended treatment protocols.

When measuring care based on a given population, David’s care measures could be aggregated with those of others, such as members of his local community, socioeconomic group, and state. These measures of personal health can be evaluated in combination with data reflecting the public and population health systems to better assess the overall health care system. This information would depict how well those in his population were living with their chronic illnesses, as well as provide tangible data for comparison with other populations.

Shared accountability. David’s multiple caregivers should take responsibility for ensuring that his care is well coordinated and responsive to his individual needs. This would require his clinicians to embrace a more holistic approach to care, as opposed to practicing in a way that targets a single specialty. For example, David’s cardiovascular specialist would also want to ensure that preventive testing, such as foot and eye exams, was performed. If these tests were not performed, she could take corrective action and contact David’s other providers. This does not ensure that David’s health care will be more coordinated; however, it is important that all the players involved with providing David’s care have the opportunity to affect his health without having to worry about being held liable for the actions of others. Finally, just as David is an example of how well an individual patient’s physicians interact, care provided to larger patient populations reflects the interactions among the various systems these populations encounter.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

Comprehensive Measurement

Current performance measure sets are far too limited in scope. The vast majority of current measures assess the quality of health care in terms of effectiveness and safety. Only a few, limited measures examine timeliness and provide insight into patients’ experiences, and hardly any adequately assess the efficiency or equity of care. Nor do measures adequately cover the entire human lifespan, as very few evaluate care for children, adolescents, or those at the end of life. Finally, too few measures exist that address matters particularly salient for the Medicare population, such as chronic obstructive pulmonary disease, stroke, dementia, and Alzheimer’s disease.

The committee believes a complete set of measures should offer a far more comprehensive assessment of performance across all of these important dimensions. A measurement system should fully address all six quality aims, in part to help keep providers from focusing on only one area of improvement to the detriment of others. Achieving such comprehensive measurement will require substantial investment in both fine-tuning existing measures and developing new measures where significant gaps exist.

There are a number of reasons for the limited breadth of existing measures, such as the absence of a leader to coordinate and guide existing efforts; a shortage of consensus, evidenced-based guidelines; inadequate financial support for ongoing measurement-related activities; and consensus-driven processes as opposed to goal-driven agendas. The committee recommends that the NQCB assume a leadership role by establishing national goals on which future measure development should focus. In this role, the NQCB should collaborate with stakeholders to produce guidelines that can serve as the foundation for measure development. With this more focused and coordinated effort, private and public funding could be garnered to support innovation and measure development that would aid in achieving national goals.

Longitudinal Measurement

The committee’s emphasis on longitudinal measurement is based on two distinct concerns. First, both the U.S. fee-for-service system and the performance measures currently in use reinforce, although not intentionally, the separation of settings of care by design (i.e., ambulatory care, home health care, hospital care, and nursing home care). This emphasis on separate care settings has several adverse effects, including fragmentation, lack of continuity, and poor communication. Second, the effectiveness of a care system should ideally be reflected in its capacity to prolong life, maintain or improve functioning and the quality of life, and achieve these health outcomes with a high degree of patient centeredness and efficiency. Achievement of these results generally involves care that crosses boundaries, rather than the actions

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

of a particular caregiver at a specific point in time. Measurement that focuses only on such fragments of care misses too much of what really matters to patients. Rather, measure sets should focus on measures of continuity and transitional care, as well as on longitudinal assessments of health outcomes and costs (Coleman et al., 2003; Rogers et al., 2004).

The committee recognizes that measuring care across settings, long-term outcomes, and costs for selected conditions will be complex, as it will require a shift away from assessing and reporting how care is delivered at one point in time to a given patient in a given setting. It will also be necessary to acknowledge and incorporate patients’ perspectives on their care and health outcomes when evaluating quality. The committee believes, however, that these areas of measurement are integral to a broader understanding of how well health services are provided and can be addressed through organized and focused research efforts.

Measures of Continuity and Transitions

Patient transfers between care settings are common. Issues of care transition affect primarily those living with multiple or complex conditions and are highly relevant to the Medicare population of adults 65 and over, to children with special health care needs, and to the disabled. A study tracking posthospital transitions for 30 days after discharge among a national sample of Medicare beneficiaries found that 61 percent of care episodes resulted in one transition, 18 percent in two transitions, 9 percent in three transitions, and 4 percent in four or more transitions, while 8 percent resulted in death. Transitions in this study were defined as transfers to or from an acute hospital, skilled nursing or rehabilitation facility, or home with or without home health care (Coleman et al., 2004b). No measurement system that ignores the integrity and quality of these transitions can be considered complete.

Attending to transitions implies, among other design principles, listening directly to patients’ reports on their own care. Patients and their family caregivers are uniquely positioned to report on their care experiences as they are often the only common thread across disparate health care settings (Coleman et al., 2004a). Therefore, in addition to following patients across multiple settings to assess the care provided instead of focusing on single sites, it is essential to ask patients and their families about their experiences with the care in each setting, the transitions, and overall.

Longitudinal Measures of Outcomes and Costs

Research has documented important differences across providers in the outcomes of care following major surgical procedures (Finlayson and

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

Birkmeyer, 2002; Hannan et al., 1999; O’Connor et al., 1999) and medical hospitalizations (Barnato et al., 2005; Dudley et al., 2000; Shapiro et al., 1999), as well as in the care of those with chronic diseases (Every et al., 2000). In addition, substantial differences in the longitudinal costs of care for similar populations have been documented at both the community and provider levels, with no evidence that greater costs resulted in higher-quality or better outcomes (Baicker and Chandra, 2004; Fisher and Wennberg, 2003; Fisher et al., 2003, 2004; Wennberg, 1999). The committee believes standardized performance measure sets must incorporate the routine monitoring and reporting of long-term health outcomes (mortality and functional status) and costs for selected conditions to promote the attainment of better outcomes at lower cost.

Individual-Patient-Level, Population-Based, and Systems-Level Measurement

The committee proposes several innovative approaches to collecting and reporting performance measures. The key notion is to collect data on each measure at the level of the individual patient and maintain individual-level records to allow the aggregation of measures along three important dimensions: (1) composite measures of the care provided to the individual, documenting, for example, whether a patient received all of his or her recommended preventive services within a specified time window; (2) reporting of measurement results for strata of the population defined on the basis of socioeconomic status, race, and ethnicity; and (3) reporting of measurement results at multiple levels of the care delivery system—physician, physician group, hospital, and community—to identify gaps in performance and foster accountability at each level. These approaches to aggregation are applicable both to the starter set of measures proposed by the committee and to future measure development. Their implementation is dependent upon adequate data collection, reporting, and aggregation, key functions of a performance measurement and reporting system that the NQCB will ensure. The committee strongly believes that data collection protocols should be planned and implemented to support these functions.

Individual-Patient-Level Perspective

A critical concept that emerged from the committee’s deliberations was the use of composite measures. Composites are a relatively new concept in the area of performance measurement, denoting, at minimum, the combining of dichotomous indicators for several specific measures into a single number. The term can also refer, for example, to calculation of a simple mean of rates for several measures, the mean of the fraction of appropriate

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

processes of care received, and the fraction of opportunities to receive all appropriate care for a defined population. The committee has chosen to define composites as the bundling of measures for specific conditions to determine whether all critical aspects of care for a given condition have been achieved for an individual patient, thereby enhancing measurement to extend beyond tracking performance on separate measures.

The committee chose this definition of composite measures for multiple reasons. First, it allows for continuous measurement across providers through aggregation by patient (reinforcing the approach longitudinal measurement) and for an examination of all aspects of required care at the community/population level. On a larger scale, composite measures thus defined can provide a different and potentially deeper view of the reliability of the care system as a whole, encouraging and facilitating systems-level changes by highlighting the need for better care coordination and accountability across multiple providers. They can also serve as a powerful stimulus for the adoption of electronic health records to ensure that patients receive recommended care. The committee believes patients could play a more active role in their care if they were armed with evidence-based information on the complete set of clinical services they should expect and ultimately demand. In addition, the use of composite measures does not require the large sample sizes needed for some other approaches. Thus, the committee proposes that this approach to measurement be taken in addition to measuring performance on discrete indicators.

The committee believes this concept represents a turning point, and a relatively new challenge, for performance measurement. Composite scores centered on individual patients could be calculated for many preventive, acute, and chronic care services, with careful consideration for age and gender appropriateness. The use of composite measures suggests performance goals considerably more stringent than those captured by the usual single-variable measures. Using composites in this manner allows for a patient-centered approach that takes into account the full constellation of health care needs (McGlynn et al., 2003a).

The technical challenges to the construction of accurate, valid, and reliable composite measures and their elements for all conditions are substantial. Among these challenges are the following:

  • The rate automatically tends to go down as more process measures are added to the composite, since it is more difficult to provide all the required measures of a large set than of a small one.

  • Improvement will not be fully reflected by composites if several processes are measured, some of which are received at a high rate and others at a generally low rate.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
  • The composite score will be lower if different people receive each of the various processes than if some people receive all and most receive none, for a given rate on each process.

Thus, although the committee’s approach to composites readily identifies poor performance, it is not necessarily appropriate for making comparisons or solely summarizing improvement.

The efforts of HealthPartners Inc., the Centers for Medicare and Medicaid Services (CMS), the RAND Corporation, the Foundation for Accountability, and other organizations that have explored this approach, as well as a simple inspection of the scientifically grounded array of current measures, can serve as a good starting point for the development of an initial set of composite measures. One example of how the proposed approach to composites can be implemented is described in Box 4-2.

Despite these efforts, further research is required before this concept of composites can be fully developed and expanded upon. An important issue to be addressed is whether the various components of a composite should be weighted differently, such as according to their level of clinical importance. On the whole, the committee favors a simple yet integrated approach whereby composite measures in the first instance would be “all or none”

BOX 4-2
Example of the Implementation of Composite Measures

A pioneering organization in composite measurement, HealthPartners Inc., a health plan in Minnesota, has been collecting and publicly reporting composite scores for diabetes, coronary artery disease, and preventive care. HealthPartners calculates a composite score for its diabetic population by examining the percentage of its members with Type I and Type 2 diabetes aged 18 through 75 who are optimally managed, not just for each but for all of the following factors: HbA1c ≤ 8 percent mg/dl; LDL cholesterol ≤ 130 mg/dl; blood pressure <130/85 mmHg; aspirin use for members >40 years old; and documentation of nonuse of tobacco. A single rate is then reported, indicating the percentage of eligible members who achieved this complete bundle of intermediate outcomes. In the 2004 reporting period, although each separate clinical variable showed performance in the range of 45.5 percent (BP ≤130/85) to 82.8 percent (not smoking), the composite score revealed that only 18.4 percent of eligible patients were receiving the complete set of needed interventions (Personal communication, G. Amundson, HealthPartners, December 2004; Amundson et al., 2005).

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

measures. Accordingly, a composite measure would be designated as “1” only if all the required services or procedures had been performed or all outcomes reached and as “0” if at least one of those services or procedures had not been performed or all outcomes reached; thus weighting would not be an issue. The rationale for this proposal is a recognition that if any of the services required for taking care of an individual (or a population) is absent, care is suboptimal. However, this notion should ultimately be tempered so as to be applied in addition to the proportion of criteria met, as the provider’s cost of improvement has many implications for how a weighting system should be structured.

Population-Based Perspective

The Institute of Medicine’s (IOM) definition of quality includes population health: “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge” (IOM, 1990:4). The term “population health” is widely used and often understood to be a product of multiple determinants of health—genetic endowment and physical environment and social environment. The personal health care delivery system, which focuses on the care of individuals, represents an important but limited element of population health, whereas the public health system takes a broader and more inclusive perspective on these determinants. The focus of this report is on the contribution that the personal health care delivery system can make to improving population health—a measure that encompasses not only health but also its distribution among the population. The report does not attempt to speak to the full range of measures one would want in a population health monitoring and reporting system, such as environmental measures, as this would be beyond the committee’s charge. The committee does, however, address how the personal health care delivery system can contribute to the public health system in the domains of health promotion, disease prevention, and clinical preventive services.

For example, the nation’s public health goals, as articulated by Healthy People 2010, include many areas of overlap between the personal health care delivery system and the public health system, such as preventive screenings, immunizations, and tobacco cessation counseling (U.S. DHHS, 2000). Another example of a measure set that intersects both of these systems is the Prevention Quality Indicators developed by the Agency for Healthcare Quality and Research (AHRQ), based on hospital inpatient data that reveal how well care is being delivered by identifying such events as avoidable hospitalizations. The committee recognizes the substantial need to bridge the gap between public and private health care systems and to promote core performance measures that can foster collaborative efforts. Communica-

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

tion among stakeholders in these systems is critical to enhancing performance measurement and achieving the ultimate goal of better health.

The committee calls for a move toward an important method for narrowing the divide between the personal health care and public health systems: the more comprehensive system of individual-patient-level measures, drawn from the population of a community and aggregated by different levels of care providers (individual physician, group practice, hospital, or nursing home), geographic regions (community, state, or national), and demographic groupings (race and ethnicity, socioeconomic status, class, age, sex) when appropriate. This more clinical perspective on health care services is distinct from measuring the determinants of the health of populations and non–health care services.

This approach to gathering performance data allows information to be collected across multiple sites instead of in a site-specific manner, and to be used for multiple purposes, such as internal quality improvement, accountability, and public reporting at the provider, community, and national levels. Moreover, this approach supports an important shift in focus from the care delivered by some part of the health care system (a health practitioner, for example) to the care needs met by the overall system. This change in measurement strategy will support analyses of the extent to which all patients are receiving the right care at the right time for their specific individual needs.

Unfortunately, this concept of measuring care across the continuum of time and space conflicts with how care is currently organized and financed, as the individual patient is usually the only consistent factor across settings of care (and noncare). The way data are managed today tends to render the care continuum opaque, not transparent. Data are often exclusively stored and “owned” by specific care settings and providers, not by patients themselves. This problem of ownership and control compounds the difficulty of sharing and analyzing data across settings in a timely fashion, with or without electronic health records (IOM, 2004; Walker et al., 2005). The committee anticipates that a commitment to a population-based approach to data collection and management will generate the scaling and data management requirements that will eventually drive the use of data warehousing, information technology, and other data management capabilities and strategies, and accelerate the universal adoption of electronic health records as an American standard.

An evolving population-based perspective also is facilitated by patient-level data warehouses that can provide opportunities for testing emerging hypotheses, such as examining the effects of interventions designed for patients with coincident diabetes, heart failure, and depression. The current absence of patient-level data that can be aggregated for populations of evolving interest requires researchers to build a cohort, follow a sample of pa-

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

tients with a designated condition or set of conditions, and respecify and recollect data each time a new hypothesis is formulated. Patient-level data, capable of flexible and varied aggregation to reflect populations of interest, form an “epidemiologic utility” that could be used for knowledge development (Halvorson and Isham, 2003; Wallace, 2005).

A population perspective also addresses the quality aim of equity and the related issue of access by assessing the delivery of the right treatment to the right person at the right time for everyone who would potentially benefit. The italicized phrase distinguishes the proposed approach to measuring health care delivery from other perspectives, drawing attention to issues of equity and the existence and impact of disparities among groups. An adequate performance measurement system should illuminate the status of people who do not receive care as much as that of people who do. Especially important from a population perspective are performance measures that target improvement in the health status of different ethnic, racial, and class groups. The heterogeneity of any community offers a strong incentive for organizing the data on its constituents at the patient level. Yet these patient-level data can then be aggregated to any level of granularity, from individual patient reports to the entire community, making it possible, for example, to measure both over- and underuse of interventions within whole populations. Examples of waste and neglect can be obscured if granularity to the patient level is not obtained. Arguably, especially within this perspective, missing or inaccessible data to support the delivery of needed services within a population are defects in quality of care.

Systems-Level Perspective

Few Americans receive their health care from fully integrated delivery systems. Nonetheless, patients and communities often depend upon systems-level performance that requires effective interactions among discrete caregivers and institutions and across time, regardless of whether those providers are in formal, intentional relationships with each other. The committee believes a complete set of performance measures must encompass this type of systems-level performance. As discussed above, measures obtained at the individual patient level could be aggregated to different levels of the system, including physician groups, hospitals, the continuum of care, or communitywide care delivery systems.

At the hospital level, systems-level performance measures could be applied to the hospital as a whole, with its executive and clinical leadership and governing board presumably being responsible for improvement on these measures. At the level of the continuum of care, an accountable entity could be difficult to identify absent an accountable integrated delivery system with responsibility for the care of a defined population over space and

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

time. Nonetheless, patients and families should and do care about outcomes and processes at this level. The committee therefore proposes that such measures be developed, used, and reported to drive shared accountability throughout the health care system. In addition, participation is required from policy makers at all levels if the health care delivery system is to improve. Adoption of systems-level measures should help American communities become more aware of their met and unmet health care needs, and over time could induce innovations and relationships among care providers that could lead to better performance.

The ultimate measures of the performance of American health care would assess the nation’s effectiveness in meeting the needs of communities. Few American communities organize their health care as a communitywide system. By measuring and tracking systems-level performance at the community level, however, it may ultimately be possible to assess the national consequences of policy and financing environments as a whole. For assessment of performance at the level of the community as a whole, federal agencies such as the Centers for Disease Control and Prevention can offer guidance. In addition, the state-level reports produced by AHRQ in the context of the National Healthcare Quality and Disparities Reports are good first steps in community-level performance measurement (AHRQ, 2003).

A commitment to systems-level performance measurement will require both scientific innovation and new loci of responsibility for measurement itself, as well as the taking of action in accordance with measurement results. Below the committee proposes a uniform set of hospitalwide measures and measures across the continuum of chronic disease care as the starting point for this effort.

Shared Accountability

Improved performance on many of the measures proposed by the committee can be achieved only through the collaborative efforts of multiple providers and multiple care settings. The committee believes the NQCB should include and report on measures—such as care transitions and longitudinal outcomes and costs—that reflect the performance of multiple providers who should, ideally, collaborate to improve the quality of care. As discussed earlier, the committee also believes that measures should be diffused to different levels of the delivery system, including the community. For example, performance measures for racial, ethnic, or socioeconomic groups (such as the uninsured) should be collected and reported at multiple levels. Analysis of these data would force discussion of the underlying reasons for disparities and the opportunities available to multiple stakeholders for addressing these issues. This notion of shared accountability will have substantial impacts on payment-based incentive policies and will be further

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

addressed by the payment incentives report in this Pathways series (see Chapter 1).

In short, the committee concludes that measurement of the health care delivery system should not be impeded by the impossibility of first identifying an accountable actor or the perception that responsibility for care is outside one’s realm of control. Indeed, one valuable and intended effect of the integrated measurement system proposed by the committee could be to induce new parties to assume such responsibility. This position represents a significant break from commonly accepted criteria for performance measurement.

SELECTION OF SPECIFIC PERFORMANCE MEASURES

To this point, the discussion has focused primarily on insights that emerged from the committee’s review of currently available performance measures and an analysis of the quality of these measures against the goals and aims of health care measurement. The committee’s primary charge was to recommend a subset of measures—derived from leading performance measure sets—that could be used to align performance with payment under the Medicare program. The committee addressed this task within a more general framework designed to move the U.S. health care system toward the overarching goals discussed earlier. Its ultimate objective was the creation of a measure set that would be consistent both with the goals and aims for health care improvement set forth earlier in this chapter and with the 10 design principles for performance measurement articulated in Chapter 2. The resulting measures encompass what we need to know about health care quality to guide future payment policies and practices.

Criteria for Selection

In addition to the 10 design principles articulated in Chapter 2, the committee identified criteria to guide the selection of specific measures. The criteria in Box 4-3 apply to individual characteristics of either a specific measure (e.g., validity and reliability) or the collective measure set (e.g., comprehensiveness). Other groups have articulated these criteria: measures should be scientifically sound, feasible, important, aligned with other leading measure sets, and comprehensive. However, it is important to point out the absence of one criterion often used by other groups: that a measure be within the control of an identifiable actor. As discussed above, the committee takes the position that improvement across many important domains of care will require action by multiple parties—including patients, providers, and other stakeholders (such as health plans, payers, and public health agencies), and the committee therefore endorses public reporting on measures, such as longitudinal care, that foster shared accountability.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

BOX 4-3
Criteria for Measure Selection Considered by the Committee and Other Selected Groups

  • Scientifically Sound: This criterion concerns the reliability, validity, and explicitness of the evidence base. Reliability means a measure consistently produces the same result when repeated within the same population and setting. Validity addresses the question of whether a measure reflects what it is intended to measure. Finally, the evidence base from which a measure is derived must be explicit—for example, randomized controlled trials, case control studies, observational studies, or formal consensus processes.

  • Feasibility: To assess feasibility, the data needed for a measure must be in current use, available across the system, and examined for the cost or burden of measurement on providers.

  • Importance: The health problem addressed by a measure should be a leading cause of death or disability or associated with high resource use. A measure must have an impact on health, be tied to national goals, and be susceptible to being influenced by the health care delivery system. Ideally, a measure should be stratified by race, gender, and age.

  • Alignment: Optimally, measures should be selected from existing leading measure sets that are calculated with the same technical specifications for both the numerator and denominator to reduce redundancy and the burden of reporting.

  • Comprehensiveness: Measures selected should be part of a set to reflect quality in a particular area of care or bundled services of necessary care for a given condition. Each measure in the set should meet the criterion of importance to warrant inclusion. To demonstrate comprehensiveness, the set of measures must address the way the care is delivered and the nature of the quality problem involved—underuse, misuse, or overuse.

Note that, as discussed in the text, the committee did not support the criterion that only measures under the control of a specific system of care should be used.

SOURCE: AHRQ, 2001; CMS, 2004a; McGlynn et al., 2003a,b; MedPAC, 2005a; NCQA, 2001.

Methodological Limitations of Existing Measures

The committee recognized that many current measures, while meeting the above criteria, have methodological limitations that may reduce their applicability or utility in certain settings. These limitations include a degree of statistical variability for some measures that may constrain the ability to

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

characterize the performance of individual physicians or small practices, a need for case-mix or risk adjustment in some instances that cannot be met by currently collected data, and requirements for data collection that may impose a substantial burden on providers in the absence of registries or computerized health information systems (Birkmeyer, 2004; Birkmeyer et al., 2004; Hofer et al., 1999; Landon et al., 2003). (A more detailed overview of the methodological limitations of existing structure, process, and outcome measures is provided in Appendix F.) Experience has shown that starting with less-than-perfect publicly reported measures can stimulate the development of improved measures, as illustrated by care safety measurement efforts and by the development of the National Committee for Quality Assurance’s Health Plan Employer Data and Information Set (HEDIS) measures of the performance of health maintenance organizations (Personal Communication, Arnold Milstein, October 11, 2004). Thus the committee is confident that aggressive implementation of existing measures would both improve those measures and, assuming that the measures led to action, enhance the quality of care.

Recommended Measures for Implementation

The committee’s analysis of existing performance measure sets revealed many measures consistent with one or more of the articulated measurement goals that also meet the criteria shown in Box 4-3. For some of the six quality aims, such as efficiency, equity, and patient-centeredness, however, the committee was unable to identify standardized performance measures already in widespread use. Many measures have been used in research settings or are at various stages of pilot testing and development for use in standardized performance measurement. Therefore, the committee recognized that the creation, promulgation, and reporting of new measures need to be included in a research agenda to achieve the goals of performance measurement set forth above.

The committee recommends the immediate implementation of the starter set of measures derived from leading measure sets shown in Table 4-2 and discussed in detail in the next section. To this end, a data repository system will need to be in place, along with a mechanism for public reporting. In addition, the NQCB will need to identify a strategy for data aggregation. There are two particularly thorny issues to be addressed:

  • National versus local/regional data repositories—Data could be submitted to a national repository and then transmitted to the local/regional level for reporting purposes. Another alternative is to create local/regional repositories that would transmit data to the national level. Each strategy has implications for data confidentiality and security, operational

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

TABLE 4-2 Recommended Starter Set of Performance Measures

Ambulatory Care

Ambulatory care Quality Alliance (26)

Prevention measuresa (7), coronary artery diseasea (3), heart failurea (2), diabetesa (6), asthmaa (2), depressiona (2), prenatal carea (2), quality measures addressing overuse or misuse (2)

 

Ambulatory Care Survey

CAHPS Clinician and Group Survey: getting care quickly, getting needed care, how well providers communicate, health promotion and education, shared decision making, knowledge of medical history, how well office staff communicate

Acute Care

Hospital Quality Alliance (20)

Acute coronary syndromea (7), heart failurea (3), pneu-moniaa (5), smoking cessationa (3), surgical infection preventiona (from the Surgical Care Improvement Project) (2)

 

Structural Measures (computerized provider order entry, intensive care unit intensivists, evidence-based hospital referrals)

 

Hospital CAHPS

Patient communication with physicians, patient communication with nurses, responsiveness of hospital staff, cleanliness/noise level of physical environment, pain control, communications about medicines, discharge information

Health Plans and Accountable Health Organizations

Health Plan Employer Data and Information Set (HEDIS) (61)

Integrated delivery systems (health maintenance organizations): effectiveness (26), access/availability of care (8), satisfaction with the experience of care (4), health plan stability (2), use of service (15), cost of care, informed health care choices, health plan descriptive information (6)

Preferred provider organizations within Medicare Advantage: selected administrative data and hybrid measures

 

Ambulatory Care Survey

CAHPS Health Plan Survey: getting care quickly, getting needed care, how well providers communicate, health plan paperwork, health plan customer service

Long-Term Care

Minimum Data Set (15)

Long-term care (12), short-stay care (3)

 

Outcome and Assessment Information Set (11)

Ambulation/locomotion (1), transferring (1), toileting (1), pain (1), bathing (2), management of oral medications (1), acute care hospitalization (1), emergent care (1), confusion (1)

End-Stage Renal Disease

National Healthcare Quality Report (5)

Transplant registry and results (2), dialysis effectiveness (2), mortality (1)

Longitudinal Measures of Outcomes and Efficiency

1-year mortality, resource use, and functional status (SF-12) after acute myocardial infarction

aThe committee recommends the aggregation of individual measures to patient-level composites for these areas.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

costs and complexity, ongoing innovation and local access/acceptability, and locus of management. The committee does not endorse one strategy over the other, as these issues require further deliberation by the NQCB. The Ambulatory care Quality Alliance (AQA) is currently developing a model that includes a framework and governing structure for aggregating, sharing, and stewarding data that could provide guidance in this area (AHRQ, 2005).

  • Comprehensive scope—Data repositories that included data for all patients (i.e., privately insured patients, Medicare and Medicaid beneficiaries and other publicly insured patients, and the uninsured) would provide a more complete picture of an individual provider’s practice if the provider cared for multiple populations. Comprehensive repositories would also provide better population-level information. But legal, regulatory, ownership, and operational issues must be addressed if such repositories are to be established.

Additionally, as learned from the committee’s case studies, technical and financial assistance to providers will require greater attention. Providers of all types will need assistance in implementing quality improvement strategies in addition to data collection and reporting. These and many other issues will require careful assessment before the NQCB can move forward.

Recommendation 4: The NQCB should promulgate measure sets that build on the work of key public- and private-sector organizations. Specifically, the NQCB should:

  • As a starting point, endorse as national standards performance measures currently approved through ongoing consensus processes led by major stakeholder groups.

  • Ensure that a data repository system1 and public reporting program capable of data collection at the individual patient level are established and open to participation by all payers and providers.

  • Ensure that technical and financial assistance is available to all providers who need help in establishing performance measurement and improvement capabilities.

The following discussion details the starter set of performance measures proposed by the committee. This starter set of measures represents what can be done now to move toward a national system for performance measurement and reporting.

1  

The data repository system would collect, validate, and aggregate provider performance data (see Recommendation 1).

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

RECOMMENDED STARTER SET OF PERFORMANCE MEASURES

The committee recommends the leveraging of existing efforts, but stands firm that an immediate gearing up of resources must occur to address the shortcomings in current approaches to performance measurement discussed in this chapter.

Starter Set Measures for Ambulatory Care Performance

To accelerate performance measurement in the ambulatory care setting, the committee proposes the immediate adoption of the 26 clinical performance measures recently selected by AQA. The individual measures in this set, detailed in Appendix G-1, cover four domains of care in which quality problems are well documented and continue to persist:

  • Preventive care—cancer screening, vaccinations, and tobacco use/counseling

  • Chronic care—coronary artery disease, heart failure, diabetes, asthma, and depression

  • Prenatal care—HIV screening and administration of anti-D (Rh) immune globulin

  • Efficiency of care—appropriate prescribing of antibiotics to children

The committee proposes that patient-level composite scores, as previously described, be collected and reported for measures of asthma, coronary artery disease, depression, diabetes, heart failure, and prenatal care, as well as a preventive care composite consisting of age- and sex- appropriate services.

Preventive Care

The committee devoted considerable attention to preventive care, with the rationale that these services, such as earlier diagnoses for common cancers, would yield benefits in the long run in terms of both improved quality of life and in some cases potentially lower costs. Measures of preventive care provide an opportunity to highlight issues associated with both effective and equitable care. Disparities among racial and ethnic groups in cancer-related deaths and survival rates are well documented. For example, death rates for breast cancer are higher among African Americans than among whites—36 per 100,000 versus 27 per 100,000.2 In addition, 5-year survival rates for breast cancer (74 percent) among African Americans are

2  

Age-adjusted to the 2000 U.S. standard population.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

lower than those among whites (88 percent) (American Cancer Society, 2004; IOM, 2003).

Assessing adults for tobacco use and providing tobacco cessation counseling ranks second among the top 30 clinical preventive services recommended by the U.S. Preventive Services Task Force based on the criteria of clinically preventable burden and cost-effectiveness.3 However, counseling services have one of the lowest national delivery rates (less than 50 percent) (Coffield et al., 2001). More than 440,000 tobacco-related deaths occur annually as a result of cardiovascular diseases, cancers, respiratory diseases, and perinatal conditions. Accordingly, the committee endorses the rapid uptake of these measures (Centers for Disease Control and Prevention, 2002).

Chronic Care

Serious quality problems, particularly underuse of services, have been documented for all of the chronic conditions in the AQA set. For those chronic conditions—diabetes, asthma, depression, heart failure, and coronary artery disease—a national study found that Americans receive only 45–68 percent of recommended care (McGlynn et al., 2003a). Recent data on elderly Medicare beneficiaries also demonstrate serious quality problems. For example, only one-third of the elderly received effective treatment for depression, while only one-quarter of elderly diabetics received an annual dilated eye exam, a recommended screening test for retinopathy (Leatherman and McCarthy, 2005). Since the number of individuals with chronic conditions continues to grow (an estimated 133 million Americans in 2005, expected to rise to 157 million in 2020), and 78 percent of all health care spending in all care settings is attributable to these conditions, performance measurement in this area becomes a top priority (Partnership for Solutions, 2002).

Prenatal Care

In 2002, approximately 4 percent of pregnant women either did not receive prenatal care until the third trimester or received no such care at all (National Center for Health Statistics, 2004). Inadequate prenatal care can lead to infant mortality, as well as complications during pregnancy and childbirth. The United States ranked twenty-second in infant mortality among Organisation for Economic Cooperation and Development countries

3  

Clinically preventable burden is defined as the proportion of disease and injury prevented by the clinical preventive service in usual practice if the service were delivered to 100 percent of the target population at recommended intervals. Cost-effectiveness is defined as net cost per quality-adjusted life-years saved.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

in 2003, with a rate of 7.0 deaths per 1,000 live births (Organisation for Economic Co-operation and Development, 2005), as compared with Iceland, ranked first with a rate of 2.4 deaths per 1,000 live births. The cost burden over a lifetime for a child born with birth defects is estimated to be $8 billion (U.S. Preventive Services Task Force, 1996). These problems are the outcome of many factors; however, they can begin to be alleviated through better prenatal care.

Two preventive services for prenatal care are included in the AQA measure set and are supported by the United States Preventive Services Task Force: anti-D (Rh) immune globulin and HIV screening. Providing anti-D (Rh) immune globulin to women who are Rh negative promotes prevention of life-threatening outcomes, such as newborn hemolytic disease due to maternal sensitization. HIV screening significantly lowers rates of mother-to-child HIV transmission, an important benefit as between 280 and 370 newborns are diagnosed with HIV in the United States each year (Bulterys et al., 2002). Thus the committee proposes the inclusion of both of these measures in the starter set by the NQCB.

Efficiency of Care

Overuse and misuse of resources are results of poor-quality care. This issue is addressed in the AQA set through measurement of appropriate treatment of viral infections leading to upper respiratory conditions and pharyngitis. Antibiotics are effective only in treating bacterial infections. Therefore, the use of antibiotics for respiratory infections that are viral in nature is not efficacious and leads to the negative consequence of increased microbial antibiotic resistance. While the trend in prescribing antibiotics for children has been declining (from a rate of 838 per 1,000 in 1989 to 503 per 1,000 in 1999), the practice remains unacceptably common (McCaig et al., 2002). To counter these trends in overuse and misuse, the committee proposes inclusion of these measures in the starter set.

Ambulatory Care Surveys

Although the AQA measures are a reasonable starting point for performance measurement in the ambulatory care setting, assessment in this area would be incomplete without a component of patient feedback. To complement the above clinical measures, the committee proposes implementing surveys of ambulatory care in conjunction with those measures upon completion of field testing.4 The CAHPS program has developed two prod-

4  

See Chapter 2 for an overview of the CAHPS family of surveys.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

BOX 4-4
Core Domains of Ambulatory Care Surveys

CAHPS Health Plan Survey

CAHPS Clinician and Group Survey

Getting care quickly

Getting needed care

How well providers communicate

Health plan paperwork

Health plan customer service

Getting care quickly

Getting needed care

How well providers communicate

Health promotion and education

Shared decision making

Knowledge of medical history

How well office staff communicate

ucts specific to ambulatory care: the CAHPS Health Plan Survey and the CAHPS Clinician and Group Survey (AHRQ, 2004). Core domains in each of these surveys are presented in Box 4-4.

Starter Set Measures for Acute Care Performance

Of the $1.5 trillion spent on health care in 2002, 33 percent is attributable to hospital care (American Hospital Association, 2004). Widespread performance measurement in hospitals has built upon past efforts involving collaboration among a multitude of stakeholders, as discussed in Chapter 2. These efforts have culminated in the measures chosen by the Hospital Quality Alliance (HQA), a partnership of 13 public and private sponsors (see Appendix G-2). Measures were selected on the basis of severity of clinical condition and ease of data submission for public reporting. The 20 measures endorsed by the National Quality Forum (NQF) originate from the voluntary starter set of 10 measures that, under the Medicare Prescription Drug, Improvement and Modernization Act of 2003, are linked to a 0.4 percent reduction in Medicare annual payment update if not reported. Currently, an estimated 4,200 hospitals are participating in this public reporting effort. As with the AQA measures, the committee proposes the reporting of these individual measures as patient-level composites for the following areas: acute coronary infarction, heart failure, pneumonia, smoking cessation, and surgical complications.

Structural Measures

In an effort to address patient safety in the hospital setting, the committee proposes assessment of the following structural measures: (1) implementation of computerized provider order entry for prescriptions, (2) staff-

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

ing of intensive care units with intensivists, and (3) evidence-based hospital referrals. These measures originate from the Leapfrog Group’s original “three leaps,” which have been widely implemented and are part of the NQF’s 30 safe practices (NQF, 2003).

Hospital CAHPS

Hospital CAHPS is currently slated for inclusion in the HQA measure set by 2007. The domains of measurement for this survey are listed in Box 4-5. The committee strongly supports the expedient collection and reporting of these patient-centered measures.

Starter Set Measures for Health Plan Performance

Health plans have a long and credible history of collecting and reporting performance measures, beginning with the adoption of HEDIS measures. The 2005 HEDIS measure set includes 61 measures that are recommended for the starter set. With respect to Medicare, specifically Medicare Advantage, however, the data collected and reported by preferred provider organization (PPO) plans and health maintenance organization plans will need to be reconciled. Currently, PPOs are required initially to report only those HEDIS measures for which administrative data can be used; nonetheless, additional infrastructure necessary to collect data for measures requiring chart abstraction must be in place and fully functional by 2008 to enable reporting of the full set of HEDIS measures by PPOs (see Appendix G-3) (CMS, 2004a).

Starter Set Measures for Long-Term Care Performance

To receive payment, nursing homes and home health settings are required by CMS to collect data on long-term care measures routinely using the Minimum Data Set (MDS) and the Outcome and Assessment Instru-

BOX 4-5
Hospital CAHPS Domains

Patient communication with physicians

Patient communication with nurses

Responsiveness of hospital staff

Cleanliness/noise level of physical environment

Pain control

Communications about medicines

Discharge information

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

ment Set (OASIS), respectively (CMS, 2004b, 2005a). The MDS, collected since 1990, evaluates such areas as cognitive/behavior patterns, quality of life, functional status, and pain. OASIS, implemented in 2000, assesses outcomes for home care patients, with the data intended for use in quality improvement efforts, including evaluations of sociodemographics, environment, support systems, health status, functional status, and health service utilization.

As these measures are already being collected by providers, the committee proposes that the long-term care measures being publicly reported by CMS5 in both the MDS and OASIS data sets, as listed in Appendix G-4, initially be used for describing performance in these settings. However, the committee recognizes that measures in the MDS need additional research and development before they can be linked to pay for performance mechanisms (MedPAC, 2005b).

Starter Set Measures for End-Stage Renal Disease Performance

End-stage renal disease (ESRD) affected more than 430,000 people in 2002 at a cost of $17 billion to Medicare (U.S. Renal Data System, 2004). Under the Social Security Act, ESRD patients are eligible to obtain all Medicare benefits, including dialysis and renal transplant. As a result, almost all ESRD patients are covered under Medicare, with only a small percentage paying out of pocket or through private insurers. Data have been collected on this special population since 1988 through a partnership among the National Institutes of Health, CMS, and the United States Renal Data System, which documents the incidence and prevalence of ESRD and its patients, and identifies and furthers the research agenda associated with this disease. Five of these measures6—targeting transplant registries and overall dialysis effectiveness—are being collected in AHRQ’s National Healthcare Quality Report (see Appendix G-5).

Starter Set Measures for Longitudinal Measurement of Outcome and Efficiency Performance

The committee believes the above starter set measures are sector-specific and thus have several serious shortcomings: they do little to foster improved

5  

CMS has developed the Web-based resources Nursing Home Compare and Home Health Compare, which publicly report selected quality indictors to help consumers make informed choices when selecting a nursing home or home health agency, respectively; see Appendix G-4 for measures in both of these sets (CMS, 2005b).

6  

The three outcome measures are derived from the University of Michigan; the two process measures are from the United States Renal Data System.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

coordination across all care settings; they provide virtually no information on the costs of care, especially for a population over time; and they offer very limited measures of the outcomes of care. To begin to address these shortcomings, the committee proposes longitudinal measures of outcomes and costs, starting with 1-year mortality, resource use, and functional status measures for acute myocardial infarction.

CLOSING COMMENTS

The next step in enhancing current performance measurement and reporting capabilities is to address the gaps identified in this chapter through a research agenda. Chapter 5 provides a strategy for the development of a research agenda to support the aggressive development of the resolution of underlying methodological issues, the improvement of public reporting methods, and the evaluation of the overall progress of the national system for performance measurement and reporting proposed in this report.

REFERENCES

AHRQ (Agency for Healthcare Research and Quality). 2001. Approach to Performance Measurement. [Online]. Available: http://www.ahrq.gov/about/gpra2001/gpra01app1.htm [accessed September 1, 2004].

AHRQ. 2003. National Healthcare Disparities Report. Rockville, MD: AHRQ.

AHRQ. 2004. CAHPS. [Online]. Available: http://www.ahrq.gov/qual/cahpsix.htm [accessed March 3, 2005].

AHRQ. 2005. Ambulatory Care Quality Alliance Third Invitational Meeting Summary (continued). [Online]. Available: http://www.ahrq.gov/qual/performance3/perfm3d.htm [accessed October 20, 2005].

American Cancer Society. 2004. Cancer Facts and Figures 2004. Atlanta, GA: American Cancer Society.

American Hospital Association. 2004. Trends Affecting Hospitals and Health Systems. Washington, DC: American Hospital Association.

Amundson GM, Gentilli S, Wehrle D. 2005. HealthPartners 2005 Clinical Indicators Report . 13th ed. Bloomington, MN: HealthPartners, Inc. [Online]. Available: http://www.healthpartners.com/files/23463.pdf [accessed December 6, 2005].


Baicker K, Chandra A. 2004. Medicare spending, the physician workforce, and beneficiaries’ quality of care. Health Affairs. Jan-June Suppl Web Exclusive: W4:184–197.

Barnato AE, Lucas FL, Staiger DP, Wennberg DE, Chandra AP. 2005. Hospital-level racial disparities in acute myocardial infarction treatment and outcomes. Medical Care 43(4):308–319.

Birkmeyer J. 2004. Does Risk Adjustment Always Matter? Presentation to the IOM Committee on Payment, Performance Measures, and Performance Improvement. December 1–3, 2004. Washington, DC.

Birkmeyer JD, Dimick JB, Birkmeyer NJO. 2004. Measuring the quality of surgical care: Structure, process, or outcomes? Journal of the American College of Surgeons 198(4):626–632.

Bulterys M, Nolan ML, Jamieson DJ, Dominguez K, Fowler MG. 2002. Advances in the prevention of mother-to-child HIV-1 transmission: Current issues, future challenges. AIDScience 2(4):1–10.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

Centers for Disease Control and Prevention. 2002. Annual Smoking-Attributable Mortality, Years of Potential Life Lost, and Economic Costs—United States, 1995–1999. Morbidity and Mortality Weekly Report 51:300–303.

CMS (Centers for Medicare and Medicaid Services). 2004a. Medicare Advantage PPO HEDIS Measurement Feasibility Assessment Report. Washington DC: CMS.

CMS. 2004b. OASIS Overview. [Online]. Available: http://www.cms.hhs.gov/oasis/hhoview.asp [accessed February 25, 2005].

CMS. 2005a. Minimum Data Set (MDS) 3.0. [Online]. Available: http://www.cms.hhs.gov/quality/mds30/ [accessed March 2005].

CMS. 2005b. Medicare Compare Series: Nursing Home Compare, Home Health Compare, Dialysis Compare, and Hospital Compare. [Online]. Available: http://www.Medicare.gov [accessed January 24, 2005].

Coffield AB, Maciosek MV, McGinnis JM, Harris JR, Caldwell MB, Teutsch SM, Atkins D, Richland JH, Haddix A. 2001. Priorities among recommended clinical preventive services. American Journal of Preventive Medicine 21(1):1–9.

Coleman EA, Boult C, American Geriatrics Society Health Care Systems Committee. 2003. Improving the quality of transitional care for persons with complex care needs. Journal of the American Geriatrics Society 51(4):556–557.

Coleman EA, Mahoney E, Parry C. 2004a. Assessing the quality of preparation for post-hospital care from the patient’s perspective: The Care Transitions Measures (CTM). Medical Care 43(3):246–255.

Coleman EA, Min S, Chomiak A, Kramer AM. 2004b. Post-hospital care transitions: Patterns, complications, and risk identification. Health Services Research 39:1449–1465.


Dudley RA, Johansen KL, Brand R, Rennie DJ, Milstein A. 2000. Selective referral to high-volume hospitals: Estimating potentially avoidable deaths. Journal of the American Medical Association 283(9):1159–1166.


Every NR, Fihn SD, Sales AE, Keane A, Ritchie JR. 2000. Quality enhancement research initiative in ischemic heart disease: A quality initiative from the Department of Veterans Affairs. QUERI IHD Executive Committee. Medical Care 38 (6 Suppl 1):I-49–I-59.


FACCT (Foundation for Accountability). 1997. The FACCT Consumer Information Framework: Comparative Information for Better Health Care Decisions. [Online]. Available: http://www.facct.org/information.html [accessed June 4, 2002].

Finlayson EV, Birkmeyer JD. 2002. Should consumers trust hospital quality report cards? Journal of the American Medical Association 287(24):3206; author reply 3207–3208.

Fisher ES, Wennberg JE. 2003. Health care quality, geographic variations, and the challenge of supply-sensitive care. Perspectives in Biology and Medicine 46(1):69–79.

Fisher ES, Wennberg DE, Stukel TA, Gottlieb DJ, Lucas FL, Pinder EL. 2003. The implications of regional variations in Medicare spending. Part 1: The content, quality, and accessibility of care. Annals of Internal Medicine 138(4):273–287.

Fisher ES, Wennberg DE, Stukel TA, Gottlieb DJ. 2004. Variations in the longitudinal efficiency of academic medical centers. Health Affairs Suppl Web Exclusive: VAR19-32.


Halvorson, G, Isham G. 2003. Epidemic of Care: A Call for Safer, Better, and More Accountable Health Care. San Francisco, CA: Jossey-Bass.

Hannan EL, van Ryn M, Burke J, Stone D, Kumar D, Arani D, Pierce W, Rafii S, Sanborn TA, Sharma S, Slater J, DeBuono BA. 1999. Access to coronary artery bypass surgery by race/ethnicity and gender among patients who are appropriate for surgery. Medical Care 37(1):68–77.

Hofer TP, Hayward RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG. 1999. The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease. Journal of the American Medical Association 281(22):2098–2105.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

IOM (Institute of Medicine). 1990. Medicare: A Strategy for Quality Assurance (Volume 1). Washington, DC: National Academy Press.

IOM. 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press.

IOM. 2003. Priority Areas for National Action: Transforming Health Care Quality. Adams K, Corrigan JM, eds. Washington, DC: The National Academies Press.

IOM. 2004. 1st Annual Crossing the Quality Chasm Summit: A Focus on Communities. Adams K, Greiner A, Corrigan J, eds. Washington, DC: The National Academies Press.


Landon BE, Normand S-LT, Blumenthal D, Daley J. 2003. Physician clinical performance assessment: Prospects and barriers. Journal of the American Medical Association 290(9):1183–1189.

Leatherman S, McCarthy D. 2005. Quality of Health Care for Medicare Beneficiaries: A Chartbook. New York: Commonwealth Fund.


McCaig LF, Besser RE, Hughes JM. 2002. Trends in antimicrobial prescribing rates for children and adolescents. Journal of the American Medical Association 287(23):3096–3102.

McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA. 2003a. The quality of health care delivered to adults in the United States. New England Journal of Medicine 348(26):2635–2645.

McGlynn EA, Cassel CK, Leatherman ST, DeCristofaro A, Smits HL. 2003b. Establishing national goals for quality improvement. Medical Care 41(Suppl. 1):I-16–I-29.

MedPAC (Medicare Payment Advisory Commission). 2005a. Report to the Congress: Medicare Payment Policy. Washington, DC: MedPAC.

MedPAC. 2005b. Report to the Congress: Medicare Payment Policy. Section C: Skilled Nursing Facility Services. Washington, DC: MedPAC.


National Center for Health Statistics. 2004. Health, United States. [Online]. Available: http://www.cdc.gov/nchs/hus.htm [accessed November 15, 2005].

NCQA (National Committee for Quality Assurance). 2001. Principles for Performance Measurement in Health Care. [Online]. Available: http://www.ncqa.org/communications/news/prinpls.htm [accessed August 30, 2004].

NQF (National Quality Forum). 2003. Safe Practices for Better Healthcare. Washington, DC: NQF.


O’Connor GT, Quinton HB, Traven ND, Ramunno LD, Dodds TA, Marciniak TA, Wennberg JE. 1999. Geographic variation in the treatment of acute myocardial infarction: The Cooperative Cardiovascular Project. Journal of the American Medical Association 281(7):627–633.

Organisation for Economic Co-operation and Development. 2005. Infant Mortality Rate, Deaths per 1000 Live Births. [Online]. Available: http://www.oecd.org/dataoecd/35/19/35027658.xls [accessed August, 2005].


Partnership for Solutions, A Project of The Johns Hopkins University and The Robert Wood Johnson Foundation. 2002. Chronic Conditions: Making the Case for Ongoing Care. [Online]. Available: http://www.partnershipforsolutions.com/DMS/files/chronicbook2002.pdf [accessed November, 14, 2005].


Rogers WH, Kazis LE, Miller DR, Skinner KM, Clark JA, Spiro III A, Graeme FB. 2004. Comparing the Health Status of VA and Non-VA Ambulatory Patients. Journal of Ambulatory Care Management 27(8):1–14.


Shapiro MF, Morton SC, McCaffrey DF, Senterfitt JW, Fleishman JA, Perlman JF, Athey LA, Keesey JW, Goldman DP, Berry SH, Bozzette SA, Additional Authors from the HCSUS Consortium. 1999. Variations in the care of HIV-infected adults in the United States: Results from the HIV cost and services utilization study. Journal of the American Medical Association 281(24):2305–2315.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×

U.S. DHHS (U.S. Department of Health and Human Services). 2000. Healthy People 2010: Understanding and Improving Health. Washington, DC: U.S. Government Printing Office.

U.S. Preventive Services Task Force. 1996. Guide to Clinical Preventive Services. Baltimore, MD: Williams & Wilkins.

U.S. Renal Data System. 2004. Annual Data Report. [Online]. Available: http://www.usrds.org/ [accessed March 3, 2005].


Walker J, Pan E, Johnston D, Adler-Milstein J, Bates DW, Middleton B. 2005. The Value of Health Care Information Exchange and Interoperability. [Online]. Available: http://content.healthaffairs.org/cgi/content/abstract/hlthaff.w5.10 [accessed January 25, 2005].

Wallace P. 2005. Personal communication to Karen Adams, March 28.

Wennberg JE. 1999. Understanding geographic variations in health care delivery. New England Journal of Medicine 340(1):52–53.

Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 83
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 84
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 85
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 86
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 87
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 88
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 89
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 90
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 91
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 92
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 93
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 94
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 95
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 96
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 97
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 98
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 99
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 100
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 101
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 102
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 103
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 104
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 105
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 106
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 107
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 108
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 109
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 110
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 111
Suggested Citation:"4 Moving Forward: What Should Be Measured?." Institute of Medicine. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press. doi: 10.17226/11517.
×
Page 112
Next: 5 Research Agenda »
Performance Measurement: Accelerating Improvement Get This Book
×
Buy Hardback | $75.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Performance Measurement is the first in a new series of an ongoing effort by the Institute of Medicine (IOM) to improve health care quality. Performance Measurement offers a comprehensive review of available measures and introduces a new framework to examine these measures against the six aims of the health care system: health care should be safe, effective, patient-centered, timely, efficient, and equitable. This new book also addresses the gaps in performance measurement and introduces the need for measures that are longitudinal, comprehensive, population-based, and patient-centered. This book is directed toward all concerned with improving the quality and performance of the nation's health care system in its multiple dimensions and in both the public and private sectors.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!