2
Putting Performance Measurement in Context

The panel views performance measurement as a management and oversight tool intended to improve desired outcomes by focusing attention on quantifiable measures of those outcomes, on changes (or lack of change) in those measures, and on the processes and capacity being applied to achieve the outcomes. The principal aim of this report is to address technical and policy issues associated with the data and information systems needed to provide effective support for performance measurement for publicly funded health-related programs. Consideration of these issues must, however, take into account the broader policy context in which performance measurement is used. It is within this context that performance goals are defined and then translated into performance measures, for which information systems must be able to produce data of the needed scope and quality. This chapter reviews the characteristics and uses of performance-based management and accountability systems, some of their strengths and limitations, and examples of their application in federal and state government settings and in the private sector for health care organizations. It also notes ways in which such mechanisms rely on data that are already available and some of the potential limitations of those data for accurately assessing performance.

Use of Performance Measurement in Accountability Systems

As noted in Chapter 1, the movement to increase the accountability of organizations and programs for achieving desired outcomes, particularly in the public sector, has led to renewed interest in performance measurement. This approach



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 30
--> 2 Putting Performance Measurement in Context The panel views performance measurement as a management and oversight tool intended to improve desired outcomes by focusing attention on quantifiable measures of those outcomes, on changes (or lack of change) in those measures, and on the processes and capacity being applied to achieve the outcomes. The principal aim of this report is to address technical and policy issues associated with the data and information systems needed to provide effective support for performance measurement for publicly funded health-related programs. Consideration of these issues must, however, take into account the broader policy context in which performance measurement is used. It is within this context that performance goals are defined and then translated into performance measures, for which information systems must be able to produce data of the needed scope and quality. This chapter reviews the characteristics and uses of performance-based management and accountability systems, some of their strengths and limitations, and examples of their application in federal and state government settings and in the private sector for health care organizations. It also notes ways in which such mechanisms rely on data that are already available and some of the potential limitations of those data for accurately assessing performance. Use of Performance Measurement in Accountability Systems As noted in Chapter 1, the movement to increase the accountability of organizations and programs for achieving desired outcomes, particularly in the public sector, has led to renewed interest in performance measurement. This approach

OCR for page 30
--> to the management of public programs and policies is believed to be superior to other management approaches that are based on micromanagement, process controls, and oversight of resources and activities, and that place little emphasis on results (Osborne and Gaebler, 1992; Wholey and Hatry, 1992). Performance-Based Accountability Systems As defined by the National Performance Review (1993), the guiding principle of governmental performance-based accountability systems is the provision of increased flexibility to lower-level units of government, or ''partners," in exchange for increased accountability for results. This increased flexibility may take the form of consolidation of funding streams, elimination of micromanagement, devolution of decision making, or a reduction in bureaucratic paperwork and reporting. Increased accountability for results means that partners focus on outcomes, rather than inputs and processes, as the basic measures of success. Some accountability systems may use such measures to allocate resources or apply incentives to reward desirable outcomes. Performance-based accountability systems are being established in the public sector between the legislative and executive branches of governments and between levels of government. The Performance Partnership Grants (PPGs) that were proposed by the U.S. Department of Health and Human Services (DHHS) for several of its state block grant programs are an example of a system intended to operate between levels of government. Such arrangements can be established between federal and state, federal and local, or state and local units of government. Even in the absence of formal PPG legislation, performance partnership agreements can be expected to function in this manner. Public-sector agencies are extending performance-based accountability into their relationships with the private sector through mechanisms such as performance-based contracting for the delivery of services. For example, state substance abuse or mental health agencies often contract with private providers to deliver publicly funded services. With performance-based contracts, those providers can be held responsible for certain overall outcomes among the people they serve. Performance-based accountability can even be extended to interrelationships in a broad community context. The community health improvement process described by the Institute of Medicine (1997) relies on performance measurement to monitor progress toward health improvement goals and ensure accountability of specific segments of the community for the processes and outcomes for which they have accepted responsibility. These management and accountability arrangements between and within units and levels of government can be viewed as a substitute for the private sector's market mechanism (see Wholey and Hatry, 1992). In the private sector, it is assumed that in the long run, the discipline of the marketplace will motivate firms to strive for cost-efficiency and maximization of returns to stockholders.

OCR for page 30
--> Measures such as profits, rates of return on investments, and market share can be used to assess a unit's success at maximizing efficiency. Furthermore, market forces and signals provide the sorts of feedback managers need to achieve such objectives. In contrast, the public sector is not governed by the economic forces of competition and profits. Residents who find their state services inadequate or overpriced generally cannot choose (unless they move) to use the services of another state the way a consumer can choose to buy a competing product. To judge the performance of the public sector, stakeholders must rely on other, noneconomic indicators related to human, social, and natural "capital" that must be preserved and invested wisely. There is less consensus on what these indicators should be than on the economic indicators of business performance. The performance partnership mechanism is one of a much broader class of performance-based systems that have been considered and used in public-sector management over the past 30 to 40 years.1 These systems include performance-based accountability, performance-based budgeting, performance monitoring, and benchmarking systems. While differing in various ways, all are predicated on a common view that government agencies and organizations need to be more accountable to legislatures, and ultimately to the public, for the resources they receive, and that this accountability should be based on improvements in the dimensions of well-being that such agencies seek to affect. Problems encountered in earlier efforts to apply performance-based systems offer lessons for current performance-based approaches (Florida Office of Program Policy Analysis and Government Accountability, 1997; U.S. General Accounting Office, 1997c). The extensive information needs of those earlier efforts were not adequately supported by the available record-keeping systems, staff expertise, and computer and information resources. Thus substantial staff time was necessary to meet reporting requirements. Despite this investment of staff time and other resources in producing the required reports, these efforts had little observable impact on funding decisions. The performance-based management approaches used in the past often lacked key leadership support in the executive and legislative branches of government. Furthermore, the analytic character of these approaches made them insensitive to the political aspects of deci- 1   The U.S. General Accounting Office (1997c) has reviewed previous performance-based management efforts. Those efforts can be traced back to the 1950s and program budgeting, which adopted budget categories based on activities and projects rather than classes of expenditure, such as salaries and capital equipment. In the early 1960s, the planning-programming-budgeting system (PPBS) was an attempt to link budgeting with both short- and long-range program plans. Management by objectives (MBO) was adopted in the early 1970s. It called for annual operating plans and targets for program objectives and focused more on productivity assessments than on outcomes. Over the course of the 1970s, MBO was replaced by zero-based budgeting (ZBB), which relied on an annual reexamination of the components of program budgets to assess opportunities for improved management and efficiency.

OCR for page 30
--> sion making. If performance measurement is to succeed, it must avoid problems such as these. Operation of Performance-Based Systems While various performance-based systems differ in their particulars, there appear to be several key steps in the operation of such systems. These steps are briefly reviewed here.2 In Box 2-1, these steps are illustrated by a hypothetical state immunization program and performance measures suggested in the panel's first report. Step 1: Develop an explicit set of goals and objectives and articulate strategies for achieving them. The first step for a performance-based system is to delineate clearly the goals and objectives of an agency or program. These goals and objectives are often captured in a strategic plan that includes a mission statement and a discussion of how the goals and objectives will be achieved. Furthermore, a strategic plan may outline the resources that will be used to meet these goals and objectives; it may explicitly stipulate the necessary expenditures as well. As noted earlier, one would expect the goals and objectives to focus on outcomes, not process. Such is certainly the case for the two recent federal initiatives in this area—the National Performance Review and the Government Performance and Results Act (GPRA). A key part of the process of setting goals and developing strategic plans is identifying and involving a program's stakeholders and balancing their potentially competing interests (e.g., reduce costs, increase services, improve quality, replace one activity with another). Much of the recent literature (see, e.g., Wholey and Hatry, 1992; U.S. General Accounting Office, 1996) has emphasized the importance of involving all stakeholders—including policy makers, agency administrators, local program operators, clients, and in some cases members of the public—in the goal setting and planning processes. In the case of the performance partnership agreements addressed by this panel, granting agencies (e.g., various DHHS agencies) and grantee agencies (e.g., state and local agencies or organizations) may each have their own goals and strategic plans. Negotiated agreements are the mechanism for identifying the particular set of goals and objectives against which grantees' performance will be assessed. Step 2: Develop and implement strategies for measuring performance. A performance-based system must have a means of assessing progress toward stated goals. This method of assessment is provided by translating program objectives into measures of performance: quantitative or qualitative characterizations of 2   See U.S. General Accounting Office (1996, 1997c) for an overview of the structure of performance-based monitoring, accountability, and budgeting systems.

OCR for page 30
--> Box 2-1 Steps in the Operation of a Performance-Based Management System: Example of a Hypothetical State Immunization Program Step 1: Develop goals and objectives and strategies for achieving them. The strategic plan for a state's immunization program might have as a goal reducing vaccine-preventable illness by (1) increasing the age-appropriate immunization rates among children at 2 years of age and (2) increasing influenza immunization rates among adults aged 65 and older. The plan might call for achieving specific levels of immunization coverage at some point in the future (e.g., in 3 years). The strategies for achieving these goals might include enhancing a childhood immunization registry system to generate reminder notices for parents and creating an immunization awareness program to reach older adults. Step 2: Develop and implement strategies for measuring performance. The performance of the immunization program might be assessed using outcome measures, such as the incidence of measles, rubella, and other vaccine-preventable diseases among children and the incidence of influenza-related deaths among older adults; risk status measures, such as the age-appropriate immunization rates among 2-year-old children or the influenza immunization rate among older adults; and process measures, such as the proportion of parents with children under age 5 who report receiving an immunization reminder notice and the proportion of older adults living in the community who report having seen information outcomes to be achieved if those goals are to be realized, processes to be followed in efforts to achieve those outcomes, or capacity available to support those efforts. Although measures based on outcomes are a high priority, a mix of measures will generally be needed to assess the performance of a program from various stakeholder perspectives (e.g., program managers, funders, consumers). For programs that affect outcomes over the long-term (e.g., chronic disease prevention) or that guard against possible but rare adverse events (e.g., water treatment), it may be more meaningful to focus on measures that track risk reduction activities and capacity to respond than on outcome measures that would generally show little change in the short term and few differences from program to program. This panel's first report (National Research Council, 1997) provides an extensive discussion of the categories of measures deemed relevant for health-related programs (see Chapter 1 of the present report for a brief review of these categories). While the process of measuring performance, especially in terms of relevant

OCR for page 30
--> on where they could receive an influenza immunization. As a capacity measure, the state might use the proportion of children under age 2 that are included in the immunization registry. The data for these measures would be obtained from several sources. Measures of disease incidence might be limited to those diseases that the state has designated as reportable and for which the state health department collects data. Influenza-related deaths would be tabulated by the vital records system. If reasonably complete, an immunization registry could produce data on immunization rates among young children. A survey (e.g., the Behavioral Risk Factor Survey) would probably be the most effective way to obtain data on influenza immunizations among older adults, immunization reminders received by parents, and awareness of immunization services. Step 3: Use performance information to improve management practices or resource allocation. Persistently low or decreasing immunization rates would be a signal to examine the operation of the immunization program more closely. The process and capacity measures selected in Step 2 might reveal program weaknesses that could be remedied, such as improving the completeness of an immunization registry's coverage of young children in the state. Finding that the selected process and capacity measures were at desired levels would signal the need to examine other factors that might account for poor performance. For example, the year's influenza vaccine might have been less effective than usual because of the emergence of an unanticipated viral strain. outcomes that should be influenced by program activities, is likely to vary from one agency or context to another, the literature on these systems offers general guidance (e.g., Wholey, 1983; U.S. General Accounting Office, 1996). In its work on GPRA, for example, the U.S. General Accounting Office (GAO) (1996:24) has noted the importance of establishing "clear hierarchies of performance goals and measures" that reflect the roles and responsibilities at varying program levels, from planning and oversight to grass-roots delivery of services. GAO comments that the performance measures should be tied to program goals and, to the extent possible, demonstrate the results of program actions that are directed toward achieving those goals. At the broadest policy and management levels, a limited set of measures that focus on key outcomes and actions should be used. Including too many measures at this level can divert attention from key outcomes without improving the usefulness of the performance information as a management tool. These measures must, however, be chosen carefully, espe-

OCR for page 30
--> cially if they are to be used to monitor a diverse set of activities, such as those likely to be encompassed by federal block grants to states, since activities that are represented in the set of measures are likely to be seen as having a higher priority than those not represented. A greater number of measures may be appropriate at the more detailed operational levels, such as within a state or community program. Although the specific measures are likely to differ across organizational or operational levels, they should be related to each other through their relationship to activities that contribute to the achievement of program goals. Once measures have been selected, the necessary data must be collected and used to calculate those measures. For some measures, it may be possible to rely on existing data sources, while other measures may require new data collection or data processing procedures. Meaningful interpretation of performance results may also require data on other factors not directly related to program activities or goals but that can affect the environment in which a program is operating, such as widespread disease outbreaks (e.g., epidemic levels of influenza), natural disasters, or changes in the local economy (e.g., increased unemployment because of layoffs). The completeness, accuracy, consistency, and timeliness of the data must be assessed, but such assessments must be made in light of the trade-off between the benefits of improving the quality of the data and the cost of doing so. Issues related to producing performance data are at the heart of this report and are addressed at greater length in subsequent chapters. Step 3: Use performance information to improve management practices or resource allocation. The next step for a performance-based system is to apply the information obtained from performance measurement to assess progress toward desired outcomes. If progress is not adequate, performance information can inform steps taken to improve the likelihood of achieving outcome goals in the future. Some policy makers would like to use performance measures to determine resource allocation, directing additional resources to activities demonstrating "good" performance or reducing resources to those demonstrating "poor" performance. As discussed earlier, however, the panel cautions that use of performance measures in an arbitrary, formulaic approach to resource allocation generally is not appropriate because few performance measures can adequately and unambiguously represent the complex mix of factors that determine outcomes. Only if the measures are based on a definitive causal relationship between capacity and process and the outcome of interest, and if experience has demonstrated that they do not stimulate adverse unintended consequences, might it be reasonable to consider using them as a direct determinant of resource allocation decisions. The element of accountability that is central to such systems implies that performance data should be reported in a form that is accessible and useful to a program's stakeholders. It is critical to recognize that performance measurement is not an end in itself; it is a tool that should be used in a continuing process of assessment and improvement.

OCR for page 30
--> Applications of Performance Measurement Information obtained from efforts to measure performance can be used to various ends. This section highlights four potential ways such information might be used, particularly in the context of publicly funded health-related programs. The first two reflect a monitoring and reporting function for a performance-based accountability system. Accountability comes somewhat indirectly through the reactions of administrators and constituents in response to information on how an organization is performing. The latter two applications involve the use of performance information to influence program management and resource allocation more directly. These four applications of performance measurement information are not mutually exclusive, but they do differ in their implications for those whose performance is being measured. Inform various stakeholders (e.g., administrators, public officials, and citizens) of progress toward stated program goals. Performance measurement information can be used to compare actual performance with performance targets. Performance data can also be used to monitor progress over time or to compare the progress of multiple groups toward agreed-upon goals and objectives. For such comparisons to be appropriate and meaningful, the performance measurement information must be generated in ways that produce comparable data. For example, a state legislature might want to compare the state's immunization rates for 2-year-olds with the national target of 90 percent that was established in Healthy People 2000 (U.S. Department of Health and Human Services, 1991). The state might also want to assess progress toward this goal by local immunization programs across the state. Consumer-oriented reporting of performance information is illustrated by "report cards" on health care provider performance, such as that developed in conjunction with the Mental Health Statistics Improvement Program (MHSIP) (MHSIP Task force on a Consumer-Oriented Mental Health Report Card, 1996). Assess program effectiveness. Performance measurement can contribute to program management and accountability by serving as a primary method of surveillance for program effectiveness. It provides a framework to guide the systematic collection of information on desired outcomes and on the program activities that are specifically expected to contribute to the achievement of those outcomes. This performance information can provide an indication of how well programs are working. In addition, an ongoing performance measurement system can often provide data for assessing the effect of changes in other factors or programs related to health services (e.g., the growth of managed care). This panel's first report (National Research Council, 1997) advised that health-related performance measurement must include a mix of outcome, risk reduction, process, and capacity measures. The use of risk reduction measures to represent intermediate outcomes is important because, as noted earlier, many

OCR for page 30
--> health outcomes are too far "downstream" from program activities for direct causal linkages to be established or for those outcomes to be observed soon enough to be useful for program management. In general, routine and direct measurement of program processes and outcomes is not part of current practice at the state and local levels. As early, real-time indicators of program effectiveness, performance measures can signal matters warranting more attention. Additional analysis is then needed, however, to define the elements of a successful innovation or diagnose the source of a problem. Improve program performance. By providing sentinel markers of program effectiveness, performance measurement can guide program managers and policy makers in steps designed to improve program performance. Performance measurement can help focus the attention of practitioners, researchers, and policy makers on best practices. Attention to and accountability for processes and intermediate outcomes that are under more direct programmatic control than longer-term outcomes will lead to a much-needed emphasis on defining standards of practice in health program areas. From the external perspective of a funding agency, data showing poor performance may signal a program's need for increased technical assistance and for guidance in identifying appropriate practices and determining how they can be implemented. Incentives and sanctions are also used to encourage improved performance, but may prove difficult to use effectively in the public sector (Florida Office of Program Policy Analysis and Government Accountability, 1997). They can range from generally intangible positive (or negative) recognition for progress toward stated goals to specific and quite tangible financial rewards (or penalties) based on measured performance. The aim is to motivate program staff or communities to achieve desired outcomes (e.g., immunization rates, access to services, desired community behaviors) by comparing performance measurement information with targets set for program goals. As noted earlier, the private sector often relies on the prospect of financial rewards or penalties (e.g., profits, loss of market share) to create an incentive for good performance. For public-sector programs that do not operate in a competitive, market-based environment, financial penalties may only make it more difficult to improve performance. Instead other, nonfinancial tools can be used to improve performance. For example, continued poor performance that can be attributed to program mismanagement may call for penalties in the form of increased oversight, reduced flexibility, and more directive program management by the funding agency. The panel emphasizes that in the abstract, fear of sanctions may be an incentive toward improvement, but the application of sanctions will not, by itself, improve performance. Some observers suggest that fears by staff in state agencies that poor performance results will lead to penalties rather than assistance to improve performance can be a barrier to effective use of performance measure-

OCR for page 30
--> ment (Wholey and Hatry, 1992; U.S. General Accounting Office, 1994; Florida Office of Program Policy Analysis and Government Accountability, 1997). Guide resource allocation and regulation of activities. Performance measurement information is also being used for allocation of budget resources or as the basis for regulatory control to ensure a minimum acceptable performance. For example, some states have adopted performance-based budgeting systems under which decisions regarding agency budgets are directly linked to measures of agency performance (see below for additional discussion of state systems). The panel suggests that the use of performance measures in this manner for health-related programs is appropriate only when clear standards or substantial experience is available to guide actions in a manner that will avoid unintended adverse consequences. For example, linking funding for substance abuse treatment services to rates of treatment completion might discourage acceptance of clients who appear less likely to remain in treatment. In general, the panel believes that this process should not be as simple as rewarding or penalizing performance by providing or taking away resources. Indeed, as suggested earlier, such an approach may be counterproductive. Take, for example, a county with low immunization rates that have failed to improve over time. This situation could be the result of program mismanagement and poor decision making, or it could reflect especially intractable or unique local problems, such as continuing in-migration of families with underimmunized children. In either case, shifting resources away from this county to others with "better" performance would be unlikely to result in improved immunization rates. At the same time, however, a more complete understanding of program performance and its relation to outcomes will support a more rational, albeit more complex, budgeting and resource allocation decision making process. The panel is concerned that some legislative actions to mandate performance standards and impose financial penalties for failure to comply make poor use of the performance measurement tool. For example, the 1992 Synar Amendment is intended to reduce tobacco consumption among youths by reducing their access to tobacco products. This provision requires that each state reduce to less than 20 percent the proportion of inspected sales outlets that violate the ban on the sale of tobacco products to those under age 18. States that repeatedly fail to meet the required level of performance face the loss of up to 40 percent of their Substance Abuse Block Grant funds (Substance Abuse and Mental Health Services Administration, 1998). Complicating the federal-state relationship on this issue are regulations issued by the Food and Drug Administration (1996) that make the sale of tobacco products to minors a violation of federal law, and preempt most state and local laws on this matter. The panel sees at least four problems with the Synar Amendment's approach to performance-based accountability. First, the performance requirement was established without states having the opportunity to participate as partners in identifying the performance measure to be used or the level of performance to be

OCR for page 30
--> achieved. Second, the financial penalty reduces the resources available to address prevention and treatment of all forms of substance abuse, not just youth tobacco use. Third, the performance requirement and its associated penalty are not related to the typical program goals and strategies of state substance abuse agencies. Few of these agencies have any enforcement authority regarding tobacco sales, and states are specifically prohibited from using their Substance Abuse Block Grant funds for any enforcement activities other than inspections of sales outlets. Finally, the penalty is based on a single process measure of performance (the proportion of sales outlets violating the ban on sales of tobacco to minors) without an assessment of the desired (intermediate) outcome—a reduction in tobacco use among minors—or conclusive evidence of a causal link between process and outcome (see Rigotti et al., 1997). Examples of Performance Monitoring and Accountability Systems The PPG proposal that served as the impetus for the work of this panel is but one application of the performance monitoring and accountability systems that are currently in use in a variety of settings. Perhaps the most prominent governmental example is GPRA, which requires all federal executive branch agencies to implement a strategic planning and performance measurement process. Various federal programs that provide funding to states also include performance reporting requirements. The Maternal and Child Health Bureau (MCHB) of the Health Resources and Services Administration in DHHS (1997) has incorporated performance measures into the reporting requirements for the agency's block grant. The new welfare block grant program, Temporary Assistance for Needy Families (TANF), links both penalties and bonus funds to state performance in specified areas. States will also be required to develop and report on performance measures in connection with the Children's Health Insurance Program, a major initiative to extend health insurance to currently uninsured children in low-income families who are not eligible for Medicaid.3 And many state governments are adopting performance-based management and budgeting systems. In the private sector, interest in assessing and improving the quality of health care is prompting the development of performance measurement systems for health plans, health care facilities, and individual health care providers. Some of these examples of the use of performance monitoring and accountability systems are reviewed briefly below. 3   Information about the State Children's Health Insurance Program is available from the Health Care Financing Administration at <http://www.hcfa.gov/init/children.htm>.

OCR for page 30
--> makers. Furthermore, despite their commitment to performance-based management, state government personnel generally need more training in the development and use of performance measures. Information systems are recognized as necessary components of a performance-based management system, but they are frequently inadequate to generate the needed data on outcomes, program processes, and strategy-specific costs. Health Care Performance Measurement in the Private Sector Until fairly recently, performance-based accountability for health care outcomes has operated primarily on a case-by-case basis through malpractice claims and quality assurance programs, reflecting an assessment of the care provided by individual clinicians or hospitals to individual patients. More recently, quality improvement and performance measurement programs have altered this accountability framework by introducing continuous monitoring of the processes and outcomes of care for populations of patients. As under the performance-based budgeting approaches described above, clinical performance information provides management tools that can be used to promote improvements in health care. Some of the best-known recent efforts to develop performance measurement systems in health care have been led by employer groups, credentialing organizations, health maintenance organizations, hospitals, and private consultants. Among the leading private-sector efforts are those by the National Committee for Quality Assurance (NCQA), the Foundation for Accountability (FACCT), and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), which are described in more detail below. In addition, the American Medical Association (1998) has introduced an accreditation program for individual physicians that will include standardized measures of clinical performance and patient care results. There is an increasing degree of collaboration among these groups in the development of clinical performance measures and performance measurement systems. Moreover, as a growing proportion of Medicare and Medicaid services are provided by private-sector health plans, there is increasing public-private collaboration in the further development of some of these performance measurement systems. The federal government (U.S. Department of Health and Human Services, 1998) has announced plans to implement the recommendation of the President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry (1998) to establish a Forum for Health Care Quality Measurement and Reporting that will work with private-sector groups to develop a core set of measures and standards for measurement. The American Medical Accreditation Program of the American Medical Association, JCAHO, and NCQA have established a Performance Measurement Coordinating Council to coordinate their performance measurement activities and through which they anticipate working with the newly proposed forum (Joint Commission on

OCR for page 30
--> Accreditation of Healthcare Organizations, 1998b). In a more targeted collaboration, the Diabetes Quality Improvement Project has brought together FACCT and NCQA, plus the American Academy of Family Physicians, the American College of Physicians, the American Diabetes Association, the Health Care Financing Administration, and the Veterans Health Administration, to develop a set of diabetes-related performance measures suitable for use nationally (Diabetes Quality Improvement Project, 1998). National Committee for Quality Assurance One of the most prominent performance measurement tools in health care is NCQA's Health Plan Employer Data and Information Set (HEDIS), a set of standardized measures for comparing the quality of care provided by participating health maintenance organizations (National Committee for Quality Assurance, 1997a). Originally developed to inform employers purchasing health services for their employees, HEDIS has evolved to address consumer information needs as well. It now includes measures specifically for the Medicare and Medicaid populations, as well as the commercially insured. Health plan reports are filed with NCQA, which in 1997 began publishing an annual summary (National Committee for Quality Assurance, 1997e). HEDIS 3.0, the most recent version, includes 71 measures that health plans are required to use and 32 other measures (a "testing set") that are undergoing further evaluation and refinement. Each measure has a standard definition and technical specifications for data collection and calculation. For the measures based on data to be obtained through a member satisfaction survey, a standardized survey instrument has been developed (National Committee for Quality Assurance, 1997b). The HEDIS 3.0 measures cover the following domains of performance: effectiveness of care, access/availability of care, satisfaction with the experience of care, health plan stability, use of services, cost of care, informed choice, and health plan descriptive information. An ongoing review and development process has been established to support the continued evolution of HEDIS measures and the overall HEDIS system. The Committee on Performance Measurement, which oversaw the development of HEDIS 3.0, will continue to guide the review of current measures, the identification of measures to be retired, the testing of new measures, and a research agenda to support the development of new measures and overall improvements in performance measurement. Measurement advisory panels will provide additional expertise for work in specific areas (e.g., behavioral health, cardiovascular disease, women's health). HEDIS has become a widely recognized set of performance measures for assessing health care services provided by health maintenance organizations, but some have found it too limited in certain areas. In particular, the limited number of measures on mental health and substance abuse services has led to efforts by

OCR for page 30
--> others to develop suitable measures for managed behavioral health services (e.g., American Managed Behavioral Healthcare Association, 1995; J. Dilonardo, Substance Abuse and Mental Health Services Administration, personal communication, 1998). In an initial test of the feasibility of HEDIS, the Report Card Pilot Project provided useful lessons that were reflected in the development of HEDIS 3.0 (Spoeri and Ullman, 1997) and are relevant to the work of this panel. Specifically, the pilot project revealed the need to adopt a broad set of measurement domains and to field test measures before formal adoption. It also demonstrated the variation in the organization and operation of health plan information systems and the need for greater standardization to produce comparable data across plans. Clinical information systems were generally found to be weaker than those for administrative and financial data. External data audits were valuable in identifying errors and inconsistencies in data systems and in the specifications and processes used to calculate measures. The documentation for HEDIS 3.0 includes a set of audit standards (National Committee for Quality Assurance, 1997d) and a report specifically on the development of information systems that can support performance measurement using HEDIS (National Committee for Quality Assurance, 1997c). A continuing area of concern is the need for risk adjustment of HEDIS measures. Although this need has been recognized, suitable risk adjustment techniques for use across plans have not yet been developed. Foundation for Accountability FACCT was created in 1995 in response to a desire by consumer groups and purchasers of health care services for a more effective means of bringing their perspectives to bear on the assessment of health care quality (Foundation for Accountability, 1998a). Working with consumer focus groups and experts, FACCT has developed sets of measures for use in assessing care for adult asthma, alcohol misuse, breast cancer, diabetes, and major depressive disorder (Foundation for Accountability, 1998b). In terms of the panel's framework, these sets include measures of process, risk status, and outcomes, including measures of satisfaction with care for the specific condition. FACCT has also developed a set of measures that focuses on smoking as a health risk factor. Two other sets address general health status and overall consumer satisfaction with services and care (e.g., getting needed services, choice of providers). Under development are measurement sets for coronary artery disease, end-of-life care, HIV/AIDS, and pediatric care. The measures adopted by FACCT are field tested by health plans and group practices as part of the development process. FACCT has placed special emphasis on the consumer perspective and seeks to measure elements of health care quality that are important to consumers. In recent work with the Health Care Financing Administration, FACCT (1997) developed a framework intended to communicate health care performance infor-

OCR for page 30
--> mation (e.g., measures from FACCT and HEDIS) to Medicare beneficiaries in an effective manner.8 The project also explored conceptual and technical issues involved in constructing summary performance scores for health plans or health care providers. Joint Commission on Accreditation of Healthcare Organizations JCAHO has long served as one of the principal accrediting bodies for health care facilities. Its accreditation programs now include hospitals, home care agencies, long-term care facilities, behavioral health services, ambulatory health care providers, laboratories, and health care networks. Efforts over the past few years to integrate clinical performance measurement into JCAHO's accreditation process resulted in the Oryx initiative, which began in 1997 (Joint Commission on Accreditation of Healthcare Organizations, 1998c). Included in the Oryx program are hospitals, long-term care organizations, health care networks and health plans, home care organizations, and behavioral health care organizations. In the past, the accreditation process has been based on evidence of compliance with JCAHO standards covering such matters as staff credentials, equipment, and policies (Joint Commission on Accreditation of Healthcare Organizations, 1998a). In the panel's performance measurement framework, these standards could be viewed as focusing primarily on capacity (i.e., inputs to health care services), rather than on processes or outcomes of care. The addition of performance measures is seen as a way for the accreditation process to stimulate and contribute to quality improvement efforts. The Oryx program will allow health care organizations to meet their performance measurement requirements through the use of a variety of measurement systems. For hospitals and long-term care facilities, JCAHO has approved more than 200 measurement systems operated by a variety of organizations. These include JCAHO's own Indicator Measurement System, which offers a set of performance measures focused on specific areas of patient care (e.g., obstetrics, trauma, oncology). Measures for health care networks have been selected from measure sets developed by JCAHO, FACCT, NCQA, the University of Colorado Health Sciences Center, and the University of Wisconsin (Madison). Health care organizations will report their performance data through the organizations that manage the specific measurement systems they adopt, not directly to JCAHO. To maintain their JCAHO accreditation, health care organizations must report on a specified minimum number of measures selected from approved measure- 8   The following reporting categories are proposed: The Basics, covering elements such as access, provider skill, and communication; Staying Healthy, covering education, prevention, and risk reduction; Getting Better, covering treatment and follow-up for illness or injury; Living with Illness, covering functional status and quality of life for persons with chronic conditions; and Changing Needs, regarding end-of-life care or care at times of major changes in functional abilities.

OCR for page 30
--> ment systems. For example, hospitals and long-term care organizations must initially report on at least 2 clinical measures that together are relevant to at least 20 percent of their patient population, or they must report on 5 measures. Health care networks must initially report on 10 measures. Plans call for increasing the required number of measures and patient population coverage. Separate reporting requirements are being developed for each accreditation program. An advisory council has been established to provide a continuing review of the measurement systems included in the Oryx program. This group will also help select a set of core measures for each accreditation program. Review of candidate measures for use by hospitals is expected to begin in late 1998. Recognizing that selection of a measurement system and use of specific performance measures will be unfamiliar tasks for some of the participating organizations, JCAHO has developed a guidebook and other resources to help organizations evaluate and select a measurement system that will meet their needs. Lessons for Publicly Funded Health Programs The evolution of performance measurement in health care in the private sector offers lessons to those developing performance measures for publicly funded health programs. One key lesson is that performance measurement requires a continuing effort to select and improve measures and the measurement process. The quality and usefulness of the performance data being produced by health care organizations continue to improve, but conceptual and technical challenges remain (see, e.g., Eddy, 1998). The individualized performance ''report cards" developed in the past by some health plans lack the comparability across plans and providers that might be achieved by the larger-scale performance measurement programs, such as those of NCQA, FACCT, and JCAHO. These latter programs rely on more standardized sets of measures and guidelines for collecting relevant data using standard methodologies. The activities of these nongovernmental groups are an important resource for performance measurement for the publicly funded health-related programs that the panel is addressing. The work done by these groups to identify suitable measures for clinical care can inform the selection of measures for related aspects of public programs. Likewise, the experience these groups are gaining in developing measurement standards and information system tools to support performance measurement in a health services context may help guide related efforts in the public sector. Conclusions Although the concept of performance measurement is hardly new and the use of performance indicators has been attempted episodically in various programs, the widespread use of such indicators in federal programs as contemplated by

OCR for page 30
--> GPRA is a new and significant requirement that is also emerging among state and local governments. Similarly, the increasingly widespread use of HEDIS and other performance measurement systems in health care is evidence of changing attitudes and expectations regarding accountability and management in the private sector. Early experience with these vastly expanded requirements for accountability suggests that the new approaches offer many attractive features, but successful implementation will require substantial and continuing efforts to overcome several challenges. Conceiving and developing measures that capture performance accurately and comprehensively is often difficult and should be guided by special expertise; lack of data to support selected measures may necessitate the use of second-best choices; and multiple sets of measures may be required to satisfy the needs of varied users (e.g., program managers, funders, and the public). As more is learned about the use of performance measurement, progress is possible on all of these fronts. After reviewing performance measurement experience in other contexts, the panel concluded that several principles should guide current efforts to implement performance measurement for publicly funded health programs. Link performance measurement to program goals. Performance measurement should be viewed as a tool that facilitates the monitoring and promotion of progress toward program goals, not as an end in itself. It must be based on a clear articulation of program goals and desired outcomes—health outcomes in the context of this report—and some sense of how those goals can be achieved. Outcome measures should reflect a program's goals, and measures of process and capacity should reflect the evidence on effective methods of achieving those outcomes. Performance measurement should be a constructive process that contributes to organizational capacity to meet program goals. Adopt a "market basket" approach. A performance measurement system should promote the development of recognized sets of measures with agreed-upon definitions from which program participants (e.g., states or communities) should be expected to select specific measures that reflect the program priorities and strategies they have adopted. Even though programs generally have a core set of goals and objectives that are applicable regardless of where the program is operating, they must respond to diverse needs and regional circumstances. This means that specific program priorities and the strategies adopted to achieve them are likely to vary across states and communities. Therefore, a single, mandated set of performance measures is not appropriate. However, an effort should be made to associate particular program goals and strategies with specific outcome, risk status, process, and capacity measures so that identical activities related to those goals and strategies can be monitored using the same measures. For example, a program to reduce teenage smoking might be expected to use a standard measure of smoking prevalence. The specific process and risk status measures adopted should reflect the choice of strategies for reducing the prevalence of teenage smoking (e.g., reducing access to cigarette vending machines, restrict-

OCR for page 30
--> ing tobacco advertising near schools). Ideally, each measure should be recognized as valid, reliable, and responsive to change. Recognize differing needs for performance information. The content and number of useful performance measures should be expected to differ between a program's operating level and the policy and sponsorship level across the intergovernmental structure. Compared with other levels, the operating level is likely to require more measures, and measures that focus more on process than on outcome. A performance measurement system should recognize these differing needs, but aim to use measures that can be linked, conceptually or in practice, to provide a consistent assessment of performance across these different levels. This principle is consistent with the GAO (1996) recommendation regarding GPRA that "hierarchies" of performance goals and measures are needed to reflect differing roles and responsibilities at various organizational levels. Ensure the feasibility of data collection and analysis. The most elegant performance measures are of little use without a feasible data system to support them. Considerations such as the quality of the available data and the cost of obtaining specific data elements may limit the choice of measures, particularly in the short run. In some cases, it may be necessary to use less desirable measures while enhancing existing data sources or building better data sets. The panel's first report (National Research Council, 1997) specifically noted that the lack of data comparable across states was a significant obstacle to identifying optimal performance measures for many program areas. Given the trade-offs involved, it is clearly important to consider data collection and analysis strategies as part of the development of performance measurement systems. Assess the consequences of using performance measurement. Performance measurement may achieve the desired effect of improving outcomes, or it may inadvertently promote undesired effects. Measurement results could, for example, be misinterpreted. A state with rates of food-borne illness that are higher than those of other states could be viewed as having problems in food safety practices when, instead, the higher rates reflect a more effective surveillance system. Another undesirable effect might be neglect of program areas or activities that are not being measured. Prematurely high expectations for performance data or rapid adoption of rigid performance targets could undermine intended program goals. For example, program practices might be manipulated to achieve "good" results, perhaps by avoiding populations that are difficult to serve rather than by implementing more effective services. The performance monitoring system, including individual performance measures, should be evaluated periodically to assess the consequences of its use. Such evaluation would help ensure that the system's goals were being met and decrease the likelihood of manipulation or inadvertent adverse effects, such as reduced services to groups that may be likely to have poor outcomes. Adopt a developmental approach. The development of a successful performance measurement system should be viewed as an activity that continues to

OCR for page 30
--> evolve over time. Furthermore, because performance measurement is a new and largely unfamiliar policy mechanism, it should be tested in the contexts of goal setting, progress monitoring, and signaling of progress or problems before being used for resource allocation or regulatory purposes. The panel advocates starting with a comprehensive vision for a performance measurement system that is implemented in manageable phases, during which the participants learn and the system grows. There must be a firm commitment to ongoing research to develop new and better measures, relate these measures to program actions, and improve the performance measurement system. Research and evaluation studies must be done to test the effectiveness of performance measurement as a tool for improving health outcomes and program management. For health programs, measures should be refined or replaced as understanding of the linkages between health outcomes and program activities ("processes") improves and as better sources of data are developed. Moreover, program priorities can be expected to change over time, necessitating the identification and testing of new performance measures. Sustained investments are needed in improvements to data systems, as well as in training and technical assistance to ensure that program and policy staff develop the necessary skills and expertise. With time and experience, performance measurement may prove to be an effective basis for allocation of resources or assessment of regulatory benchmarks, but it must always be used prudently, with an understanding of both its strengths and its limitations. References Administration for Children and Families 1998a. Temporary Assistance for Needy Families (TANF). Fact sheet. February 13, 1998. U.S. Department of Health and Human Services. http://www.acf.dhhs.gov/programs/opa/facts/tanf.htm (August 4, 1998). 1998b. Formula for Awarding the First High Performance Bonus in Fiscal Year (FY) 1999. Memorandum to state agencies administering the Temporary Assistance for Needy Families (TANF) program and other interested parties. March 17, 1998. U.S. Department of Health and Human Services. http://www.acf.dhhs.gov/news/welfare/highperf.htm (August 4, 1998). American Academy of Pediatrics 1992. The medical home. Pediatrics 90:774. American Managed Behavioral Healthcare Association 1995. Performance Measures for Managed Behavioral Healthcare Programs (PERMS). Washington, D.C.: American Managed Behavioral Healthcare Association. American Medical Association 1998. American Medical Accreditation Program. http://www.ama-assn.org/med-sci/amapsite/index.htm (July 31, 1998). Diabetes Quality Improvement Project 1998. Initial Measure Set (Final Version). August 14, 1998. http://www.facct.org/DQIP-fnl.html (November 25, 1998).

OCR for page 30
--> Eddy, D.M. 1998. Performance measurement: Problems and solutions. Health Affairs 17(4):7–25. Florida Office of Program Policy Analysis and Government Accountability 1997. Performance-Based Program Budgeting in Context: History and Comparison. Report No. 96-77A (April). Tallahassee: Florida Legislature. Available at http://www.oppaga.state.fl.us/budget/reviews.html/ (April 8, 1998 ). Food and Drug Administration, U.S. Department of Health and Human Services 1996. Regulations restricting the sale and distribution of cigarettes and smokeless tobacco to protect children and adolescents. Final rule. August 28. Federal Register 61(168):44395–44445. Foundation for Accountability 1997. Reporting Quality Information to Consumers. Portland, Ore.: Foundation for Accountability. 1998a. About FACCT. http://www.facct.org/about.html (April 15, 1998). 1998b. Measuring Quality. http://www.facct.org/measures.html (December 28, 1998). Institute of Medicine 1997. Improving Health in the Community: A Role for Performance Monitoring. J.S. Durch, L.A. Bailey, and M.A. Stoto, eds. Committee on Using Performance Monitoring to Improve Community Health. Washington, D.C.: National Academy Press. Joint Commission on Accreditation of Healthcare Organizations 1998a. Accreditation Information. http://www.jcaho.org/acr_info/acr_std.htm (August 16, 1998). 1998b. Nation's Three Leading Health Care Quality Oversight Bodies to Coordinate Measurement Activities. Press release. May 19, 1998. http://www.jcaho.org/news/nb.htm (June 5, 1998). 1998c. Oryx Fact Sheet for Health Care Organizations. http://www.jcaho.org/perfmeas/oryx/sidebar1.htm (July 24, 1998). Maternal and Child Health Bureau 1997. Guidance and Forms for the Title V Application/Annual Report. Maternal and Child Health Services Title V Block Grant Program. December 22, 1997. Rockville, Md.: U.S. Department of Health and Human Services, Health Resources and Services Administration. 1998a. Office of State and Community Health. U.S. Department of Health and Human Services, Health Resources and Services Administration. http://www.hhs.gov:80/hrsa/mchb/osch.htm (December 30, 1998). 1998b. State Systems Development Initiative (SSDI) Grant Application Guidance for FY98. U.S. Department of Health and Human Services, Health Resources and Services Administration. http://www.hhs.gov:80/hrsa/mchb/guidance.htm (June 4, 1998). Melkers, J., and K. Willoughby 1998. The state of the states: Performance-based budgeting requirements in 47 out of 50. Public Administration Review 58(1):66–73. MHSIP Task Force on a Consumer-Oriented Mental Health Report Card 1996. The MHSIP Consumer-Oriented Mental Health Report Card. Final report of the Mental Health Statistics Improvement Program (MHSIP) Task Force on a Consumer-Oriented Mental Health Report Card. Rockville, Md.: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services. National Academy of Public Administration 1994. Toward Useful Performance Measurement: Lessons Learned from Initial Pilot Performance Plans Prepared Under the Government Performance and Results Act. Washington, D.C.: National Academy of Public Administration.

OCR for page 30
--> 1998. Effective Implementation of the Government Performance and Results Act. Washington, D.C.: National Academy of Public Administration. National Committee for Quality Assurance 1997a. HEDIS 3.0/1998. Washington, D.C.: National Committee for Quality Assurance. 1997b. HEDIS 3.0/1998. Vol. 3, Member Satisfaction Survey. Washington, D.C.: National Committee for Quality Assurance. 1997c. HEDIS 3.0/1998. Vol. 4, A Road Map for Information Systems: Evolving Systems to Support Performance Measurement. Washington, D.C.: National Committee for Quality Assurance. 1997d. HEDIS 3.0/1998. Vol. 5, HEDIS Compliance Audit Standards and Guidelines. Washington, D.C.: National Committee for Quality Assurance. 1997e. The State of Managed Care Quality. Washington, D.C.: National Committee for Quality Assurance. http://www.ncqa.org/news/report.htm (April 7, 1998). National Performance Review 1993. From Red Tape to Results: Creating a Government That Works Better and Costs Less. Washington, D.C. Available at http://www.npr.gov/library/nprrpt/annrpt/redtpe93/index.html (August 27, 1998). 1997. Serving the American Public: Best Practices in Performance Measurement. Washington, D.C. Available at http://www.npr.gov/library/review.html (July 23, 1998). National Research Council 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. E.B. Perrin and J.J. Koshel, eds. Panel on Performance Measures and Data for Public Health Performance Partnership Grants, Committee on National Statistics. Washington, D.C.: National Academy Press. 1998. Providing National Statistics on Health and Social Welfare Programs in an Era of Change. Summary of a workshop. C.F. Citro, C.F. Manski, and J. Pepper, eds. Committee on National Statistics. Washington, D.C.: National Academy Press. Oregon Progress Board 1997. Oregon Shines II: Updating Oregon's Strategic Plan. Salem, Ore.: Oregon Progress Board. Osborne, D., and T. Gaebler 1992. Reinventing Government: How the Entrepreneurial Spirit Is Transforming the Public Sector. Reading, Mass.: Addison-Wesley. President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry 1998. Quality First: Better Health Care for All Americans. Washington, D.C.: President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry. Available at http://www.hcqualitycommission.gov/final/ (July 27, 1998). Rigotti, N.A., J.R. DiFranza, Y.C. Chang, T. Tisdale, B. Kemp, and D.E. Singer 1997. The effects of enforcing tobacco-sales laws on adolescents' access to tobacco and smoking behavior. New England Journal of Medicine 337:1044–1051. Spoeri, R.K., and R. Ullman 1997. Measuring and reporting managed care performance: Lessons learned and new initiatives. Annals of Internal Medicine 127:726–732. Substance Abuse and Mental Health Services Administration 1998. SAMHSA's Tobacco Activities: Implementing the Synar Requirements Under the Substance Abuse Prevention and Treatment Block Grant. U.S. Department of Health and Human Services. http://www.samhsa.gov/csap/synar/sydex.htm (May 18, 1998). U.S. Department of Health and Human Services 1991. Healthy People 2000: National Health Promotion and Disease Prevention Objectives. DHHS Pub. No. (PHS) 91-50212. Washington, D.C.: Office of the Assistant Secretary for Health.

OCR for page 30
--> 1998. President Endorses Quality Commission's Final Report and Issues Executive Memorandum to Improve Health Care Quality. White House fact sheet. March 13, 1998. http://www.hhs.gov/news/press/1998pres/980313a.html (August 4, 1998). U.S. General Accounting Office 1994. Managing for Results: State Experiences Provide Insights for Federal Management Reforms. GAO/GGD-95-22. Washington, D.C.: U.S. Government Printing Office. 1996. Executive Guide: Effectively Implementing the Government Performance and Results Act. GAO/GGD-96-118. Washington, D.C.: U.S. Government Printing Office. 1997a. The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven. GAO/GGD-97-109. Washington, D.C.: U.S. Government Printing Office. 1997b. Managing for Results: Analytic Challenges in Measuring Performance. GAO/HEHS/GGD-97-138. Washington, D.C.: U.S. Government Printing Office. 1997c. Performance Budgeting: Past Initiatives Offer Insights for GPRA Implementation. GAO/AIMD-97-46. Washington, D.C.: U.S. Government Printing Office. Wholey, J.S. 1983. Evaluation and Effective Public Management. Boston: Little, Brown. Wholey, J.S., and H.P. Hatry 1992. The case for performance monitoring. Public Administration Review 52(6):604–610. Zelio, J. 1997. Update on performance budgeting. LegisBrief 5(37). National Conference of State Legislatures.