Monitoring, Evaluation, and Research: Future Directions
Pay for performance is one important means by which reimbursement mechanisms can be realigned to promote quality health care. Given both its considerable potential to impact care and the limited experience to date with implementation of such a program, a pay-for-performance program in Medicare should be closely monitored and evaluated so its design and use can quickly be modified to reflect experience gained in the real world. In addition, traditional forms of research must be used to address more technical questions. This chapter presents both a process for ongoing monitoring and evaluation and a research agenda.
Monitoring, evaluation, and research are integral components of a pay-for-performance program and should encompass the following:
Use of data collected from existing measures to help analyze the effect of pay for performance.
Processes for developing valid, robust performance measures that are increasingly linked to meaningful clinical outcomes and characterize performance comprehensively.
The ability to develop real-time monitoring systems that are capable of quickly identifying benefits—both intended and serendipitous—as well as unintended adverse consequences.
Definition of an initial research agenda to address technical matters pertaining to questions raised in this report.
Monitoring, evaluation, and research functions should not be divorced from program design and implementation or merely appended to pay-for-performance programs. Rather, their success depends on having a strong
learning system that is intrinsic to the design and activities of the program. Such a learning system would build on previous experiences and enable Medicare to better fulfill its congressional mandate to serve beneficiaries. In addition, it is expected that the private sector would closely attend to the Medicare experience, supporting the government’s mission to articulate national goals and leveraging private resources in concert with the public program. Conversely, the absence of a scientifically valid, comprehensive, integrated, and flexible system—one that facilitated learning from experience—would likely contribute to the failure of a pay-for-performance program. Aggressive actions are necessary now to take full advantage of the opportunity offered by pay for performance by increasing the knowledge base regarding the nexus of reimbursement, provider behavior, and quality of health care.
Recommendation 10: The Secretary of DHHS should implement a monitoring and evaluation system for the Medicare pay-for-performance program in order to:
Assess early experiences with implementation so timely corrective action can be taken.
Evaluate the overall impact of pay for performance on clinical quality, patient-centeredness, and efficiency.
Identify the best practices of high-performing delivery settings that should be shared with others to improve care throughout the nation.
This active learning system should be complemented by the identification of a more conventional research agenda through consensus among the major stakeholders and at the national level. This research agenda should address identified gaps in payment methodologies and the incorporation of new measures, and create the context for future investigations as actual experience with pay for performance raises new questions.
FUNDAMENTALS OF EVALUATION AND RESEARCH
Performance measures will be used to evaluate the benefits of pay for performance and to identify the effects of its implementation on health outcomes. Ideal measures should have the following characteristics:
Pertain to the domains of interest—clinical quality, patient-centeredness, and efficiency.
Link to meaningful clinical outcomes.
Be clearly defined.
Be easily implemented.
The first report in the Institute of Medicine’s (IOM’s) Pathways series—Performance Measurement: Accelerating Improvement—identified measures of potential value as well as gaps in existing measures to be filled through directed research (IOM, 2006b). The second report in the series— Medicare’s Quality Improvement Organization Program: Maximizing Potential—emphasized the need for expert technical assistance to providers attempting to use performance measurement for quality improvement purposes (IOM, 2006a). Performance measurement requires attention to the integrity and validity of baseline assessments so the impact of aligning reimbursement mechanisms through pay for performance can be accurately assessed. Performance measures pertinent to pay for performance should have clear metrics that are easy to interpret and implement; the burden on providers associated with the collection of the data should be reasonable; and the results should be clearly presented and easy to understand by all consumers of the information. Measures should emphasize meaningful patient outcomes rather than simply the more easily assessed processes of care, though it is of value to define and support the link between these two types of data. Special consideration should be given to measures that evaluate patient experiences and access to care. Consistent with the fundamental principle of “first, do no harm,” it is vital that new measures be able to detect evidence of unintended adverse consequences as quickly as possible.
To be of maximum use, lessons learned through the measurement of performance and the subsequent analysis of performance data should be publicly reported. This information should be communicated quickly and clearly in a manner that makes it useful to a wide variety of decision makers—patients, health care providers, payers, health plans, and regulators.
AN ACTIVE LEARNING SYSTEM
Lessons learned from those pay-for-performance programs already in place (see Chapter 2) have considerable potential to enrich the practical understanding of effective design principles for such programs: those that will achieve desired change without incurring significant adverse unintended consequences. The committee envisions a learning system that will focus on the impact of pay for performance on Medicare beneficiaries, but optimally will be able to identify safeguards and benefits that could be generalized to other patient populations and to programs in the private sector. The knowledge base supporting the effectiveness of the pay-for-performance concept is currently incomplete (Petersen et al., 2006); nevertheless, considerable program assessment and guidance are possible now.
To achieve success with the greatest efficiency, actual experience with pay for performance must be accurately and objectively assessed, and the assessment results—both best practices and practices that produce adverse
consequences—disseminated to those who need to make use of them. This learning system will require definition and coordination at the national level; the Medicare program is well positioned to take the lead in implementing this function. To be an effective leader, Medicare must collaborate with other public and private efforts in pay for performance. Examination of the collective experience with pay for performance and coordination of research will help advance learning and the entire research agenda. This collaboration should also lead to an alignment of pay-for-performance programs in order to reduce provider confusion and aid in achieving program goals.
The committee proposes that an active learning system be developed and incorporated into any Medicare pay-for-performance program; indeed, such a system, properly designed, will inform and guide the activities of the program itself. The active learning system should have the following characteristics and capabilities:
Focus data collection efforts on a robust set of performance measures that address national health care goals.
Collect, aggregate, analyze, and disseminate data in a fashion that allows for timely decision making.
Facilitate real-time program modifications based on evidence of benefits or adverse effects.
The passage of time, coupled with objective reflection, provides both a filter and a lens for program assessment. Caution, not tentativeness or hesitation, should characterize the implementation of a pay-for-performance program. An effective learning system seeks to instill an attitude of openness and inquiry that recognizes these programs as works in progress that should be guided, validated, and enhanced by data. The system should also create a climate in which modifications based on experience are welcomed.
Learning organizations have five key elements (Senge, 1990):
Systems thinking—looking at the dynamics of the system as a whole and in the long term, especially the interactions among individual parts of the system.
Personal mastery—learning and vision at the individual level, including the individual’s recognition of his or her role in the system.
Mental models—examination of how individuals approach problems; in a learning organization, individuals must learn how to recognize their own perceptions and work at being open to new models.
Building of a shared vision—creation of a picture of the future that is shared and desired by all individuals in the system, and therefore leads to increased enthusiasm and clarity.
Team learning—a process of using and enhancing the capabilities of individuals within the system, including ongoing dialogue, to achieve the shared vision.
Effective learning organizations also have strong leaders involved in creating the guiding ideals of the system, committing to and managing the shared vision, and helping to empower the individuals in the system. It is important for the Centers for Medicare and Medicaid Services (CMS) to assume this leadership role in the initiation of a successful learning system within a pay-for-performance program in Medicare.
All of the elements of a good learning organization also depend on an ongoing dialogue among the members of the organization. Such a dialogue includes continuous examination and reflection, as practiced commonly in the rapid-cycle improvement approach espoused by Shewhart and Deming (Value Based Management.net, 2006). In this approach, a group or individual learns about the consequences of an action and then responds to those results to improve the system (commonly known as a PDCA or PDSA cycle: Plan-Do-Check/Study-Act). This cycle involves planning a small change in a process, executing the change, studying the results, and ultimately taking action to react to the results. In a pay-for-performance program, this type of cycle could be used continuously to improve the program and react immediately to any consequences detected—either to expand upon a change that produces benefits or curtail a change that produces adverse effects.
Many others have studied the important elements of learning organizations that can help achieve continuous improvement (Garvin, 1993; Senge, 1996; Ferlie and Shortell, 2001; Frankl and Gibbons-Carr, 2001; Garcarz and Chambers, 2003; Rushmer et al., 2004). In the United Kingdom, Garcarz and colleagues (2003) developed a toolkit for creating a learning organization. Box 6-1 presents a list of the factors they identified as necessary for effective learning.
While many theories and perspectives exist as to what makes a learning system successful, there are certain overlapping elements CMS should consider when designing a pay-for-performance program in Medicare: providing strong leadership; developing a shared vision; creating an environment that allows for learning from experience (including mistakes); and, especially, considering the program as a whole, including the interactions among all individual elements.
The first report in the Pathways series, Performance Measurement: Accelerating Improvement (IOM, 2006b), proposed an aggressive agenda for
Factors Necessary for Effective Learning to Take Place
Empowerment of the workforce
Culture enabling learning from mistakes
Commitment to teamwork
Knowledge management systems
Education, training, and development needs analysis
Appraisal, performance, and personal development plans
Identified and protected education and training budget
Opportunity to apply new skills and knowledge
Sustained change and improvement
Time for learning and reflection
Feedback and evaluation
SOURCE: Garcarz et al., 2003.
further research. As articulated in that report, to help realize the vision of quality health care articulated in the Quality Chasm report (IOM, 2001), three aims of care should receive particular focus in the development of new performance measures: efficiency, equity, and patient-centeredness (IOM, 2001). The Performance Measurement report also called for measures to better assess the quality of care offered during a patient’s transition from one provider setting to another (e.g., from hospital to nursing home or from home to emergency department); such transitions have previously been identified as problematic periods during which errors are made, and important aspects of the comprehensive care plan are overlooked (Moore et al., 2003; Coleman and Berenson, 2004; Coleman and Fox, 2004; Coleman et al., 2005). Additionally, standards of care for patients with multiple chronic diseases are lacking. These patients are frequent users of the health care delivery system, seeing a variety of providers and utilizing many resources. Development of such standards is imperative to better understand how best to treat these patients who may derive most benefit from these efforts.
Numerous challenges must be faced in the development, implementation, and ongoing evaluation of performance measures that can align payment incentives with quality health care. Multiple methodological considerations—risk adjustment reflecting patient populations of varying acuity, small sample sizes at the individual practitioner level, comparative weighting of
measures based on their contribution to achieving identified goals, and attribution of responsibility among multiple providers participating in the care of a single patient—have already been identified as high-priority areas for further research to better asses the true impact of payment incentives.
Development of an Evidence Base
Many conclusions in this report are based on an analysis of reasonable alternatives rather than a firmly established, rich evidence base. Much additional data about pay for performance must be gathered, aggregated, and analyzed to inform decision makers. Full advantage should be taken of the research potential of every pay-for-performance program that is implemented; in particular, the committee reiterates the importance of conducting an ongoing assessment of any program implemented within Medicare.
CMS should carry out demonstration projects to evaluate options that are theoretically sound but untested. Such projects could limit risks and accelerate progress in realignment of reimbursement by confirming benefits and averting undue hardships for beneficiaries or providers. The following are examples of questions that should be addressed by demonstration projects or by other methods:
What is the threshold magnitude for rewards that will lead to significant changes in provider behavior? Does it vary among different types of providers (such as hospitals versus physicians)?
What are the effects of rewarding incremental change by recognizing relative improvement in practice versus rewarding only attainment of an absolute level of performance?
What criteria should be used to determine how rewards should be structured? Does the selected structure equitably recognize shared accountability among providers?
Should performance measures that enhance value to the patient always be the first priority for rewards? To what extent is control of costs advantageous to patients?
Should performance measures that recognize achievement in different domains of performance—clinical quality, patient-centeredness, and efficiency—be weighted differently in determining rewards?
How can a pay-for-performance program promote comprehensive rather than episodic care?
Should rewards be made at the organizational level or at the level of the individual provider?
Are virtual groups a feasible alternative?
How can best practices be identified and incorporated into a pay-for-performance program?
Is risk adjustment necessary to balance effectiveness and fairness?
How can a pay-for-performance program promote efficient health care without compromising clinical quality or patient-centeredness?
How should a pay-for-performance program be structured to sustain meaningful quality improvement over time?
A detailed analysis of methods for answering these research questions is beyond the scope of this report; however, a broad range of methodologies should be considered. In particular, mining of large databases offers real potential, particularly if the public and private sectors can be linked in a meaningful way.
It appears likely that oversight of any significant program of research and evaluation for pay-for-performance initiatives will have to occur at the national level, both within and outside the Medicare program and consistent with national consensus goals for health care. The National Quality Coordination Board proposed in the first report in the Pathways series (IOM, 2006b) is a particularly applicable model for this purpose. The Board might make use of the services of other organizations, both public (CMS, the Agency for Healthcare Research and Quality, the Government Accountability Office) and private (the Joint Commission on Accreditation of Healthcare Organizations, the National Committee for Quality Assurance, the National Quality Forum), that are already performing some of these functions with demonstrated ability.
This chapter has addressed issues related to the future directions for monitoring and evaluation of a pay-for-performance program within Medicare and proposed a research agenda. Key points made are as follows:
Monitoring, evaluation, and research will depend upon the development of valid and robust measures, as well as a real-time monitoring system.
A successful pay-for-performance program must encompass the elements of a true learning system, including having strong leadership, a shared vision, and an environment that allows for action in response to observations (including the opportunity to learn from mistakes).
A research agenda must address the fundamentals of performance measurement as necessary to align payment incentives with quality improvement. In the short term, the development of new measures should address
in particular the domains of clinical quality, patient-centeredness, and efficiency, and should also be focused on enhanced care coordination. Research should attempt as well to build an evidence base upon which the design of future pay-for-performance programs can be based.
Coleman EA, Berenson RA. 2004. Lost in transition: Challenges and opportunities for improving the quality of transitional care. Annals of Internal Medicine 141(7):533–536.
Coleman EA, Fox PD. 2004. One patient, many places: Managing health care transitions, Part III: Financial incentives and getting started. Annals of Long-Term Care 12(11): 14–16.
Coleman EA, Mahoney E, Parry C. 2005. Assessing the quality of preparation for posthospital care from the patient’s perspective: The care transitions measure. Medical Care 43(3):246– 255.
Ferlie EB, Shortell SM. 2001. Improving the quality of health care in the United Kingdom and the United States: A framework for change. The Milbank Quarterly 79(2):281–315.
Frankl SN, Gibbons-Carr M. 2001. Creating a school without walls and building a learning organization: A case study. Journal of Dental Education 65(11):1253–1263; discussion 1264.
Garcarz W, Chambers R. 2003. Creating and sustaining a learning organisation in the NHS. Quality in Primary Care 11(4):255–256.
Garcarz W, Fisher A, Chambers R, Ellis S. 2003. Towards a Learning Organization Toolkit. Commissioned by the Shropshire and Staffordshire Workforce Development Confederation. Unpublished.
Garvin D. 1993. Building a learning organization. Harvard Business Review 71(4):78–91.
IOM (Institute of Medicine). 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press.
IOM. 2006a. Medicare’s Quality Improvement Organization Program: Maximizing Potential. Washington, DC: The National Academies Press.
IOM. 2006b. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press.
Moore C, Wisnivesky J, Williams S, McGinn T. 2003. Medical errors related to discontinuity of care from an inpatient to an outpatient setting. Journal of General Internal Medicine 18(8):646–651.
Petersen L, Woodard L, Urech T, Daw C, Sookanan S. 2006. Does pay-for-performance improve the quality of health care? Annals of Internal Medicine 145(4):265–272.
Rushmer R, Kelly D, Lough M, Wilkinson JE, Davies HTO. 2004. Introducing the learning practice—I. The characteristics of learning organizations in primary care. Journal of Evaluation in Clinical Practice 10(3):375–386.
Senge P. 1990. The Fifth Discipline: The Art and Practice of the Learning Organization. New York: Doubleday.
Senge P. 1996. Building learning organizations. MIT Sloan Management Review 24(1): 96–104.
Value Based Management.net. 2006. The Deming Cycle. [Online]. Available: http://www.valuebasedmanagement.net/methods_demingcycle.html [accessed May 25, 2006].