Creating a Business Case for Quality Improvement Research: Expert Views: Workshop Summary (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

3 Research as a Driving Force for Change “Never doubt that a small group of thoughtful committed people can change the world; indeed it’s the only thing that ever has!” —Meade T here is an increasing challenge to translate and disseminate evaluation results in such a manner that they may be used in decision-making processes to improve health and health care, said Lori Melichar of the Robert Wood Johnson Foundation. To address the state of quality improvement research and its role in improving the health care system, panelists discussed the roles of foundations and academia in research, providing a sense for how research should be used to inform decision making and cre- ate the business case. Panelists were asked to discuss the following questions: • Are there effective research-oriented models in practice aimed at the translation of outcomes from improvement research to effec- tive operational practices? • What are the relevant direct research capabilities and infra- structure support required to build and sustain this research, and what are the current and future extramural funding sources that will share the investment costs with institutions? • What measurements are relevant to evaluate the return on this investment and future sustainability of quality improvement research? • Can a priority agenda for quality improvement research be identified nationally to stimulate and validate such research efforts? 32 

RESEARCH AS A DRIVING FORCE FOR CHANGE 33 Overview Jeffrey Alexander of the University of Michigan opened the session, offering high-level impressions of the state of quality improvement research and suggesting changes likely to make qual- ity improvement research more relevant, useful, and practical to decision makers. Beginning with his perspective on quality improvement research, Alexander noted that many studies are performed in sin- gle organizations (e.g., large university teaching hospitals), which raises questions about whether the interventions would be effective in community hospitals, small inner-city hospitals, or other insti- tutions. Second, the literature indicates that research tends to be opportunistic rather than systematic. Research is often conducted by faculty members at large teaching hospitals who see opportunities for testing interventions, such as process changes that have been introduced by administrative or clinical leaders in their organiza- tions. However, research should be conducted in a more planned, systematic manner. Third, quality improvement research often suf- fers from imprecise measurement and description of the quality improvement intervention, which limits adoption of the interven- tion by other organizations or replication of the research by others. In contrast to the way clinical research is advanced (i.e., replication of trials with different samples in different settings), little replica- tion exists in quality improvement research. Fourth, most studies are of relatively short duration, often lasting 12 to 18 months or less, precluding conclusions from being drawn about sustainability of changes. Fifth, there are often no explicit considerations of the organizational contexts or factors affecting implementation of an intervention. Sixth, explicit considerations of cost or value are often lacking. As a result of these six problems with the quality improve- ment literature, Alexander concluded there is inconsistent informa- tion regarding what works, when it works, where it works, and what it costs. An opportunity therefore exists to rethink how quality improvement research could be conducted to become more valid, generalizable, and useful to decision makers. To act on the opportunity for quality improvement, Alexander looked toward policy, informational, and financial barriers to quality improvement. The reimbursement system does not pay for quality, despite the Centers for Medicare & Medicaid Services’ (CMS’) recent efforts to not pay for preventable errors and pay for performance. The tipping point has not yet been reached where these programs will make large differences in care, Alexander said. From the per- 

34 CREATING A BUSINESS CASE FOR QIR spective of consumers, information is still not readily available to help patients distinguish between good and poor quality care. Fur- ther, the information that is available has not been shown to have influenced patients’ decision making, Alexander said. Additionally, little consideration has been given by major funding agencies to study implementation of quality improvement practices. The traditional, linear model of basic research to the deploy- ment of a treatment is not working, as evidenced by the insufficient uptake of organizations adopting best practices. A new model is needed that would take into consideration implementation as a component of quality improvement research and the effects of con- textual factors on implementation and quality improvement effec- tiveness. Although implementation research is not yet at a point that it can be called implementation science, Alexander noted, it is clear that factors affecting implementation can be divided into categories such as process, content, internal context, and external context. However, how these issues work together to predict and design effective implementation strategies remains unclear. Distinc- tions are often made among adoption, implementation, diffusion, and institutionalization or sustainability, Alexander added. While considerable research exists about adoption, relatively little exists about implementation, and almost none exists about perhaps the most important pieces: diffusion and institutionalization within and across organizations. To develop the capacity to strengthen quality improvement research, Alexander offered five suggestions. First, funding must be increased and more nonrandomized controlled trials should be supported. Second, multidisciplinary teams should be used more broadly because research is currently being conducted mostly by physicians, not economists or organizational researchers. Third, implementation should be considered part of the intervention, not as a by-product. Fourth, the duration of studies should be lengthened in order to draw conclusions about sustainability of an intervention. Finally, cost-effectiveness should be incorporated into the research agenda. Examples of quality improvement research Patrick Romano of the University of California, Davis, presented two examples for quality improvement research, offering lessons learned from each to improve both the validity and the interpreta- tion of research. Computerized provider order entry (CPOE) systems were found 

RESEARCH AS A DRIVING FORCE FOR CHANGE 35 to reduce serious medication errors by 60 percent in one major teaching hospital (Bates et al., 1999). As a result, a substantial move- ment toward adopting these systems began; in particular, efforts spearheaded by the Leapfrog Group gained attention. Although these findings were persuasive, Romano said, a significant decrease in adverse drug events was not demonstrated. Recently, questions have been raised about these findings. First, a qualitative study identified 22 different types of errors facilitated by CPOE systems (e.g., separation of functions resulting in double dosing and incom- patible orders; system crashes delaying medication orders; auto- matic cancellation of medications after surgery), some of which resulted inadvertently in harm (Koppel et al., 2005). Another study described unintended adverse consequences of CPOE systems, such as errors in communicating and coordinating processes (Ash et al., 2004), while a third study found a dramatic increase in risk-adjusted mortality among children transferred into an academic children’s hospital for specialized care, as a result of implementation problems (Han et al., 2005). Several of these single-system studies showed conflicting results, largely because of heterogeneity in how CPOE is implemented. Although pressure for widespread adoption exists, more time may be needed to fully assess the evidence. More harm than good may have resulted from the implementation of CPOE systems in some settings, but there is no way of knowing because no ongoing system for monitoring the nationwide impact exists, Romano said. The second example Romano presented was about one of the Joint Commission’s core measures of hospital quality: time to first antibiotic dose within 4 hours of admission for community-acquired pneumonia. This indicator, supported and endorsed by CMS and the National Quality Forum, is based on evidence from observational studies of thousands of patients, showing 15 percent reductions in both in-hospital and 30-day mortality with prompt administration of antibiotics. However, the evidence has limitations: The exact time period was never well established (no significant difference was found between 4 and 8 hours), and mortality did not decrease for those with prior antibiotic treatment. Recently, concerns about the measure’s validity arose. Due to the 4-hour imperative, 20 percent of patients treated according to this measure at one center left the emergency department without a diagnosis of pneumonia. Another   The Leapfrog Group is sponsored by the Business Roundtable to improve health care quality. One of its main “leaps” forward to improve patient safety is to encourage widespread implementation of CPOE systems. 

36 CREATING A BUSINESS CASE FOR QIR study showed that delayed antibiotic therapy in the emergency department is often due to legitimate diagnostic uncertainty or heavy patient load, highlighting the need for adequate systems of care outside the hospital. Finally, researchers discovered that since the measure’s introduction, pneumonia has been overdiagnosed. At one teaching hospital, 30 percent of patients admitted with a presumptive diagnosis of pneumonia are now discharged with a noninfectious diagnosis. The consequences of such overdiagnosis remain unclear, Romano said. Many lessons can be learned from these examples. Most impor- tantly, quality improvement interventions differ fundamentally from prescription drugs in that they are inherently heterogeneous. CPOE systems all differ, while the same system likely varies between hos- pitals. As a result, implementation and context make huge differ- ences, Romano concluded. Premature acceptance of new interven- tions based on surrogate markers is a real and serious issue, as described in the previous examples. Unintended consequences must be considered before widespread implementation occurs. Surveil- lance systems should be established to facilitate ongoing monitoring of these unintended consequences when new information technolo- gies are introduced. Specifically addressing quality improvement research, problems of internal and external validity may arise, Romano said. Although threats to internal validity do not necessarily affect all studies, they can be avoided through clever study designs. These threats include confounding, information bias, and patient selection and attrition biases. Threats to external validity include generalizability to other regions, settings, and facilities, as well as publication bias. To face these threats, Romano suggested that a number of research meth- ods could be employed. Cluster randomization is the best method for evaluating quality improvement interventions that are imple- mented at the hospital or clinic level. Recognizing the need for concurrent control or comparison groups, quasi-experimental study designs could be used to minimize confounding and improve inter- nal validity. Interrupted time series analyses are another approach to isolating the effect of an intervention. One key challenge is to create data systems that would allow evaluators to easily and effi- ciently find control variables. To minimize other biases, researchers should be blinded, implementation processes should be measured, and long-term outcomes should be evaluated in addition to short- term outcomes. Infrastructure should be enhanced to allow quality improve- ment research to advance health care. To improve generalizabil- 

RESEARCH AS A DRIVING FORCE FOR CHANGE 37 ity, studies should involve multiple institutions from a variety of regions and practice types. Data systems and registries must be supported to facilitate ongoing evaluations of the impact of inter- ventions, Romano said. This expanded infrastructure for quality improvement research will enable more cross-institutional stud- ies, multidisciplinary studies, researcher training, and funding, which are also necessary to improve the infrastructure. Conceptual frameworks from the social sciences and analytic methods from biostatistics and econometrics are examples of what should be bor- rowed and adapted from other disciplines. With respect to training, organizations such as the Department of Veterans Affairs (VA), the Agency for Healthcare Research and Quality, and the Health Resources and Services Administration fund programs in this area, but there has been little systematic effort to bring nonphysicians into the field. With respect to funding, the current investment in research and evaluation is inadequate. State and local governments should rec- ognize that they have large stakes in health care quality and should become more involved in applied quality improvement research, leveraging programs such as Medicaid and worker’s compensa- tion. Another point of examination should be the payment sys- tem because those people investing resources to improve quality tend not to be the actual recipients of financial rewards. To supple- ment current efforts, quality improvement should be linked to the National Institutes of Health’s Clinical and Translational Science Awards program and focus on translating science from the clinical level to the population or community level. Department of Veterans affairs initiatives The VA has been a leader in improving quality, said Joel Kuper- smith of the VA. With an annual budget of $35 billion, 5.5 million patients, and more than 1,400 sites of care, the VA has implemented a wide variety of efforts to improve care delivery, such as adoption of evidence-based practice guidelines, quality measures, leadership, and an electronic health record system (which, Kupersmith noted, was a bottom-up development by individual academic physicians). The VA also developed the Quality Enhancement Research Initiative (QUERI) to make evidence-based practices part of routine clinical practice, resulting in fundamental cultural change. Cultural change should trump technological change every time, Kupersmith said. To facilitate this, QUERI promotes research with continuous evalua- tion, usually of disease-specific processes. QUERI also develops and 

38 CREATING A BUSINESS CASE FOR QIR implements strategies for change, implemented first in single-site pilots and eventually implemented systemwide. Giving two examples of quality improvement research at the VA, Kupersmith first introduced the administration’s work in schizophrenia. Schizophrenia effects 1 percent of people, about 100,000 VA patients a year. Although this number is less than 2 percent of VA patients, care for people with schizophrenia equates to a much larger percentage of the VA’s health care costs. It had been shown that outcomes can improve with an evidence-based approach (e.g., appropriate medications, caregiver involvement, vocational rehabilitation), but many schizophrenics do not receive proper care. The purpose of the VA’s program was to identify gaps between evidence and practice. By performing formative and sum- mative research assessments, measures of evidence-based prescrib- ing and side-effect management both improved for schizophre- nia. Medication noncompliance was not significantly impacted. However, the success of the program led to the realization that management and improvement in treatment of schizophrenia are possible. As a result, a second program began, based on the forma- tive evaluations of the study. Offering a second example, Kupersmith said patients with spi- nal cord injuries were observed to have high mortality rates from respiratory disorders. A random survey of patient characteristics showed that patients with specific characteristics such as being older and nonsmokers got their vaccines, but others did not. However, the findings were only validated by modifying the study’s protocol during the study (e.g., more broadly targeting veterans, including more disorders, increasing the use of standing orders). Different approaches in basic, clinical, and organizational research will be required to fully study the implementation of quality improvement interventions. Researchers do not often view quality improvement research as true hard-core research, Kupersmith said, while chief executive officers do not necessar- ily consider it organizational. The field is more qualitative, with different end points and a different vocabulary. This research is observational and interventional, formative as well as summative. The VA has successfully advanced the quality-of-care agenda by leveraging the administrative system, the electronic health record, and high-quality research capabilities. Substantial improvement requires organizational and cultural changes, which are difficult to achieve. 

RESEARCH AS A DRIVING FORCE FOR CHANGE 39 Discussion Generalizability In response to a question about generalizability, Kupersmith noted that it is harder to translate efforts outside the VA than within the VA because it is a system with specific attributes. Translating efforts within the VA is not entirely possible because patients are all different. The challenge is to learn how to be vigilant within one’s own institution and learn how to share lessons with others without fear of liability or embarrassment, Romano said. Groups must share what they have learned and publish that information so others can benefit. Using the pneumonia example, must people be needlessly scared about getting unnecessary antibiotics in order for others who need the antibiotics to get them on time? Health care should learn from other industries that also encounter the need to reduce errors, such as the airline industry, where the costs associated with having error-free systems are built into business models. These trade-offs are beginning to be identified, but must be made explicit. Kuper- smith agreed, stating that transparency is critical for any quality improvement effort or research. Culture Alexander suggested two truisms of culture. First, changing cul- ture in a small organization is different from in large, complex orga- nizations. This is partly because large organizations do not have one culture, but multiple subcultures. Instead, a superordinate culture should be created that embraces the subcultures. This task is differ- ent from changing culture within a small hospital. Second, physician groups face less of a culture issue but more of an organizational climate issue. These groups often practice with a “siege mentality” and do not want to look outside of what is currently available, given the other pressures they are experiencing. A lot of work is involved to change behaviors in these organizations, Alexander concluded.