Click for next page ( 79


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 78
4 Implementation and Evaluation We separation of policy design from implementation is fatal Pressman and W~ldavsly, Implementation The primary charge to the study committee was to advise AHCPR and the Forum about definitions and attributes of guidelines with an emphasis on the agengy's near-term responsibilities. Therefore, the committee was quite selective in its attention to implementation and evaluation matters. The major proposition here is that implementation and evaluation strategies should reinforce the credibility, validity, clarity, and other attributes of good guidelines identified in Chapter 3. The committee's examination of implementation and evaluation issues began with six basic propositions. Fast, guidelines are not self-implementing. Second, the effectiveness of guidelines in reaching their intended objectives cannot be assumed but must be tested. Third, attention to implementation and evaluation issues should be built into the processes of guidelines development rather than dealt with sequentially. . Fourth, many parhes must contribute if guidelines are to be suc- cessfully implemented and evaluated. Fifth, the eventual revision or redesign of guidelines and their 78

OCR for page 78
IMPl FMENTATION AN'D EVAllJATION 79 implementation processes needs to be anticipated as guidelines are first disseminated, used, and evaluated. Sixth, implementation processes will not be static but will evolve as better methods are identified and less successful approaches are discarded. Such evolution may complicate evaluation. The rest of this chapter emphasizes practice guidelines and makes only occasional comments about medical review criteria, performance measures, and standards of quaky. Implementation is discussed first, followed by a shorter section on evaluation. TYPES OF IMPLEMENTATION Implementation refers to the concrete activities and interventions undertaken to turn policy objectives into desired outcomes (Pressman and Wildavsky, 1973~. In the context of this report, these activities are viewed as a set of partly coordinated, partly disjointed steps or activities that include formatting, disseminating, applying, and revising or updating guidelines. Ho overlapping but distinct implementation tasks can be distinguished. One is implementing the public program established by OBRA 89. The other is implementing the practice guidelines themselves. PRO GRIM IMPLEMENTATION OBRA 89 gives primary responsibility for establishing a program to develop and promote practice guidelines to the Department of Health and Human SeIvices (DHHS) through AHCPR and its Forum. Necessary steps for implementing this program include hiring staff, developing a program agenda, establishing an advisory council, letting contracts, convening ex- pert panels, and generally establishing and administering a broad, ongoing federal program.) The IOM committee's advice about definitions and at- tnbutes of guidelines is intended lo assist the agency with two aspects of program implementation: working with expert panels and assessing the results. Congress provided for an Advisory Council for Health Care Polio, Research, and Evaluation to advise the Secretary and the administrator of AHCPR on a broad array of activities. Council advice on the development of guidelines and the conduct of outcomes research must be handled by a special Subcouncil on Outcomes and Guidelines. Neither the Council ~ Private organizations, such as the American College of Physicians (ACP) and others that de- velop guidelines, face program implementation tasks that are similar in many respects to those faced By the Forum (ACE, 1986) The same statement applies to guidelines implementation.

OCR for page 78
80 CLINICAL PR4CTICE GUIDELINES nor the Subcouncil has explicit responsibilities for program oversight, and it remains to be seen what their specific roles will be with respect to the program. The Forum can underscore its intent to examine critically and improve its program and products in at least three ways. First, it should ask its expert panels for feedback on the strengths and weaknesses of the procedures followed. Second, the Forum should pretest (or arrange for the pretesting of) all guidelines developed under its aegis. This can be done on a pilot basis in a real delivery setting, on a set of prototypical cases, or by both methods, but it needs to be done. Third, the Forum should try tO evaluate the effectiveness of intermediate actions (for example, formatting, dissemination, incentives) that are necessary if guidelines are to have their intended effects on health practices, outcomes, and costs. Each of these steps can be part of a learning process for the Forum and others. GUIDELINES IMPLEMENTATION The second Implementation task the focus of this chapter involves taking a set of practice guidelines, once they have been developed, into the actual world of health care delivery. It is in such a sphere that guidelines will prove themselves as effective or ineffective interventions. Government Responsibilities for Implementing Guidelines OBRA 89 outlined certain responsibilities for AHCPR and the Forum that relate directly to the implementation of practice guidelines. The Forum was specifically directed to concern itself with formats of guidelines (for medical educators, consumers, practitioners, and medical review organ- izations) and with disseminating and otherwise making guidelines avail- able. More generally, the legislation requires that the Secretary of Health and Human Services "provide for the use of ithe initial set of] guide- lines. . .to improve the quality, effectiveness, and appropriateness of care provided under Title XVIII" This and other legislative language implies the involvement of other federal agencies such as the Health Care Financ- ing Administration (HCFA) and the National Library of Medicine (NLM). Although nothing is said about the role of government as a direct provider of health care services, the health care systems administered by the De- partment of Defense, the Department of Veterans Affairs, and the Public Health Service clearly allow a government role in implementing guidelines in health care settings.

OCR for page 78
IMPLEMENTATION AND EVALUATION 81 Roles of Other Organizations in Implementing Guidelines Absent further legislation or regulations to the contrary, the federal government will not be the major actor in implementing guidelines. Rather, an array of private individuals and organizations will be the crucial partic- ipants in every stage of implementation. OBRA 89 refers explicitly to the involvement of provider and consumer organizations, medical educators, peer review organizations, and accrediting bodies in the dissemination of guidelines, but dissemination is only one step in assuring that guidelines are applied by clinicians and patients to improve health care outcomes. What goes on in delivery settings such as hospitals, physician offices, and nursing homes is the key to successful implementation. The foundation for such success, however, starts much earlier during the process of guidelines development. LINKS BEIWEEN DEVELOPMENT AND IMPLEMENTATION STAGES In discussing the attributes of good guidelines in Chapter 3, the com- mittee has underscored the link between development and implementation in several ways, including the following examples. 1. The credibility of the development process, the participants, and the scientific grounding of guidelines must be clear to intended users. 2. A truly multidisciplinary approach to guidelines development will facilitate acceptance and use of guidelines by members of the groups represented and by other, secondary groups. 3. Guidelines should be specific, comprehensive, and flexible enough to be useful in the varied settings and circumstances of everyday medical practice and in the evolving programs to assess the appropriateness of care provided in these settings. 4. Guidelines language, logic, and symbols should be easy to follow and unambiguous, so that movement from guidelines to educational tools, review criteria, or other instruments is unimpeded. 5. Pretesting is highly useful; it can be done on a pilot basis in a real delivery setting, on a set of prototypical cases, or by both methods. 6. The guidelines should specify what information about the clinical problem, the patient's circumstances and preferences, and the delivery setting should be recorded to permit later evaluation of the appropriateness of care Judged against criteria generated from the guidelines). Developers of guidelines need lo be cognizant of the practical realities involved in translating guidelines into medical review criteria and applying those criteria for quality assurance and other purposes. Such foresight is important whether the processes of guidelines and criteria development

OCR for page 78
82 CONICAL PRACTICE GUIDELINES are undertaken by the same or different bodies. Of the precepts listed above, those relating to clinical specificity, clarity, and data recording can be particularly useful in assisting the move from guidelines for practitioners to criteria for assessing health care practice. ~ say that implementation needs to be considered as guidelines are developed is not to imply that every step and detail can be or needs to be foreseen. That kind of foresight is impossible. Moreover, just as there is no "one best way" yet identified or demonstrated to develop guidelines, neither is there one best way to implement them. Pluralism in both phases is likely to be the norm for the present. PIANNING FOR IMPLEMENTATION Clearly, successful implementation will depend on many factors in addition to the quality and credibility of the guidelines and their design process. Among those factors are (1) the funding for dissemination and other implementation activities; (2) the incentives and supports for the guidelines to be used by practitioners, health plans, and others; (3) the accessibility, scope, accuracy, and timeliness of a variety of intra- and in- terorganizational information systems; and (4) the ability of multiple parties to plan and execute the various steps needed to implement guidelines. Compared with the processes of designing guidelines, the processes of implementation tend to be more diffuse. The time hornon extends beyond the near future, the number of involved parties multiplies, local circumstances become more important, responsibilities blur, and actions become more difficult to track These conditions make it more difficult to specify attributes of an implementation process in the way Chapter 3 specified the attributes of guidelines. Nonetheless, some of the factors that need lo be considered in making implementation decisions can be articulated as noted below. The particular objectives to be served by the implementation pro- cess. Examples: rapidly informing practitioners of new guidelines that depart significantly from previous guidelines; providing immediate and continuously available assistance to practitioners as they diagnose or treat particular patients; educating consumers about the use of screening services (Avorn and Somerai 1983; Lundberg, 1989; Somerai and Avorn, 1990~. The expected effecaveness of alternative strategies in achieving the objectives in question. Examples: the impact of direct mail notification compared with publication in a journal; the accessibility of an interactive computer system versus printed instructions (Jacoby and Clarke, 1986~. The cost of alternative strategies in relation to the elected benefit and the available resources. Examples: the cost of a press conference versus

OCR for page 78
IMPLFMENTATION AND E~AlIJATION 83 a direct mailing; the cost of a desktop compendium of guidelines versus a computer-based expert system. The demands made on target users by different alternatives. Ex- ample: the learning required for user-friendly versus non-user-friendly computer software. The manageability of the tasks for administrators or others respon- sible for implementing a decision. Example: setting up a system of financial incentives compared with setting up an information feedback system. Selecting the particular elements of an implementation plan requires assessment of these and other vanables. Inevitably, trade-o~s will be required among some factors such as expected effectiveness and cost or manageability. ELEMENTS OF IMPLEMENTATION In this context, implementation has four main aspects: formatting, disseminating, applying, and updating. These categorizations partly reflect OBRA 89 language. As a result' the discussion below uses some rather narrow definitions that should not be applied rigidly. FORMATTING Formatting refers to the presentation of guidelines in physical arrange- ments or media that can be readily understood by a designated set of users, for example, practitioners or patients. Different formats may be appropri- ate for different users and settings, for different means of dissemination, and for different types of guidelines. 1b the extent that the Forum asks developers of guidelines to assume responsibility for formatting, this task is less an implementation step than an initial design step. However, because formatting is so closely related to dissemination and application and because the responsibility for formatting is likely to be shared between developers and implementers, it is discussed here rather than ~ Chapter 3. In any case, the Forum will need to prepare some instructions on layout and sequencing of material for its panels and contractors. Formatting here emphasizes physical layout and logic; dissemination, on the other hand, focuses on the roles of different parties and media in getting information to different groups. Thus, for example, the physical properties and logic of documents or computer software are formatting issues, but decisions about how much and when to rely on written documents versus computer software are dissemination issues. Although the committee examined an extensive set of examples of

OCR for page 78
84 CONICAL PRACTICE GUIDELINES printed formats of practice guidelines and related materials,2 it was not able to review any empirical evidence on the effectiveness of different formats. Thus, the committee considered it inappropriate to recommend specific for- mats to the Forum. The materials examined viewed in the context of the attributes described in Chapter 3 did lead to several subjective judgments about simple features that distinguish better formats from inferior formats. These features include (1) one- or two-page summaries of key recommen- dations and rationales; (23 readily located descriptions of the development process, assumptions, objectives, methods, definitions, and participants; (3) selective use of boldface, subheadings, arid other highlighting techniques; (4) attractive typefaces and graphic aids; and (5) uncrowded layouts (for example, pages with ample margins and other '~white space"~. An index to major elements of the guidelines and a glossary of key terms and symbols might also be considered. In addition, a prominent listing of sources for additional information on the guidelines (or for related guidelines) may be a useful adjunct. More generally, regardless of whether formatting is treated as a design step or an implementation step, the attributes of guidelines related to credibility and accountability should be reinforced by the physical layout. For example, the user should quickly "see" that the guidelines emerged from a multidisciplinary process strongly grounded in scientific evidence and analysis including projections of health and cost outcomes. The attributes of Claris and reliabili~cyJreproducibility discussed in the preceding chapter are central. Logical presentation, precise terminology, clear and consistent use of words, phrases, and symbols, and similar prop- erties must be features of acceptable formats so that the guidelines are correctly and consistently understood. Likewise, users should be able to locate easily the descriptions of the populations covered by the guidelines and the identified exceptions. If guidelines describe patient care documen- tation that practitioners should provide, a prominent summary or checklist of such documentation needs is desirable. How the attributes of good guidelines can best be reinforced by format- ting choices will diner depending on whether the medium of dissemination is, for example, a journal article or a computerized decision support system. 21he majority of the materials reviewed by the committee pertained to clinicians (mainly physi- cians but also nurses) and included algorithms to guide patient management; statements about the appropriateness (orinappropmateness) of specific preventive, diagnostic, ortherapeuticpro- cedures; information about prescription drugs from the Physicians Desk Reference (1990~; the AMA Diagnostic and Therapeutic Technology Assessment (DACHA) Evaluation series; the on- line cancer protocols of Physician Data Query (PDQ); and up-to-date information from Sc~ennfic An2e7ican Medicine (Rubenstein and Federman, 1989~. The committee also considered materi- als aimed at patients, such as the algorithms developed by Rickety and Fnes (1986) to guide decisionmaking by patients at home, and other materials focused more exclusively on medical education.

OCR for page 78
IMPLEMENTATION AND EVAI~JATION 85 Again, tradeoffs are inevitable. For instance, "desktop" compendiums of guidelines will involve trade-offs among appealing formats, accessible lan- guage, and ready availability of many different guidelines. The importance of appealing, accessible presentations of guidelines for practitioners and patients is often underestimated (and the develop- ment of such presentations underfunded). Presentation is not merely a frill but an aspect of guidelines implementation that requires serious attention. Nonetheless, as for every other step described in this report, the develop- ment of effective formats will have to compete with other priorities and will involve learning over time about what works better. ~ help that learning occur, the Forum should allocate some of its limited resources (or seek assistance from outside sources) to evaluate the effectiveness of alternative formats and media. In the short term, the Forum should encourage its ex- pert panels or contractors to offer suggestions about layout and to produce, in addition to any format required by the Forum, an alternative document if the group judges that alternative to be superior. DISSEMINATION Dissemination as it is used here means getting guidelines to the in- tended users, particularly when they are initially published, adopted, or updated. Dissemination may occur in several phases or waves (Kaluzny, 1990~. The first step comes when the government or other sponsor of guidelines development begins to publicize the development of a new set of guidelines. Generally, this publicity involves enlisting the aid of other organizations such as medical specialty societies. Second, even without the sponsor's intervention, other parties including the news media, com- puter information systems, professional colleagues, and workplace health promotion programs may help spread information. As hospitals, HMOs, and other organizations decide that the guidelines should be adopted for internal use, they will communicate them to physicians, nurses, and others. Because many organizations are developing guidelines, the AHCPR and its Forum are considering the need for a clearinghouse function and their relation to it. The committee did not consider this role in depth but notes that the quality of guidelines needs to be considered in any clearinghouse activity. Wider dissemination of poor guidelines is not in the public interest. As described below, the focus of dissemination is on creating awareness and general understanding. Providing the practitioner with ongoing, routine access to information in an actual delivery setting is treated here as an administrative issue (as described in the next section)-. Legislatively and practically, the Forum's dissemination strategies will rely heavily on the capacities and preferences of private or quasi-public

OCR for page 78
~6 CLINICAL PRACTICE GUIDE' JNES groups such as medical organizations, consumer groups, and peer review organizations (PROs). As a consequence, no matter how cooperative these organizations are, many specific dissemination decisions will lie beyond the substantial influence of the Forum. Recognizing this, the committee offers only a few observations that may help the Forum and others in thinking about dissemination and its limits. The themes of credibility and accountability are relevant to decisions about dissemination. For example, using the journals, conferences, and other communication vehicles of medical specialty groups is more likely tO lend credibility to the guidelines than using a Forum press release or a PRO newsletter. The former approach may also provide more opportunity for "full disclosure," thereby enhancing the accountability of the process. Personal presentations by those involved in developing guidelines may likewise have more impact than publicity statements, but such presentations imply the availability of a-cadre of well-prepared presenters. Itade-offs are inevitable. For example, a press release may be the quickest vehicle for bringing high-priority guidelines3 to public and practi- tioner attention, but this timeliness typically comes at the loss (at least in the short run) of important information for different parties and the loss of some credibility with practitioners. PRO and carrier announcements are likely to be a swift mechanism for bringing Medicare-relevant guidelines to the attention of the medical community, but they are far less likely to reach the general public. The credibility and completeness of the information made available through on-line computer data bases (for example, MEDLARS) will depend on a variety of specific decisions about content, the extent of material to be available, retrieval rules, and development and access costs. Some evidence on the NLM's Physician Data Query (PDQ) suggests that patients are as likely as physicians to use that computerized information system, a finding that, if true of other, similar data bases, could raise some questions about bow to target and present information (L. Blankenbaker, comments during an AHCPR methodology workshop, Rockville, Md., May 30, 1990~. Dissemination of guidelines by PROs in the form of review crite- ria may be particularly helpful in focusing practitioner attention on the groups covered by the guidelines and on the exceptions. lathe PRO path of dissemination may, in turn, create an early feedback loop for updating the guidelines or criteria when important omissions are identified. For 3Such guidelines include those involving a dramatic change from previous guidelines or those involving information that, if quickly applied, could significantly affect mortality or morbidity (Moldover, 1990; Steinbrook and Lo, 1990, "Word of Spinal-Inju~y Drug Not Getting Out," Washingm Bet, 10~.

OCR for page 78
IMPr F::MEN1ATION AND EYALUAT10N 87 these reasons, early and formal involvement of the PRO community in the guidelines development process seems highly desirable. The development of computer-based expert systems may also include feedback provisions. More important, the use of such systems is likely to enhance the reliability and validity of clinical decisionmaking. APPLICATION AND ADMINISTRATION OF GUIDELINES Administering practice guidelines refers to the practical activities re- quired for users tO apply the guidelines in making specific decisions about appropriate health care for particular patients or classes of patients. As noted earlier, it is unrealistic to expect the developers of guidelines to an- ticipate all the contingencies that different users of guidelines will routinely face. Some adaptation is an inevitable and frequently desirable feature of guidelines application. The primary individual appliers of practice guidelines are, in the short run, likely tO be physicians, nurses, or other clinicians whose services come within the scope of a particular set of guidelines. The greater the role of the patient in malting a health care decision, the more the patient will be a primary individual user of guidelines. The primary organizational appliers of practice guidelines will be health care providers including hospitals, nursing homes, group practices, and public clinics. The primary users of medical review criteria, standards of quality, and performance measures should be, first, health care providers in their internal review and monitoring programs and, second, external quality review programs, health benefit plans, and claims payers. Placing health care providers first as users of these instruments reinforces the call of the recent IOM report on quality assurance for Medicare (1990) for increased emphasis on professional self-review, outcomes assessment, and information feedback. OBRA 89 holds AHCPR and its Forum accountable for administering a government program, not for administering guidelines in a health care setting. The Forum and other government agencies may, however, support the application of guidelines at the practice level by funding the devel- opment of computer-based medical decision systems and similar activities. HCFA and its contractors are likely to administer medical review criteria for such purposes as preprocedure review, quality assurance, and claims administration for Medicare. Other agencies of the federal government, such as the Department of Veterans Affairs, the Department of Defense, and the Indian Health Service, presumably can have a more direct role in using guidelines, per- formance measures, medical review criteria, and standards of quality in

OCR for page 78
88 CLINICAL PRACTICE GUIDELINES hospitals, clinics, and similar delivery settings. Conceivably, these agen- cies could be laboratories for testing guidelines and alternative ways of implementing them. Provider Organizations Administrative processes to support the application of guidelines drill vary depending on an organization's purposes, structure, and resources. For example, staff-model HMOs, independent practice associations (IPAs), and public clinics may differ in how they integrate the use of guidelines with ongoing organizational programs such as staff education and patient outreach. The type of targeted mammography screening guidelines and programs that a staff-model HMO with a stable membership can manage (Field, 1990) would seem to be of questionable feasibility for a clinic serving migrant workers. Similarly, organizations with more resources (for example, libraries, video centers, telephone hotlines, personal computers, and access to local and national computer information systems) will be able to facilitate the use of guidelines in ways that are out of reach for less resource-rich organizations. Some organizations may want to establish formal programs to encourage adherence to guidelines. Such programs might include (1) preparation of standard operating procedures for staff (for example, for checking anesthesia equipment), (2) contacting with practitioners based on their acceptance of selected guidelines, or (3) periodic feedback on how an individual practitioner's behavior compares with both peer practices and specific guidelines. These mechanisms can reinforce guidelines along several dunensions including validity, reliability, and credibility. Review Organizations Medical review criteria and other evaluation instruments, if properly developed and sensitively applied, can create incentives for adherence to practice guidelines. If poorly developed and insensitively applied, they can undermine support for practice guidelines. Fortunately, the processes of translating guidelines into review criteria and then applying them provide one opportunity for early identification and correction of omissions. The Forum may want to consider establishing early contact with the peer re- view and quality assurance communities to help ensure the best possible movement of guidelines into these applications. No one, however, should underestimate the challenges this movement entails. As discussed in Chapter 3, this committee's charge did not include recommendations on principles and techniques for translating guidelines into practice evaluation instruments. However, because review procedures have the potential either to support or undermine guidelines, the committee

OCR for page 78
IMPLEMENTATION AND EVALUATION 89 wishes tO set forth several principles for structuring and managing the review process based on earlier IOM reports on utilization management (1989) and quality assurance (1990~. 1. Consistent with the recommendations of the IOM report on private-sector utilization management, review criteria should be public with respect to their content and their development process (This does not require that software and other administrative tools be public.) 2. When criteria are used to assess quality of care, deny payment for specific services, or take similar steps, an appeals process needs to be pro- vided. This process must be clearly described to patients and practitioners and be free from unreasonable complexity, delay, or other barriers. 3. Review organizations should make their review activities as man- ageable and nonu~ousive as possible for both patients and practitioners. These organizations are one contributor to the growing"hassle factor" in medical care. 4. Insofar as possible, review organizations should provide construc- zve information and feedback to physicians aimed at improving practice rather than punishing missteps. The Forum will need to work with provider groups, HCFA, the PROs, and other organizations to encourage careful development and application of review criteria and other practice evaluation tools. These are not simple or unimportant tasks. Other Organizations Organizations involved with medical informatics and education clearly have a role to play in the dissemination and administration of practice guidelines. Many researchers, practitioners, and institutions are now in- volved in developing prototype expert systems and other computer support for information retrieval, decisionmaking, and monitoring (Shortliffe, 1987; Greenes and Shortliffe, 1990, Williamson et al., 1989~. The relative utility of different approaches, for example, comprehensive nationwide systems (such as PDQ) versus institution- or seIvice-specific systems, should be evaluated. Intended users of information systems may not, in fact, find them useful. The widespread use of information and decision support systems will, however, probably require extensive involvement of commercial enterprises with the capital and expertise to develop, market, and maintain generally usable software, consulting services, and so forth. AHCPR may want to give early consideration to its stance on such commercial developments as they pertain tO the guidelines emerging from expert panels.

OCR for page 78
go CLINICAL PRACTICE GUIDEI ONES In another area, the medical specialty boards may provide a power- ful incentive for the understanding and use of guidelines. For instance, information and practice behaviors relating to guidelines can be built into both certification and periodic recertification procedures. Several medi- cal specialty boards are considering the role of practice guidelines in the practice assessments they undertake or plan to undertake as part of their certification activities. Although steps such as these carry the application and administration of practice guidelines far from the immediate practical concerns of AHCPR, the potential importance of many disparate national and local organizations in the broader scheme of guidelines implementation should not be ignored. UPDATING AND REVISING In discussing attn~utes of guidelines, the committee proposed that developers of practice guidelines specify a date or timetable for a review to determine whether revisions to the guidelines are warranted. Reviews could come earlier than scheduled if there were indications of new clin- ical evidence or changing professional consensus. The credibility and ac- countabili~ of this process is as essential as that of the initial process of developing guidelines. The Forum thus needs to establish a follow-up mechanism to see that scheduled review occurs and that unscheduled reviews are initiated when necessary. In addition, the Forum should specify procedures for determining whether revisions are actually warranted and, if so, how those revisions will take place. The first step could be undertaken by AHCPR staff, by contractors, or by an advisory panel. It could include, in addition to the review of new clinical literature or evaluations of the guidelines' impact, a request for comments published in the Federal Register. The second step could involve convening new expert panels to make revisions consistent with the attributes specified in the previous chapter. 1b the extent that parties outside the government are developing similar or related guidelines, the Forum may also want to support a separate effort to review such "outside" guidelines as a means of informing its own priorities and processes. This activity could be incorporated into a clearinghouse function. The committee was concerned during its deliberations with the problem of inconsistent guidelines. By determining how the guidelines sponsored by AHCPR and those developed by others conform to each other and then investigating the reasons for inconsistencies, the agency may identify aspects of its guidelines that need updating and revising. As discussed in Chapter 5, however, inconsistencies in guidelines are not necessarily unacceptable. One particular implementation problem for the Forum and other

OCR for page 78
IMPLEMEN7CATION AND EVAL1JATION 91 sponsors of guidelines is ensuring that outdated versions of guidelines are abandoned. To the extent that a set of guidelines have been integrated into the operations of thousands of local organizations and practitioner of- fices and incorporated into specialized computer software and information systems, this practical element of updating will be a particular challenge. (The analogue in medical practice is the abandonment of obsolete proce- dures and therapies.) In some cases, only parts of guidelines may need to be withdrawn for instance, the clinical scope of guidelines may change without much need for modification of other elements. EVALUATION OF GUIDELINES The purpose of evaluation is to determine what outcomes both de- sired and undesired, anticipated and unanticipated have occurred as a result of a policy or program (Suchman, 1967~. For practice guidelines, the pray outcome variables identified in the legislation are the qual- ity, appropriateness, effectiveness, and cost of care provided to Medicare beneficiaries. However, evaluation that concentrates solely on ultimate out- comes and ignores intervening events may be incapable of distinguishing why a policy succeeded or failed. This chapter distinguishes two kinds of evaluation practice evalua- tion and guidelines evaluation. Practice evaluation focuses on health care decisions and interventions using various methods. Some methods, for ex- ample, randomized clinical trials, explicitly evaluate the impact of clinical interventions on such health outcomes as mortality, morbidity, and quality of life (Institute of Medicine, 1985; Kanouse and Jacoby, 1988; Kosecoff et al., 1987; Lomas et al., 1989~. Other methods, such as those employing medical review criteria and the other practice evaluation tools described in Chapter 2, typically do not assess outcomes but instead compare how actual practices (or proposed practices) match practices set forth in the review criteria or standards. These kinds of assessments assume that there are links between such practices and better health outcomes, although, as much of the quality of care literature makes clear, this is not always a viable assumption. A second type of evaluation, which is the focus of the following dis- cussion, is better described as a form of policy and program evalaaiion. The question is whether public and private policies and programs in this area have the effects intended; that is, do practice guidelines, as a policy instrument, affect clinical practice and health outcomes? This kind of eval- uation can encompass every step in the development and implementation of guidelines and the intermediate outcomes of each of these steps. Evaluating the impact of guidelines means determining their major intended and unintended effects and, insofar as possible, the causes of

OCR for page 78
92 CLINICAL PRACTICE GUIDELINES these effects (or their absence). A recent survey of practice guidelines activities conducted for the IOM concluded that, among organizations involved with guidelines, implementation and evaluation have received secondary emphasis compared with development and promulgation (Audet and Greenfield, 1989~. Relatively few steps were under way or planned tO evaluate the impact of guidelines on the cost, quality, and outcomes of care and on patient and practitioner satisfaction This neglect of evaluation is unfortunate because the effectiveness of guidelines cannot be taken for granted. Leo areas of concern can be raised with respect to the evaluation of practice guidelines. First are narrow issues relating to specific legislative requirements for DHHS. Second are broad questions about how to evaluate the impact of guidelines and build better policies and programs based on that evaluation. This report focuses on the first set of issues. The necessary planning to meet OBRA 89 requirements should start now. Such planning is particularly important because the legislation's requirements raise several problems, which the committee understands are recognized by department officials and congressional staff. Most simply stated, although the legislation's provisions for evaluation are laudable, the 1993 timetable for evaluating the first three guidelines developed by the Forum is unrealistic. On the one hand, the guidelines probably will not have had tune to make a measurable impact. On the other hand, even if the guidelines had had fairly immediate effects, the measurement data to document such effects would generally be unavailable. For example, insurance claims or other data showing changes in the use of procedures or practices may not be accessible in the tune frame specified. Likewise, data on patient outcomes will take time to collect. Rather than ask Congress for a change in the evaluation timetable, the Forum proposes to provide a status report as of January 1, 1993. The committee considers this appropriate so long as DHHS begins serious planning for the evaluation soon and takes steps to put necessary data collection processes in place. As noted earlier, many of the steps needed for evaluation can and should be initially considered and specified as guidelines are developed. Indeed, a hallmark of good evaluation research is that planning for the evaluation begins before the program gets under way. Over the long term, the data development responsibilities of AHCPR can be used to support guidelines evaluation as well as outcomes and effectiveness research. OBRA 89 requires evaluation of the impact of guidelines on quality, effectiveness, appropriateness, and cost of care. Information on znzennediate outcomes or znzervenzng variables is also important to determine such facts as whether the guidelines have, indeed, been received, read, understood, accepted, and remembered by practitioners and patients (Kaluzny, 1990;

OCR for page 78
IMPl FMENTATION AND EVALUATION 93 Lomas, 1990~. Such information is essential tO understanding why a set of guidelines have or have not achieved their desired outcomes and to determining whether to continue, revise, or abandon the guidelines. In general, explanations for policy success or failure need to consider evidence about the following: . the validity of the policy premises, for example, the assumption of many policymakers that broader development and use of practice guidelines will achieve significant cost savings; the quality of the implementation process, for example, the extent tO which information was disseminated or incentives were created for the use of the guidelines; the existence of countervailing events, for example, court decisions limiting the ability of health care organizations or payers to review the appropriateness of care and then deny either practice privileges or payment for practitioners providing inappropriate care; and the nature of supportive or enabling conditions, for example, the breadth of professional interest in the topic covered by the guidelines or a technical breakthrough in access to computer-based information systems. This chapter has suggested several aspects of the implementation pro- cesses for both the government program and the guidelines themselves that warrant assessment. These aspects include (1) the eDectiveness of different formats for a given guideline, (2) the impact of different dissemination strategies for different audiences, and (3) the role of alternative means of promoting day-to-day application of the guidelines. Indeed, the entire process of guidelines development will surely need investigation over time. No one approach is, in the short run (if ever), likely to prove definitively su- perior, although unsatisfactory methods can be identified and the strengths and weaknesses of other methods can be better understood. In addition, evaluation of the cost-effectiveness of different implementation activities could help in making decisions about how to allocate limited government and private resources. CONCLUSION The charge to this IOM committee was narrow: to provide timely advice to AHCPR on its initial steps to meet its responsibilities for prac- tice guidelines under OBRA 89. ~ that end, the committee focused on definitions and attributes for practice guidelines and on certain aspects of guidelines implementation and evaluation. For the latter, the emphasis is on how implementation and evaluation decisions can relate to and reinforce such attributes of guidelines as validity and reliability.

OCR for page 78
94 CLINICAL PRACTICE GUIDELINES The committee's basic conclusion is that AHCPR and the Forum must incorporate planning for implementation formatting, dissemination, ap- plication, and updating and evaluation within their development strategy. For the overall process to be successful, in fact, it may be necessary for the Forum to devote some of its scarce resources now to these more future-oriented needs. REFERENCES American College of Physicians. Clinical Efficacy Assessment Project: Procedural Manual. Philadelphia: 1986. Audet, A, and Greenfield, S. A Survey of Current Activities in Practice Guideline Development. Paper prepared for an IOM Meeting on Medical Practice Guidelines: Looking Ahead, Washington, D.C., November 8, 1989. Avorn, J., and Somerai, S. Improving Drug-Therapy Decisions Through Educational Outreach: A Randomized Controlled filial of Academically-Based "Detailing." New England Journal of Medicine 308:1457-1460, 1983. Field, M. Health Policy and Medical Practice Guidelines: The Case of Mammography Screening for Women Under 50. Paper prepared for an IOM Meeting on Medical Practice Guidelines: Looking Ahead, Washington, D.C., November 8, 1989. Greenes, R., and Shortliffe, E. Medical Informatics: An Emerging Academic Discipline and Institutional Pnonty. Journal of the American Medacal Association 263:111~1120, 1990. Institute of Medicine. Effects of Clinical Evaluation on the Diffusion of Medical Technology. Chapter 4 in Assessing Medical Technologies. Washington, D.C.: National Academy Press, 1985. Institute of Medicine. Controlling COStS and Changing; Patient Care? The Role of Utilimrion Management, B. Gray and M. Field, eds. Washington, D.C.: National Academy Press, 1989. Institute of Medicine. Medicare: A Strategy for Chalky Assurance, vols. 1 and 2, K Lohr, ed. Washington, D.C: National Academy Press, 1990. Jacol~y, I., and Clarke, S. Direct Mailing as a Means of Disseminating NIH Consensus Statements. Journal of He American Medical Association 255:132~1330, 1986. Kaluzny, A. Dissemination and Impact of Consensus Development Statements. Pp. 69~3 in Improving Consensus Development for Heals Technology Assessment: An Intemational Perspectn'G Washington, D.C.: National Academy Press, 1990. Kanouse, D.E., and Jacoby, I. When Does Information Change Practitioners' Behavior? International Journal of Technology Assessment in Heals Care 4~1~:27-33, 19~. Kosecoff, J., Kanouse, D., Rogers, W., et al., Effects of National Institutes of Health Consensus Development Program on Physician Practice. Journal of the American Medical Association 258:270~2713, 1987. Lomas, J. Words Without Action? The Production, Dissemination, and Impact of Consensus Recommendations. In Annual Review of Public Health, vol. 12, G. Omenn, ed. Palo Alto, Calif.: forthcoming (draft dated May 1990~. Llamas, J., Anderson, K, Dominick-Pie, E., et al. Do Practice Guidelines Guide Practice? Ibe Effects of a Consensus Statement on the Practice of Physicians New England Joumal of Medicine 321:1306 1311, 1989. Lundberg, G. Providing Reliable Medical Information to the Public Caveat Lector. Journal of the American Medical Association 2~2:945-946, 1989. Moldover, S. NIH Scrambles to Get Out the FAX Chevy Chase Gazette, May 3, 1990. Physician's Desk Reference, 44th ed. Oradell, NJ.: Edward Barnhart, 1990. Pressman, J., and W~ldavsly, ~ Implementation: How Great Expectations u: Washznpon Are Dashed u: Oakland. Berkeley, Calif.: University of California Press, 1973.

OCR for page 78
IMPLEMENTATION AND EVALUATION 95 Rubenstein, E., and Federman, D., eds. Scientific Amencan Medicine. New York: Scientific Amencan, 1989. Shortliffe, E. Computer Programs to Support Clinical Decision Making. Journal of the American Medical Association 258:61 66, 1987. Somerai, S.B., and Avorn, J. Principles of Educational Outreach ('`~ademic Detailing") to Improve Clinical Decision Making. Journal of the American Medical Association 263:549-556, 1990. Steinbrook, R., and Lo, B. Informing Physicians about Promising New Treatments for Severe Illnesses. Journal of the American Medical Association 263:2~7~2082~ 1990. Suchman, E. Evaluative Research. New Yori<: Russell Sage Foundation, 1967. Vickery, D., and Fries, J. Take Care of Yourself: The Consumer's Guide lo Medical Care, 3rd ed. Reading, Mass.: Addison-Wesley Publishing Co., 1986. Williamson, J.W., German, P.S., Weiss, R., et al. Health Science Information Management and Continuing Education of Physicians. Annals of Intemal Medicine 110:151-160, 1989. Word of Spinal-Injury Drug Not Getting Out, Group Says. Washington Post, April 11, 1990, p. AS.