Click for next page ( 2


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary In November 1989, Congress amended the Public Health Service Act to create the Agency for Health Care Polipy and Research (AHCPR). Under the terms of Public Law 101-239, this agency has broad responsibilities for supporting research, data development, and other activities that will "enhance the quality, appropriateness, and effectiveness of health care seIvlces.... " The needs and priorities of the Medicare program are an important but not exclusive focus of the agency. Many of AHCPR's responsibilities formerly belonged to the National Center for Health Services Research, which the agency replaced, but the emphasis on outcomes and effectiveness research is now consider- ably stronger. Other functions of the agency are new, in particular those involving a joint public-pnvate enterprise to develop, disseminate, and eval- uate practice guidelines under the sponsorship of the agency Forum for Quality and Effectiveness in Health Care. Shortly after its creation, AHCPR requested advice from the Institute of Medicine (IOM) on how it might approach these new responsibilities for practice guidelines. The IOM agreed to appoint a study committee that would work quickly to provide technical assistance and advice on definition of terms, specification of key attributes of good guidelines, and certain aspects of planning for implementation and evaluation. This report largely confines itself to these fairly specific and limited tasks. It is not a how-to- do-it manual, a methodology text, a priority-setting exercise, or a primer on guidelines for those seeking an introduction to the subject. The report does, however, aim to encourage more standardization and consistent in 1

OCR for page 1
2 Cr [NICAL PRACTICE GUIDEr rNES guidelines development, whether such development is supported directly by the Forum or is undertaken independently by medical societies and other organizations. The committee believes that the AHCPR initiative, taken as a whole, has real potential to advance the state of the art for practice guidelines, strengthen the knowledge base for health care practice, and, hence, improve the appropriateness and effectiveness of health care. One objective of this report is to encourage realistic expectations about this potential by building a broader understanding of the difficult but important steps needed to move toward the goals for practice guidelines stated in P.L 101-239, or, as it is also called, the Omnibus Budget Reconciliation Act of 1989 (OBRA 89~. CONTEXT The committee began its work with an understanding that the legis- lation establishing AHCPR is one consequence of accumulated public and private frustrations about the perceived health and economic consequences of inappropriate medical care. These frustrations and perceptions stem from many sources including ceaselessly escalating health care costs, wide variations in medical practice patterns, evidence that some health services are of little or no value, and claims that various kinds of financial, educa- tional, and organizational incentives can reduce inappropriate utilization. The combination of high expenditures and doubts about the value of that spending explains polic~makers' interest in improving the scope and application of knowledge about what works and what does not work in medical car~and at what price. AHCPR is supporting an extensive agenda of outcomes and effectiveness research. In fact, the major part of the agen~y's work involves expanding the scope of knowledge rather than applying it Of AHCPR's appropriation of nearly $100 million for fiscal year 1990, it planned to obligate around $2 million for the Forum's work on practice guidelines, compared with more than $30 million for outcomes research. Still, the agency's responsibilities for practice guidelines reflect congressional recognition of the practical need for ways to translate knowledge into patient and practitioner decisions that improve the value the nation receives for its health care spending. More generally, the creation of a practice guidelines function within AHCPR can be seen as part of a significant cultural shift, a move away from unexamined reliance on professional judgment toward more structured support and accountability for such judgment. Reflecting the first element of this shift, guidelines are intended to assist practitioners and patients in making health care decisions; reflecting the second aspect, they are to serve

OCR for page 1
SUMMARY 3 as a foundation for instruments to evaluate practitioner and health system performance. OVERVIEW OF PRACTICE GUIDELINES INITL\TIVES Public and private activities related to practice guidelines can be con- ceptualized, ideally, as having three basic stages: development, intervention, and evaluation. The second and third stages should again ideally involve feedback loops to the first stage to prompt the revision of guidelines when omissions, technical obsolescence, or other problems with a set of guidelines are identified. Guidelines are thus dynamic, not static. They reflect the interplay of scientific and technological progress, real-world organizational pressures, and changes in social values. ~ date, most government and other initiatives emphasize the first of the three stages, the development of guidelines. PUBLIC INITIATIVES Under OBRA 89, Congress created within AHCPR the Office of the Forum for Quality and Effectiveness in Health Care. The Forum must "arrange for" the development and periodic review and updating of (1) clinically relevant guidelines that may be used by physicians, educatom, and health care practitioners to assist in determining how diseases, disorders, and other health conditions can most effectively and appropriately be prevented, diagnosed, treated, and managed clinically; and (V standards of quality, performance measures, and medical review cntena through which health care providers and other appropriate entities may assess or review the provision of health care and assure the quality of such care. The phrase "arrange for" is one key indicator of the extent to which the drafters of OBRA 89 sought to create a public-private enterprise with respect to guidelines development. Their vision was that the Forum would itself develop no guidelines; guidelines were not to be federal creations. By January 1, 1991, the Forum must arrange for the development of an initial set of guidelines, standards, performance measures, and review criteria for at least three clinical treatments or conditions. The agency is also responsible for updating the guidelines developed under its auspices. Other key provisions of the legislation include the following. . The Forum can contract with public and nonprofit private organi- zations to develop and update guidelines, or it can convene expert panels to either develop guidelines or review guidelines developed by contractors. The director of the Forum must establish the standards for methods and procedures to be followed by the contractors and expert panels. The guidelines can be pilot-tested.

OCR for page 1
4 CLINICAL PRACTICE GUIDELINES Guidelines developed by private organizations independently of the agency program may be adopted by contractors or expert panels if they meet the requirements established by the legislation. The director of the Forum is to promote dissemination of guide- lines through organizations representing health care providers, health care consumers, peer review organizations, accrediting bodies, and other appro- priate entities. In addition, the guidelines must be presented In formats appropriate for use by practitioners, medical educators, and medical care reviewers. The Secretary of the Department of Health and Human Services (DHHS) "shall provide for the use of the [initial sets of] guidelines . . . to improve the quality, effectiveness, and appropriateness of care" provided under the Medicare program. Presumably, providing for the use of the guidelines will require that the Health Care Financing Administration (HCFA) and its contracting fiscal intermediaries, carriers, and peer review organizations take steps to incorporate review criteria and other evaluation instn~ments into their programs for reviewing care provided to Medicare beneficiaries. . The director of the Forum is to conduct and support evaluations of the impact of guidelines on clinical practice. More specifically, for the guidelines developed by January 1, 1991, the Secretary of DHHS must determine the impact of the initial sets of guidelines on the cost, quality, appropriateness, and effectiveness of medical care provided under the Medicare program and report the results to Congress by January 1, 1993. With respect to the research agenda of AHCPR, the director of the Forum is to recommend research projects related to the outcomes of health care, the processes for developing guidelines, and the use of guidelines. In general, when it examines clinical conditions as potential subjects for guidelines, the agency is instructed to consider how guidelines, stan- dards, performance measures, and review criteria can be expected to (1) improve health for a significant number of individuals, (2) reduce clinically significant variations in services and procedures provided by physicians, and (3) reduce clinically significant variations in the outcomes of health care. Moreover, the administrator of the agency is to consult with the Health Care Financing Administration and ensure that the needs and priorities of the Medicare program are appropriately reflected in the development of guidelines. For the initial three sets of guidelines, the legislation provides more specific selection priorities related to expenditures and services for Medicare beneficiaries. In addition to the new agency, other agencies of the federal govern- ment have related responsibilities. These agencies include the National Institutes of Health (NIH), the U.S. Preventive Services Ask Force, and

OCR for page 1
SUMMARY the Health Care Financing Administration and its contracting carriers, fiscal intermediaries, and peer review organizations. PRIVATE INITIATIVES Guidelines for clinical practice, broadly defined, are not new. The processes of organized clinical education require various sorts of guide- lines, as do the processes of professional licensure, board certification, quality assurance, utilization review, and other aspects of health services administration. However, the interest of the medical community and others in practice guidelines has grown exponentially in recent years. Moreover, there is greater emphasis today on formal procedures and methods for arriving at a widely scrutinized and endorsed consensus. Among the medical groups involved for some years with the develop- ment of guidelines are the American Academy of Family Physicians, the American College of Cardiology, the American College of Physicians, and the American Society of Anesthesiologists. In the research community, the RAND Corporation has pioneered the development of important tools and strategies. Newer initiatives are being undertaken or planned by the Amer- ican Board of Medical Specialties, the American Medical Association, the Council of Medical Specialty Societies, many individual specialty societies, and the academic medical and health services research community. Insurers, health maintenance organizations (HMOs), utilization man- agement firms, and similar organizations have not ignored the potential of practice guidelines to serge as ~ basis for refusing payment for inap- propriate care. For example, several years ago, the Blue Cross and Blue Shield Association began its Medical Necessity Project, which worked with researchers and some medical organizations to identity obsolete procedures and set guidelines for the appropriate use of many diagnostic and treatment services. The Health Insurance Association of America recently established a similar function, and the Group Health Association of America has been sponsoring programs on guidelines development. Individual members of each of these associations are involved in additional efforts to develop or adapt guidelines to meet the needs of their different health plans. The activities of dozens of firms supplying utilization management services to health plan sponsors have drawn attention to the quality, scope, and acces- sib~lity of the criteria they use to review care on a prospective or concurrent basis. lithe guidelines development efforts of private organizations are thus proceeding on many fronts. Some coordinating strategies are emerging, but important problems remain unexplained conflicts among guidelines, neglected topics, lack of follow-up, and incomplete public disclosure of the evidence, participants, and methods used to develop sets of guidelines. No

OCR for page 1
6 CLINICAL PRACTICE GUIDELINES independent entity exists to certify that guidelines are sound ~ method and content, and no "national bureau of standards" is available to set standards for methods of guidelines development or their content. The legislation creating AHCPR is one response to some of these problems. INSTITUTE OF MEDICINE COMMITTEE AND PROJECI The major immediate goal of this project was to help the Forum prepare to work with contractors or expert panels to meet its January 1991 deadline for developing at least three sets of guidelines. This assistance is also meant to help the Forum assess the soundness of the products (the guidelines) emerging from these contractors or panels. A second IOM project, described at the end of this summary, will develop a practical evaluation instrument for this purpose. 1b conduct the study requested by AHCPR, the IOM appointed a com- mittee of experts in January 1990 that met in February and again in April. The committee included practicing physicians, individuals experienced in the development of guidelines, current and potential users of guidelines, and representatives of relevant other disciplines such as nursing, law, and economics. Staff from the Forum attended both committee meetings and received copies of all draft and background materials prepared for the committee. FINDINGS AND CONCLUSIONS STATE OF THE ART The committee arrived at several general observations about the state of the art of practice guidelines development. Most generally, the prm cess of systematic development, implementation, and evaluation of practice guidelines based on rigorous clinical research and soundly generated profes- sional consensus, although progressing, has deficiencies in method, scope, and substance. Conflicts in terminology and technique characterize the field; they are notable for the confusion they create and for what they reflect about differences in values, experiences, and interests among differ- ent parties. Public and private development activities are multiplying, but the means for coordinating these efforts to resolve inconsistencies, fill in gaps, track applications and results, and assess the soundness of particular guidelines are limited. Disproportionately more attention is paid to de- veloping guidelines than to implementing or evaluating them. Moreover, efforts to develop guidelines are necessarily constrained by inadequacies in the quality and quantity of scientific evidence on the effectiveness of many services.

OCR for page 1
SUMMARY AHCPR AND THE FORUM 7 As a consequence of the above factors, AHCPR and the Forum have, at present, a somewhat restricted foundation for their work. In addition, at least three other variables must be taken into account in estimating what the agency is likely to be able to accomplish early in its guide- lines effort. First, although OBRA 89 addresses some concerns about guidelines development, implementation, and evaluation, it appropriately does not describe a precise course of action. The agency will need time to devise and revise practical and defensible approaches to guidelines develop- ment. Second, given that both the function and the organizational units (par- ticularly the Forum) are new to the Department of Health and Human Services, the legislative timetables for guidelines development and, partic- ularly, evaluation are unrealistically short. Moreover, the Forum has few staff to support the new functions, a deficit that is not likely to change in the near term. In the immediate future, these constraints and complications are facts of life; the agency is acting to meet its deadlines in as timely and meaningful a way as possible. Over the longer run, however, the committee hopes that experience with the practicalities of guidelines development will lead Congress and the agency to moderate the development and evaluation timetables or to expand the resources devoted to the tasks, or both. Third, within the government, meeting the challenge of developing good practice guidelines cannot be solely the responsibility of the Fo- rum. For instance, AHCPR's Medical Treatment Effectiveness Program (MEDIEP) will generate information of immediate importance for prac- tice guidelines. Lacunae in data identified during the guidelines develop- ment process should highlight areas that AHCPR can target for research funding. Outside AHCPR, the work of other agencies in the Public Health Service (PHS), most notably NIH's randomized controlled trials, will be essential to the long-term utility of guidelines, especially insofar as those teals include broad measures of outcomes important to patients. Outside the PHS, the agency needs to maintain close links with the Health Care Financing Administration, in part because of provisions of OBRA 89 but more importantly because HCFAs data on the Medicare population (and, to a lesser extent, on the Medicaid population) should be valuable for developing, implementing, and evaluating guidelines. ROLES OF PUBLIC AND PRIVATE SECTORS Despite the focus of this study on advice to a federal agency, the com- mittee believes that the government's role in arranging for the development

OCR for page 1
8 CLINICAL PRACTICE GUIDELINES of practice guidelines may in the end be fairly modest. Indeed, the contem- poraneous efforts of many different organizations in the private sector may significantly outpace what this one agency can do. The predominance of the private sector should be even greater for guidelines implementation, where most initiative must rest with private organizations and individuals. Even when the government plays the principal role in funding and disseminating guidelines on certain topics or clinical conditions, these guidelines will be tailored or adjusted by providers, health plans, and others to reflect differ- ent patient populations, delivery settings, practitioner sells and attitudes, levels of resources, perceptions of risk, and other factors. The committee expects that the processes of guidelines development, Implementation, and evaluation will always need to be pursued by both the public and private sectors. RECOMMENDATIONS: DEFINITIONS If the Forum is to proceed confidently with its mission, it needs clear and broadly acceptable definitions of four key terms used in OBRA 89: (1) practice guidelines, (2) medical review criteria, (3) standards of quality, and (4) performance measures. Neither the final legislation nor preceding House or Senate bills offered definitions of these particular terms, and the literature on practice guidelines and related topics is characterized by significant diversity in common and professional usage. This report aims to provide definitions that are insofar as possible- parsimonious, clear, not tautological, consistent with customary professional and legislative usage, and socially and practically acceptable to important interests. The committee recommends that the agency work with the following definitions. PRACTICE GUIDELINES are systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances. MEDICAL REVIEW CRITERIA are systematically developed statements that can be used to assess the appropriateness of specific health care decisions, services, and outcomes. STANDARDS OF QUALITY are authoritative statements of (1) minimum levels of acceptable performance or results, (2) excellent levels of perfor- mance or results, or (3) the range of acceptable performance or results PERFORMANCE MEASURES (Provisional) are methods or instruments to estimate or monitor the extent to which the actions of a health care practitioner or provider conform to practice guidelines, medical review criteria, or standards of quality.

OCR for page 1
SUMMARY The committee recognizes that the definitions will not resolve all arguments over what these and related terms mean, but it does believe that these four statements will bring a degree of badly needed clarity and uniformity to the field. Moreover, these definitions can be used by the Forum and, indeed, have already been incorporated into its work One underlying premise highlighted by these definitions is that these four terms are not synonymous. Assisting physicians, nurses, other prac- titioners, and patients in making decisions (through practice guidelines) is not the same as evaluating practice (using medical review criteria, standards of quality, and performance measures). Therefore, although the definitions may evolve, it is important to underscore that the phrases and concepts are not equivalent and should not be used interchangeably. For various practical or technical reasons, some elements of a set of guidelines may have no corresponding review criteria or other evaluation tools. Not part of the committee's definition of practice guidelines, but central to its view of the field, is the precept that every set of guidelines should be accompanied by a statement of the strength of the scientific evidence and the expert judgment behind them and by projections of the relevant health and COSt outcomes. The committee has not tried to distinguish types or levels of practice guidelines (for example, Levels 1 or 2), although this type of discrimination may be useful. RECOMMENDATIONS: ATTRIBUTES OF GOOD GUIDELINES Developing practice guidelines is a challenging task that requires di- verse skills ranging from analysis of scientific evidence tO management of group decisionmaking to presentation of complex information in under- standable forms. Jo arrange for the development of guidelines by expert panels and contractors, the Forum must be able tO state its expectations for the process and then assess the soundness of the resulting products. (OBRA 89 calls this establishing "standards and criteria" for the process. 1b avoid confusion, this report substitutes the term attributes for the statutory language. As described at the end of this summary, the IOM is preparing in provisional form a practical assessment instrument for the agency.) This report distinguishes between the priorities for selecting particular targets for guidelines and the desirable attributes of guidelines. Priority setting is a crucial but separate task for which OBRA 89 provides guidance Drawing on its members' experience and expertise and the work of past IOM committees and other relevant organizations, the committee recommends that the agency use the following eight attributes, properties, or characteristics to instruct expert panels or contractors and to assess their products. The attributes focus on practice guidelines and not on medical review criteria and other tools for evaluating practice. The committee

OCR for page 1
10 CONICAL PRACTICE GUIDELINES expects this set of attributes to be tested, reassessed, and, if necessary, revised. VALIDITY: Practice guidelines are valid if, when followed, they lead to the health and cost outcomes projected for them, other things being equal A prospective assessment of validity will consider the projected health out- comes and costs of alternative courses of action, the relationship between the evidence and recommendations, the substance and quality of the sci- entific and clinical evidence cited, and the means used to evaluate the evidence. RELIABILIlY/REPRODUCIBILIlY: Practice guidelines are reliable and reproducible (1) if-given the same evidence and methods for guidelines development" another set of experts would produce essentially the same statements and (2) if given the same circumstances the guidelines are interpreted and applied consistently by practitioners or other appropriate parties. A prospective assessment of reliability may consider the results of independent external reviews and pretests of the guidelines. CLINICAL APPLICABILITY: Practice guidelines should be as inclusive of appropriately defined patient populations as scientific and clinical evidence and expert judgment permit, and they should e~nlicitiv.stnte the nnn~,l~tion~ to which statements apply. --a ~a---- ~~~~~ ~~ r-r~~~~~~ CLINICAL FLEXIBILITY: Practice guidelines should identify the specifi- cally known or generally expected exceptions to their recommendations. C=RI1Y: Practice guidelines should use unambiguous language, define terms precisely, and use logical, easy-to-follow modes of presentation. MULTIDISCIPLINARY PROCESS: Practice guidelines should be devel- oped by a process that includes participation by representatives of key affected groups. Participation may include seating on panels that develop guidelines, providing evidence and viewpoints to the panels, and reviewing draft guidelines. SCHEDULED REVIEW: Practice guidelines should include statements about when they should be reviewed to determine whether revisions are warranted, given new clinical evidence or changing professional consensus. DOCUMENTATION: The procedures followed in developing guidelines, the participants involved, the evidence used, the assumptions and rado- nales accepted, and the analytic methods employed should be meticulously documented and described.

OCR for page 1
SUMMARY 11 The themes that under these attributes are credibility and account- abili~. The link between a set of guidelines and the scientific evidence must be explicit, and scientific and clinical evidence should take precedence over expert judgment. Every set of guidelines should be accompanied by a clear statement of the strength of the relevant scientific evidence and expert judgment. Linen the empirical evidence has important limitations and ex- perts reach conclusions that are not consistent with the evidence, then the rationale for departing from the evidence, such as it is, should be carefully explained. When expert judgment proceeds ~ the absence of direct em- pirical evidence about a particular clinical practice, as is often necessary, the general scientific reasoning and normative principles supporting the judgments should be described. This report does not take a position on whether cost considerations should be explicitly factored into practice recommendations, although some committee members had strong views that they should be. However, doc- umentation of projected health outcomes and costs to the degree possible is important to help developers and users of guidelines better understand the implications of following or not following the guidelines. One of the committee's strongest recommendations is that the process of developing guidelines include participation by representatives of key affected groups and disciplines. Such participation increases the likelihood (1) that all relevant scientific evidence will be located and critically evalu- ated; (2) that practical problems with using the guidelines will be identified and addressed; and (3) that affected groups will see the guidelines as credible and will cooperate in implementing them. Participation by physi- aans, nurses, patients, and others can be achieved in several ways including membership on the development panel, testimony at public hearings, par- ticipation in focus groups, consultation during site visits, and provision of comments on draft guidelines. The stringency of the attributes, especially taken together, is well recognized. Realistically, neither existing guidelines nor those likely to be developed by the agency in the foreseeable future will "score well" on all eight properties simultaneously. Indeed, near-perfect scores may always lie in the realm of aspiration rather than attainment. Moreover, the process of developing, assessing, using, evaluating, and revising guidelines will be evolutionary. There is today no proven "right way" to conduct this endeavor, even if there clearly are some "better ways." In addition, a balance needs to be maintained between an ideal process and one that is feasible. Guidelines that satisfactorily reflect the eight attributes proposed here may not be products of an ideal process? but in the committee's view they will be defensible.

OCR for page 1
~2 Cr JNICAL PRACTICE GUIDEr rNES RECOMMENDATIONS: IMPLEMENTATION AND EVALUATION Although AHCPR and the Forum are responsible for implementing the government program for guidelines established in OBRA 89, the main work of implementing the guidelines themselves will be in the hands of physicians, nurses, health care administrators, and others. The agency, however, has important responsibilities for evaluating the impact of guidelines. The committee's discussions were limited by its charge and centered primarily on how the processes of implementation and evaluation can reinforce and extend the eight attributes of guidelines defined earlier. One committee recommendation is that the agency instruct its expert panels and contractors to keep implementation and evaluation in mind as they develop guidelines. The tension between extraordinarily detailed or sophisticated guidelines and those that can be translated into usable medical review criteria or patient education materials must be recognized and dealt with during the development process, not after the fact. Keeping implementation and evaluation in mind during guidelines development means, among other things, understanding the following. The credibility of the development process, the participants, and the scientific grounding of guidelines must be clear to intended users. A truly multidisciplinary approach to guidelines development will facilitate acceptance and use of guidelines by members of the groups represented and by other, secondary target groups. Guidelines should be specific, comprehensive, and flexible enough to be useful in the varied settings and circumstances of everyday medical practice and in the evolving programs to assess the appropriateness of care provided in these settings. Guidelines language, logic, and symbols should be easy to fol- low and unambiguous, so that movement from guidelines statements to educational tools, review criteria, or other instruments is unimpeded. The guidelines should specie what information about the clinical problem, the patient's circumstances and preferences, and the delivery setting should be recorded to permit later evaluation of the appropriateness of care Judged against criteria generated from the guidelines). The implementation of guidelines is a diffuse, difficult-to-track process that will depend on many factors besides the quality and credibility of the guidelines. Among those factors are (1) the funding for dissemination and other implementation activities; (2) the supports and incentives for the guidelines to be used by physicians, nurses, health plans, and others; (3) the accessibility, scope, accuracy, and timeliness of a variety of intra- and interorganizational information systems; and (4) the ability of mul- tiple parties to plan and execute the various steps needed to implement guidelines.

OCR for page 1
SUMMARY 13 Users of guidelines will vary in their objectives and circumstances, and strategies for meeting particular user objectives will differ in their cost- effectiveness and manageability. Thus, different objectives and resources may call for different choices among the formats for guidelines (that is, their physical layout and logic), different roles for the available dissemination media, and different kinds of administrative supports for users of guidelines. Organizations with more resources (for example, libraries, video centers, telephone hotlines, personal computers, and network information systems) will be able to assist the use of guidelines in ways that are out of reach for less resource-rich organizations. Medical review criteria and other evaluation instruments, if properly developed and sensitively applied, can create incentives for adherence to practice guidelines. If improperly developed and applied, they can undermine support for practice guidelines. Building on earlier IOM reports on utilization management and quality assurance, the committee discussed a few broad principles for the constructive use of medical review criteria and other evaluation tools derived from practice guidelines. First, review criteria should be public with respect to their content and their development process. Second, when criteria are used to assess quality of care, deny payment for specific services, or take similar steps, an appeals process must be provided that is free from unreasonable complexity, delay, or other barriers. Third, review organizations should provide constructive information and feedback to physicians and other clinicians with the aim of improving practice rather than punishing missteps. Fourth, review organizations should make their renew activities as manageable and nonintrusive as possible. Such organizations contribute to the perceived and real "hassle factor'' in medical care, which grows out of burgeoning demands by payers and others for more information on, and justification for, health services delivered or proposed. The Forum needs to be sensitive to this issue. It should also work with HCFA and other organizations to minimize negative effects from poor translation of otherwise good guidelines into review criteria, unduly stringent application of these criteria, or both. The committee recognizes the critical importance of strong systems support for implementation. The complexities of creating such support within and across organizations are beyond the SGOpe of this report but will be an important issue in further IOM work on practice guidelines. With respect to the evaluation responsibilities of AHCPR and the Forum, the committee believes that the OBRA 89 provisions for evaluation are laudable but that the timetable is unrealistic. The Secretary of Health and Human Services is due to report on the impact of the first three guidelines by January 1, 1993. It is unlikely that measurable effects of these guidelines on the costs or quality of care will appear that quickly, and

OCR for page 1
~4 CLINICAL PRACTICE GUIDE~N:ES even if they do, it is unlikely that appropriate data on patient outcomes and program costs will be available and analyzed. Instead of a full-fledged evaluation, the agency can more reasonably be expected to provide a report on its evaluation plan, the steps being taken to implement the plan, and any preliminary evidence of impact. DIVERSITY IN CLINICAL PRACTICES AND GUII)ELINES In its discussions, the committee repeatedly returned to questions of diversity in clinical practice and inconsistency among guidelines. Diversity in clinical practice can be acceptable or unacceptable. It may be reasonable when the scientific evidence to support different courses of care is uncertain. In addition, some degree of diversity may be warranted by differences in individual patient characteristics and preferences and variations in delivery system capacities related to locale, resources, and patient populations. However, even though practice variation based on scientific uncertainty or differences in values may be acceptable, both science and values are open to change. Thus, what is perceived as acceptable diversity in clinical practice may change over time. Diversity in practice is unacceptable when it stems from poor practi- tioner skills, poor management of delivery systems, ignorance, or deliberate disregard of well-documented preferable practices. It should not be toler- ated when it is a self-serving disguise for bad practices that harm people or waste scarce resources. Guidelines can clarify what is acceptable and unacceptable variation in clinical practice, but that clarification itself has limits that may lead different groups to different and even inconsistent guidelines. Weak evidence is still weak evidence, although the processes described in Chapter 3 should allow the best use of whatever evidence is available. Nonetheless, these processes still leave room for differences of expert option about such issues as whether a flaw in research design "matters" or whether differences in results between two treatment alternatives are "clinically important" or only "statistically significant." Inconsistency among guidelines can also arise from variations in values and tolerance for rise People may simply differ in how they perceive different health outcomes and how they judge when benefits outweigh harms enough to make a seIvice worth providing. One way to approach this kind of variation is to try to establish practitioner and patient attitudes toward different benefits and harms and then identify what is known about the probabilities of those different outcomes. In some cases, the developers of guidelines may take the further step of applying their own values, but others considering the guidelines later could look at the same information and perhaps come to different conclusions. Also, for some services and clinical

OCR for page 1
SUMMARY 15 conditions, the developers of guidelines may choose not to recommend a single course of action but to lay out alternative courses of treatment that may be appropriate depending on, for example, the preferences of a patient or the characteristics of a delivery setting or community. In sum, merely identifying inconsistencies in guidelines says nothing about the legitimacy of those inconsistencies. Some inconsistencies may arise from biased or inept development processes. Some may result from reasonable differences in the interpretation of scientific evidence or in the application of patient, practitioner, or social values. Other inconsistencies may essentially disappear when the rationales for specific recommendations are closely examined. The challenge is to determine which explanation applies. Meticulous documentation of the evidence and rationales for guidelines will make this dete~ination easier. EXPECTATIONS FOR PRACTICE GUIDELINES Today the field of guidelines development is a confusing mix of high expectations, competing organizations, conflicting philosophies, and ill- defined or incompatible objectives. It suffers from imperfect and incom- plete scientific knowledge as well as imperfect and uneven means of applying that knowledge. Despite the good intentions of many involved parties, the enterprise lacks clearly articulated goals, coherent structures, and credible mechanisms for evaluating, improving, and coordinating guidelines devel- opment to meet social needs for good-quality, affordable health care. This situation will not change overnight, even though many promising activities, including those sponsored by AHCPR, are under way. Thus, expectations of quick results should be restrained. Otherwise, dashed hopes may lead to calls for premature abandonment of a useful strategy for improving the appropriate use of health services and to the adoption of more draconian measures to control costs.) The committee is also concerned about other expectations or assump- tions that may be unrealistic. One such assumption is that guidelines development is a relatively simple or straightforward undertaldng. It is not. For many clinical conditions and services, the science base is limited. Methods for analyzing evidence and developing expert opinions vary, but none of the rigorous methods can be properly applied by novices. Even in cases where considerable research has been done and sound methods 1 In this vein, see the 1973 book by political scientists Jeered L Pressman and Aaron B. W~l- davsly, Implementation: How Great ~ectanons ~ Washington Are Dashed in Oakland,. Or, Why It's A~na~ng that Federal Programs Work at All, Thus Being a Saga of Die Economic Development A~nmistranon as Told by Two Syrnpathenc Observers Who Seek to Build Morals on a Faundanon of Rained Hopes (Berkeley, Calif.: University of California Press).

OCR for page 1
16 CLINICAL PRACTICE GUIDELINES applied tO analyze it, honest clinicians and analysts may Still come to dif- ferent conclusions from the same evidence. Agreement on facts may not be matched by agreement on what health benefits are desirable at which economic cost with what tolerable accompanying health risks. Such conflicts about the interpretation of evidence and application of value judgments cannot be ignored. Indeed, the whole process of guidelines development has to be undertaken with great care at every stage: selecting participants, clarifying biases, adopting procedures and methods, identifying and analyzing evidence, considering alternatives, providing for independent reviews, preparing clear recommendations, and disclosing all important information about the process. Most of this report reinforces these points. A second assumption of concern to the committee is that there is one right way to develop guidelines. There is not. Vanations in the topics, the clinical disciplines involved, the purposes, and the audiences for guidelines will jUSti~ some differences ~ the specific methods for developing guidelines. However, tO grant some methodological diversity is not tO accept all approaches as equally good. Much remains to be tried and learned about the strengths and weaknesses of different methods. A further questionable expectation, which is sometimes explicit but often unstated, is that practice guidelines will help control health care costs. They may not. The reasons for caution on this point are several. For instance, variation in practice does not, by itself, demonstrate that high-use patterns are necessarily the inappropriate ones. Moreover, even if high use can be identified as inappropnate, such identification does not automatically change behavior. An array of incentives for behavior change may be tried, but not all will succeed. Even if behavior changes, expenditures may not. Some guidelines undoubtedly will save money by reducing the use of inappropriate services; some will increase costs by encouraging more use of underutilized services; and some will shift COStS from one service or place or payer tO another. The net impact on costs cannot be predicted with confidence, even if the priorities for guidelines development focus on clinical conditions for which overuse of expensive services is suspected. Nevertheless, if guidelines do succeed in improving the appropriateness and hence the vale of this country's expenditures for medical care, then the endeavor will be a success. This committee believes that AHCPR's practice guidelines effort has real potential to advance the state of the art in this field. The conditions for such success are demanding but not out of reach. In particular, expecta- tions for the agency and for practice guidelines per se- must be realistic regarding timetables and results. All parties concerned must act in good faith and keep the credibility and accountability of their actions in mind. Strict regard for the scientific rigor of the process is critical as is avoid- ance of premature closure on a single method of guidelines development.

OCR for page 1
SUM~4RY 17 Attention tO implementation and evaluation needs tO be factored into the development process at an early stage. The Forum can underscore itS intent tO examine critically and improve its program and products in at least three ways. First, it should ask its expert panels for feedback on the strengths and weaknesses of the procedures followed. Second, it should pretest (or arrange for the pretesting of) all guidelines developed under its aegis. This can be done on a pilot basis in a real delivery setting, on a set of prototypical cases, or through both methods. Third, it should try to evaluate the effectiveness of intermediate actions (for example, formatting, dissemination, incentives) that are necessary if guidelines are to have their intended effects on health practices, outcomes, and costs. Each of these steps can be part of a learning process for the Forum and others. NEXT STEPS FOR TlIE INSTITUTE OF MEDICINE In May 1990, a new IOM committee began an 18-month study of the development, implementation, evaluation, and revision of clinical practice guidelines. Many of the issues raised in this report will be examined in depth during this second project, which is supported by the John A. Hartford Foundation, Inc., and the Public Health Service. In preparing its report and recommendations, the new committee will . describe existing initiatives to develop, implement, and evaluate practice guidelines; identify the strengths and limitations of these efforts in light of the objectives and concerns of specific interest groups and society in general; describe different models of public and private action that might serve as prototypes for better structuring activities related to guidelines; analyze and assess the strengths, weaknesses, uncertainties, and trade-offs of different models in responding to identified problems and objectives; arid propose a framework for better structuring the development, im- plementation, evaluation, and revision of practice guidelines. . . In addition, the new committee will propose a practical methodology for AH:CPR and others tO employ in assessing guidelines before recom- mending or using them. It will focus on how the guidelines were developed, their scientific basis, their relevance to clinical practice, their Claris, and other characteristics. Such initial assessments will not substitute for later evaluations by government and others of the impact of a set of guidelines. The committee's recommendations will cover both government and private activities, and its report will identify legislative, management, and

OCR for page 1
18 CQ1VIG4L PRACTICE GUIDELINES other steps necessary lo implement the recommendations. An active pro- gram of disseminating the committee's findings and recommendation planned. The committee report should be released in the fall of 1991. FINAL COMMENT Fulfilling the mission and potential of the Forum will require heroic effort from a small staff, serious commitment from participants in the expert panels, and honest and practical support from the many involved and interested parties. Serious conceptual and practical issues remain to be confronted and resolved on a tight timetable with a modest budget. The undertaking will be strengthened if expectations are neither naively optimistic nor cynically pessimistic. This report suggests some of the ways in which the agency with the help of many others and a stance of constructive realism-can move to meet its mandate.