Click for next page ( 97


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 96
5 Conclusions and Recommendations We would consider our effort a success if more people began with the understanding that implementation, under the best of circumstances, is exceedingly difficult. They would, therefore, be pleasantly surprised when a few good things really happened. Pressman and W~ldavsly, Implementation The challenge facing AHCPR and its Forum in the area of practice guidelines is great, calling as it does for swift movement along many paths in territory that is neither well charted methodologically nor well established culturally. The challenge is intensified by the initiative's origins in the acute public and private frustration about ever-increasing health care spending for services of sometimes doubtful value. Expectations for the guidelines initiative are thus very high. In the context of current activities and hopes for practice guidelines, the basic charge to this committee was narrow: to define terms, to propose attributes of guidelines, and to explore in a preliminary way issues relating to implementation and evaluation. AHCPR needed the committee's recom- mendations quickly, and the project's short timetable limited its scope and depth. The statements singled out in this chapter as findings, conclusions, and recommendations are those that relate most directly to the committee's charge. Many of the broader issues related to the development, use, and evaluation of guidelines are being examined in a longer IOM project (see Appendix C). 96

OCR for page 96
CONCLUSIONS AND RECOMMENDATIONS 97 Throughout this report, the committee has sought to emphasize basic principles that should guide the development, use, and evaluation of guide- lines. In most case, it has not proposed specific techniques. The choice of techniques for instance, how to analyze scientific literature or how tO provide incentives for the use of guidelines will depend on practical circumstances tOO numerous tO cover comprehensively in this document. This chapter summarizes the committee's major findings and conclu- sions, reviews its recommendations about definitions of terms and attributes of guidelines, and outlines some of the complexities in implementing and evaluating guidelines that need tO be kept in mind as guidelines are de- veloped. The chapter closes with some observations about diversity and conflict among guidelines and the challenges facing the new agency. FINDINGS AND CONCLUSIONS STATE OF THE ART The committee arrived at several general observations about the state of the art of practice guidelines development. Most generally, the pro- cess of systematic development, implementation, and evaluation of practice guidelines based on rigorous clinical research and soundly generated profes- sional consensus, although progressing, has deficiencies in method, scope, and substance. Conflicts in terminology and technique characterize the field and are notable not just for the confusion they create but also for what they reflect about differences in values, experiences, and interests among different parties. Public and private activities are multiplying, but the means for coordinating these efforts to resolve inconsistencies, fill in gaps, track applications and results, and assess the soundness of particular guidelines are limited. Disproportionately more attention continues to be paid to the development of guidelines than to their implementation or evalua- tion. Moreover, efforts to develop guidelines are necessarily constrained by inadequacies in the quality and quantity of scientific evidence on the effectiveness of many services. ATICPR AND THE FORUM As a consequence of the above factors, AHCPR and the Forum have, at present, a somewhat restricted foundation for their work. In addition, other variables must be taken into account in estimating what the agency is likely tO be able to accomplish early in itS guidelines effort. Among the more important are the following. First, although OBRA 89 addresses some concerns about guidelines development, implementation, and evaluation, it appropriately does not

OCR for page 96
98 Cr JNICAL PRACTICE GUIDELINES describe a precise course of action for the agency. The committee expects that the agency will need time to devise and revise practical, defensible approaches to guidelines development. Second, given that both the function and the organizational units (par- ticularly the Forum) are new to the Department of Health and Human Services, the legislative timetables for guidelines development and, partic- ularly, evaluation are unrealistically short. Moreover, the Forum has few staff to support the new [unctions, and this is not likely to change in the near term. In the Immediate future, these constraints and complications are facts of life; the agency is acting to meet its deadlines in as timely and meaningful a way as possible. Over the longer run, however, the committee hopes that experience with the practicalities of guidelines development will lead Congress and the agency to moderate the development and evaluation timetables or to expand the resources devoted to the tasks, or both. Third, within the government, meeting the challenge of developing good practice guidelines cannot be solely the responsibility of the Forum. Other parts of AHCPR, for instance, its Medical Treatment Effectiveness Program (MEDTEP), will surely generate information of immediate im- portance for practice guidelines. Moreover, the exchange of information among units of government is a two-way process; lacunae in data identi- fied during the guidelines development process should highlight areas that AHCPR can target for research funding. Outside AHCPR, the work of other agencies in the Public Health Service, most notably the National Institutes of Health's randomized controlled trials, will be essential to the long-term utility of guidelines, especially insofar as those trials include broad measures of outcomes important to patients. The agency also needs to maintain close links with HCFA, in part because of certain provisions of OBRA 89 but more importantly because HCFA has data on the Medicare population (and, to a lesser extent, on the Medicaid population) that should be of value in developing, implementing, and evaluating guidelines. ROLES OF PUBLIC AND PRIVATE SECTORS Despite the focus of this study on advice to a federal agency, the com- mittee believes that the government's role in arranging for the development of practice guidelines may in the end be fairly modest. I-he contempora- neous efforts of many different organizations in the private sector may significantly outpace what AHCPR can do. This should be even more true for guidelines implementation, where most initiative must rest with private organizations and individuals. Even when the government does play the principal role in funding and disseminating guidelines on certain topics or clinical conditions, guidelines will be tailored or adjusted by providers, health plans, and others to reflect different patient populations, delivery

OCR for page 96
CONCLUSIONS AND RECOMMENDATIONS 99 settings, practitioner shills and attitudes, levels of resources, perceptions of risks, and other factors. The committee expects that the processes of guidelines development, implementation, and evaluation will always need to be pursued by both the public and private sectors. RECOMMENDATIONS: DEFINITIONS The committee sought to formulate definitions that, insofar as possible, would be clear, concise, and not tautological; consistent with professional and legislative usage; and practically and symbolically acceptable to im- portant interests. It recommends that the Forum work with the following definitions of the four key terms used in OBRA 89. PRACTICE GUIDELINES are systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances. MEDICAL REVIEW CRITERIA are systematically developed statements that can be used tO assess the appropriateness of specific health care decisions, services, and outcomes. STANDARDS OF QUALITY are authoritative statements of (1) minimum levels of acceptable performance or results, (2) excellent levels of perfor- mance or results, or (3) the range of acceptable performance or results. PERFORMANCE MEASURES (Provisional) are methods or instruments to estimate or monitor the extent to which the actions of a health care practitioner or provider conform to practice guidelines, medical review criteria, or standards of quality. The committee recognizes that these definitions will not resolve all arguments over what these and related terms mean, but it believes these four statements will bring a degree of badly needed clarity and uniformity to the field. Moreover, these definitions can be used by the Forum and, indeed, have already been incorporated into its work. One underlying premise highlighted by these definitions is that these four terms are not synonymous. Assistance to physicians and patients in making decisions is not the same as tOOk for evaluating practice. Therefore, although the definitions may evolve, it is important to underscore that these are not equivalent concepts or phrases and should not be used interchangeably. Not part of the committee's definition of practice guidelines but central tO itS view of the field is the precept that every guideline should be accompanied by a statement of the strength of the evidence and the expert judgment behind it. Lee committee has not tried to distinguish types or

OCR for page 96
100 Cr rNICAL PRACTICE GUIDELINES levels of practice guidelines (for example, Level 1 or Level 2 guidelines), although this may be useful. For now, the committee recommends that every set of guidelines describe the strength of the evidence and consensus so that potential users can make more informed decisions. RECOMMENDATIONS: ATTRIBUTES OF GOOD GUIDELINES In proposing attributes of good guidelines, the committee tried to define properties or characteristics that would be compatible with profes- sional use and legislative expectations; clearly defined and justified; and demanding but realistically approachable. Prospective assessments of the guidelines (rather than evaluations of them ultimate impact) should use the attributes as benchmarks for judging the soundness of the guidelines. The focus should be on guidelines as a set rather than as isolated statements. Creating a practical assessment instrument for AHCPR, based on the attributes below, is one task of a second IOM project on practice guidelines (Appendix C). In this first project, the committee has tried to be sensitive to the challenges involved in moving from abstract concepts to real applications. The committee recommends that the agency look for the following eight attributes, properties, or characteristics when it assesses the soundness of guidelines. Relatedly, the agency should instruct its contractors and expert panels lo adhere to procedures that will produce guidelines in keeping with these attributes. VALIDII Y: Practice guidelines are valid if, when followed, they lead to the health and cost outcomes projected for them, other things being equal. A prospective assessment of validity will consider the projected health out- comes and costs of alternative courses of action, the relationship between the evidence and recommendations, the substance and quality of the sci- entific and clinical evidence cited, and the means used to evaluate the evidence. RELL\BILITY/REPRODUCIBILITY: Practice guidelines are reliable and reproducible (1) if given the same evidence and methods for guidelines development another set of experts would produce essentially the same statements and (2) if given the same circumstances the guidelines are interpreted and applied consistently by practitioners or other appropriate parties. A prospective assessment of reliability may consider the results of independent external reviews and pretests of the guidelines. CLINICAL APPLICABILITY: Practice guidelines should be as inclusive of appropriately defined patient populations as scientific and clinical evidence

OCR for page 96
CONCLUSIONS AND RECOMMENDATIONS 101 and expert judgment permit, and they should explicitly state the populations lo which statements apply. CLINICAL FLEXIBILITY: Practice guidelines should identify the specifi- cally known or generally expected exceptions to their recommendations. CLARITY: Practice guidelines should use unambiguous language, define terms precisely, and use logical, easy-to-follow modes of presentation. MULTIDISCIPLINARY PROCESS: Practice guidelines should be devel- oped by a process that includes participation by representatives of key affected groups. Participation may include solving on panels that develop guidelines, providing evidence and viewpoints to the panels, and reviewing draft guidelines. SCHEDULED REVIEW: Practice guidelines should include statements about when they should be reviewed to determine whether revisions are warranted, given new clinical evidence or changing professional consensus. DOCUMENTATION: The procedures followed in developing guidelines, the participants involved, the evidence used, the assumptions and raho- nales accepted, and the analytic methods employed should be meticulously documented and descnbed. The stringency of these attributes, especially taken together, is well recognized. Realistically, neither existing guidelines nor those likely to be developed by the agency in the foreseeable future will "score well" on all eight properties simultaneously; indeed, near-perfect scores may always lie in the realm of aspiration rather than attainment. Nevertheless, the committee wishes to emphasize the importance of worldng toward practice guidelines that exhibit these characteristics. _ , , RECOMMENDATIONS: IMPLEMENTATION AND EVALUATION The main work of disseminating and applying guidelines will be in private hands, but as OBRA 89 recognizes, the agency cannot and should not be divorced from implementation. Evaluation will be a major agency concern. The committee's discussions centered on how the processes of implementation and evaluation can reinforce and extend the eight attributes defined above. The Forum should instruct its expert panels and contractors to keep the challenges and difficulties of implementation and evaluation in mind as they develop guidelines. For instance, the tension between extraordinarily detailed, complex, or sophisticated guidelines and those that can be readily translated into medical review criteria, or into documents understandable by the average patient, has to be recognized and dealt with dunng the

OCR for page 96
102 CLINICAL PRACTICE GUIDELINES development process, not after the fact. Keeping implementation and evaluation in mind means, among other things, the following. The credibility of the development process, the participants, and the scientific grounding of guidelines must be clear to intended users. A truly multidisciplinary approach to guidelines development will facilitate acceptance and use of guidelines by members of the groups represented and by other, secondary target groups. Guidelines should be specific, comprehensive, and flexible enough to be useful in the varied settings and circumstances of everyday medical practice and in the evolving programs to assess the appropriateness of care provided in these settings. Guidelines language, logic, and symbols should be easy to fol- low and unambiguous, so that movement from guidelines statements to educational tools, review criteria, or other instruments is unimpeded. The guidelines should specie what information about the clinical problem, the patient's circumstances and preferences, and the delivery setting should be recorded to permit later evaluation of the appropriateness of care Judged against criteria generated from the guidelines). The Forum can underscore its intent to examine critically and improve its processes and products In at least three ways. First, it should ask its expert panels for feedback on the strengths and weaknesses of the proce- dures followed. Second, the Forum should pretest the guidelines either on a pilot basis in a real delivery setting or on a set of prototypical cases, or using both methods. Third, it should try to evaluate the effectiveness of intermediate steps (for example, formatting, disseminations that are nec- essary if guidelines are to have their intended effects on health practices, outcomes, and costs. Each of these steps can be part of a learning process for the Forum and for other interested parties. Attention to implementation and evaluation must start during the pro- cess of guidelines development; these aspects of the guidelines effort should not be dealt with sequentially. Moreover, many parties must contribute to these efforts, and the processes recommended and adopted over the years will evolve as sponsors and participants become more experienced. Today there is no "one best way" to develop guidelines, although preferable-and unacceptable approaches can be identified. Finally, the effectiveness of any guidelines in reaching desired COSt and quality goals cannot be assumed. Useful evaluations of impact will take a real commitment of resources. This report distinguishes between the implementation of the guidelines program called for in OBRA 89 and the implementation of the guidelines themselves. AHCPR and its Forum have the primely responsibility for the former, but the latter will depend largely on private individuals and organizations and will inevitably be harder to coordinate, monitor, and

OCR for page 96
CONCLUSIONS AND RECOMMENDATIONS 103 evaluate. The committee's discussion of implementation in Chapter 4 fo- cuses primarily on issues involved in guidelines implementation. However, the discussion of definitions and attributes in Chapters 2 and 3 speaks to two aspects of program implementation: the committee's work with expert panels and its assessment of the guidelines they produce. The committee also distinguishes between two kinds of evaluation. One is the evaluation of medical practice. Such evaluations may employ clinical trials and other outcomes-oriented methods, or they may rely on medical review criteria and similar tools that compare actual practice with recommended practice. The other kind of evaluation, considered in Chapter 4, asks about the Impact of guidelines on the quality, effectiveness, and cost of care. The implementation of guidelines is a diffuse, difficult-to-track process that will depend on many factors besides the quality and credibility of the guidelines. These factors include the resources devoted to the different steps in implementation, the incentives for use of the guidelines, and the availability of supportive data systems. Different users will have different objectives, and strategies for meeting particular objectives will vary in their cost-effectiveness and manageability for the parties involved. That is, different objectives may call for different choices among (1) formats (physical layout and logic), (2) dissemination media, and (3) administrative supports for users of guidelines in the form of computer-based information systems, periodic feedback, or standard operating procedures. The committee also discussed a few broad policy issues related to the use of medical review criteria and other evaluation tools derived from practice guidelines. First, consistent with the recommendations of the 1989 IOM report on private-sector utilization management and observations of the 1990 IOM Medicare quality assurance report, review criteria should be public with respect to their content and their development process. This does not require that subsequently developed software and other administrative tools be public, although public monies should not be used to generate, in the first instance, proprietary products. Second, when criteria are used to assess quality of care, deny payment for specific services, or take similar steps, an appeals process must be pro- vided. This process must be clearly described to patients and practitioners and be free from unreasonable complexity, delay, or other barriers. The attributes identified by this committee- particularly those of clinical appli- cability and flexibility combined may help reduce the need for appeals. The point is that individual, idiosyncratic cases will surely surface, no matter how complex and comprehensive the guidelines are, and these eventualities can best be met with an appeals process. A third issue relating to implementation of medical review programs is the burden on patients and practitioners. Review organizations should

OCR for page 96
104 CLINICAL PRACTICE GUIDELINES make their review activities as manageable and nonintrusive as possible for all those affected by the guidelines. Such organizations are one contributor to the perceived and real "hassle factor" in medical care, which grows out of burgeoning demands by payers and others for more information and justification for services delivered or proposed. The Forum needs to be sensitive to this issue during the guidelines development phase; it also needs to work with HCFA and other organizations to minimize negative effects from poor translation of otherwise good guidelines into review criteria, from unduly stnugent application of such criteria, or both. As noted previously, early involvement of provider groups and respected representatives of review organizations, such as the Medicare PROs, is desirable as one means of enhancing the later manageability of guidelines. Fourth, review organizations should, insofar as possible, provide con- structive information and feedback to physicians and other clinicians. This material, and its mode of presentation, should aim to improve practice rather than punish missteps, a view consistent with the overall strategy proposed by the IOM in its Medicare quality assurance study (1990~. Re- gardless of their form, guidelines will not be success) al if they are perceived (correctly or not) as vehicles solely for the external control of an obdurate professional community, and the Forum needs to be sensitive to this point. With respect to evaluation, the study committee believes that the OBRA 89 timetable is unrealistic. The Secretary of Health and Human Services is due to report on the impact of the first three guidelines by January 1, 1993. It is unlikely that these guidelines would have measurable effects on health care or costs that quickly, and even if they did, it is unlikely that appropriate data on patient outcomes and program costs would be available and analyzed. Instead of a full-fledged evaluation, the agency can more reasonably be expected to provide a report on its evaluation plan, the steps being taken to implement the plan, and any preliminary evidence of impact. DIVERSITY IN CLINICAL PRACTICES AND GUIDELINES In in discussions, the committee repeatedly returned to questions of diversity in clinical practice and inconsistency among guidelines. Diversity in clinical practice can be acceptable or unacceptable. It may be reasonable when the scientific evidence to support different courses of care is uncertain. In addition, some degree of diversity may be warranted by differences in individual patient characteristics and preferences and variations in delivery system capacities related to locale, resources, and patient populations. However, even though practice variation based on scientific uncertainty or differences in values may be acceptable, both science and values are

OCR for page 96
CONCLUSIONS AND RECOMMENDATIONS 105 open tO change. Thus, what is perceived as acceptable diversity in clinical practice may change over tine. Diversity in practice is unacceptable when it stems from poor practi- tioner skills, poor management of delivery systems, ignorance, or deliberate disregard of well-documented preferable practices. It should not be toler- ated when it iS a self-se~ving disguise for bad practices that harm people or waste scarce resources. Guidelines can clarify what is acceptable and unacceptable variation in clinical practice, but that clarification itself has limits that may lead different groups to different and even inconsistent guidelines. Weak evidence is still weak evidence, although the processes described in Chapter 3 should allow the best use of whatever evidence is available. For example, the more formal methods of analyzing and characterizing evidence can reduce the opportunity for inconsistency arising from poor analysis of evidence. Nonetheless, these methods can still leave room for differences of expert opinion about such issues as whether a flaw in research design "matters" or whether differences in results between two treatment alternatives are "clinically important" or only "statistically significant." Inconsistengy among guidelines can also arise from variations in val- ues and tolerance of nsL People may simply differ in how they perceive different health outcomes and how they judge when benefits enough out- weigh harms to make a service worth providing. One way to approach this kind of variation is for guidelines developers to try to establish practitioner and patient attitudes toward different benefits and harms and then identity what is known about the probabilities of those different outcomes. In some cases, the developers of guidelines take the further step of applying their own values, but others considering the guidelines later might look at the same information and perhaps come tO different conclusions. Also, for some services and clinical conditions, the developers of guidelines may choose not to recommend one course of action but to lay out alternative courses of treatment that may be appropriate, depending on, for example, the preferences of a patient or the characteristics of a delivery setting or community. In sum, merely identifying inconsistencies in guidelines says nothing about the legitimacy of those inconsistencies. Some inconsistencies may arise from biased or inept development processes. Some may result from reasonable differences in the interpretation of scientific evidence or in the application of patient, practitioner, or social values. Other inconsistencies may essentially disappear when the rationales for specific recommendations are closely examined. The challenge is to determine which explanation applies. Meticulous documentation of the evidence and rationales for guidelines will make this determination easier.

OCR for page 96
106 CLINICAL PRACTICE GUIDELINES CONCLUDING COMMENTS As stated at the outset of this report, this committee believes that AHCPR's practice guidelines effort has real potential to advance the state of the art in this field, strengthen the knowledge base for health care practice, and, hence, improve the appropriateness and effectiveness of health care. The conditions for such success are demanding but not out of reach. In particular, expectations for the agency and for practice guidelines per se-must be realistic regarding timetables and results. All parties concerned must act in good faith and keep the credibility and accountability of their actions in mind. Strict attention to the scientific rigor of the process is critical as is avoidance of premature closure on a single method of guidelines development. Attention to implementation and evaluation needs to be factored into the development process at an early stage. Fulfilling the agencies mission and potential will require heroic effort from a small staff, serious commitment from participants in the expert panels, and honest, practical support from the many involved and interested parties. Many conceptual and practical issues remain to be confronted. The undertaking will be strengthened if expectations are sturdily realistic, that is, neither too naively optimistic nor too cynically pessimistic.