Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 96
5
Conclusions and Recommendations
We would consider our effort a success if more people began with the
understanding that implementation, under the best of circumstances,
is exceedingly difficult. They would, therefore, be pleasantly surprised
when a few good things really happened.
Pressman and W~ldavsly, Implementation
The challenge facing AHCPR and its Forum in the area of practice
guidelines is great, calling as it does for swift movement along many paths in
territory that is neither well charted methodologically nor well established
culturally. The challenge is intensified by the initiative's origins in the acute
public and private frustration about ever-increasing health care spending
for services of sometimes doubtful value. Expectations for the guidelines
initiative are thus very high.
In the context of current activities and hopes for practice guidelines,
the basic charge to this committee was narrow: to define terms, to propose
attributes of guidelines, and to explore in a preliminary way issues relating
to implementation and evaluation. AHCPR needed the committee's recom-
mendations quickly, and the project's short timetable limited its scope and
depth. The statements singled out in this chapter as findings, conclusions,
and recommendations are those that relate most directly to the committee's
charge. Many of the broader issues related to the development, use, and
evaluation of guidelines are being examined in a longer IOM project (see
Appendix C).
96
OCR for page 97
CONCLUSIONS AND RECOMMENDATIONS
97
Throughout this report, the committee has sought to emphasize basic
principles that should guide the development, use, and evaluation of guide-
lines. In most case, it has not proposed specific techniques. The choice
of techniques for instance, how to analyze scientific literature or how
tO provide incentives for the use of guidelines will depend on practical
circumstances tOO numerous tO cover comprehensively in this document.
This chapter summarizes the committee's major findings and conclu-
sions, reviews its recommendations about definitions of terms and attributes
of guidelines, and outlines some of the complexities in implementing and
evaluating guidelines that need tO be kept in mind as guidelines are de-
veloped. The chapter closes with some observations about diversity and
conflict among guidelines and the challenges facing the new agency.
FINDINGS AND CONCLUSIONS
STATE OF THE ART
The committee arrived at several general observations about the state
of the art of practice guidelines development. Most generally, the pro-
cess of systematic development, implementation, and evaluation of practice
guidelines based on rigorous clinical research and soundly generated profes-
sional consensus, although progressing, has deficiencies in method, scope,
and substance. Conflicts in terminology and technique characterize the field
and are notable not just for the confusion they create but also for what
they reflect about differences in values, experiences, and interests among
different parties. Public and private activities are multiplying, but the means
for coordinating these efforts to resolve inconsistencies, fill in gaps, track
applications and results, and assess the soundness of particular guidelines
are limited. Disproportionately more attention continues to be paid to
the development of guidelines than to their implementation or evalua-
tion. Moreover, efforts to develop guidelines are necessarily constrained
by inadequacies in the quality and quantity of scientific evidence on the
effectiveness of many services.
ATICPR AND THE FORUM
As a consequence of the above factors, AHCPR and the Forum have,
at present, a somewhat restricted foundation for their work. In addition,
other variables must be taken into account in estimating what the agency
is likely tO be able to accomplish early in itS guidelines effort. Among the
more important are the following.
First, although OBRA 89 addresses some concerns about guidelines
development, implementation, and evaluation, it appropriately does not
OCR for page 98
98
Cr JNICAL PRACTICE GUIDELINES
describe a precise course of action for the agency. The committee expects
that the agency will need time to devise and revise practical, defensible
approaches to guidelines development.
Second, given that both the function and the organizational units (par-
ticularly the Forum) are new to the Department of Health and Human
Services, the legislative timetables for guidelines development and, partic-
ularly, evaluation are unrealistically short. Moreover, the Forum has few
staff to support the new [unctions, and this is not likely to change in the
near term. In the Immediate future, these constraints and complications
are facts of life; the agency is acting to meet its deadlines in as timely and
meaningful a way as possible. Over the longer run, however, the committee
hopes that experience with the practicalities of guidelines development will
lead Congress and the agency to moderate the development and evaluation
timetables or to expand the resources devoted to the tasks, or both.
Third, within the government, meeting the challenge of developing
good practice guidelines cannot be solely the responsibility of the Forum.
Other parts of AHCPR, for instance, its Medical Treatment Effectiveness
Program (MEDTEP), will surely generate information of immediate im-
portance for practice guidelines. Moreover, the exchange of information
among units of government is a two-way process; lacunae in data identi-
fied during the guidelines development process should highlight areas that
AHCPR can target for research funding. Outside AHCPR, the work of
other agencies in the Public Health Service, most notably the National
Institutes of Health's randomized controlled trials, will be essential to the
long-term utility of guidelines, especially insofar as those trials include
broad measures of outcomes important to patients. The agency also needs
to maintain close links with HCFA, in part because of certain provisions of
OBRA 89 but more importantly because HCFA has data on the Medicare
population (and, to a lesser extent, on the Medicaid population) that should
be of value in developing, implementing, and evaluating guidelines.
ROLES OF PUBLIC AND PRIVATE SECTORS
Despite the focus of this study on advice to a federal agency, the com-
mittee believes that the government's role in arranging for the development
of practice guidelines may in the end be fairly modest. I-he contempora-
neous efforts of many different organizations in the private sector may
significantly outpace what AHCPR can do. This should be even more true
for guidelines implementation, where most initiative must rest with private
organizations and individuals. Even when the government does play the
principal role in funding and disseminating guidelines on certain topics
or clinical conditions, guidelines will be tailored or adjusted by providers,
health plans, and others to reflect different patient populations, delivery
OCR for page 99
CONCLUSIONS AND RECOMMENDATIONS
99
settings, practitioner shills and attitudes, levels of resources, perceptions
of risks, and other factors. The committee expects that the processes of
guidelines development, implementation, and evaluation will always need
to be pursued by both the public and private sectors.
RECOMMENDATIONS: DEFINITIONS
The committee sought to formulate definitions that, insofar as possible,
would be clear, concise, and not tautological; consistent with professional
and legislative usage; and practically and symbolically acceptable to im-
portant interests. It recommends that the Forum work with the following
definitions of the four key terms used in OBRA 89.
PRACTICE GUIDELINES are systematically developed statements to assist
practitioner and patient decisions about appropriate health care for specific
clinical circumstances.
MEDICAL REVIEW CRITERIA are systematically developed statements
that can be used tO assess the appropriateness of specific health care
decisions, services, and outcomes.
STANDARDS OF QUALITY are authoritative statements of (1) minimum
levels of acceptable performance or results, (2) excellent levels of perfor-
mance or results, or (3) the range of acceptable performance or results.
PERFORMANCE MEASURES (Provisional) are methods or instruments
to estimate or monitor the extent to which the actions of a health care
practitioner or provider conform to practice guidelines, medical review
criteria, or standards of quality.
The committee recognizes that these definitions will not resolve all
arguments over what these and related terms mean, but it believes these
four statements will bring a degree of badly needed clarity and uniformity
to the field. Moreover, these definitions can be used by the Forum and,
indeed, have already been incorporated into its work.
One underlying premise highlighted by these definitions is that these
four terms are not synonymous. Assistance to physicians and patients in
making decisions is not the same as tOOk for evaluating practice. Therefore,
although the definitions may evolve, it is important to underscore that
these are not equivalent concepts or phrases and should not be used
interchangeably.
Not part of the committee's definition of practice guidelines but central
tO itS view of the field is the precept that every guideline should be
accompanied by a statement of the strength of the evidence and the expert
judgment behind it. Lee committee has not tried to distinguish types or
OCR for page 100
100
Cr rNICAL PRACTICE GUIDELINES
levels of practice guidelines (for example, Level 1 or Level 2 guidelines),
although this may be useful. For now, the committee recommends that
every set of guidelines describe the strength of the evidence and consensus
so that potential users can make more informed decisions.
RECOMMENDATIONS: ATTRIBUTES OF GOOD GUIDELINES
In proposing attributes of good guidelines, the committee tried to
define properties or characteristics that would be compatible with profes-
sional use and legislative expectations; clearly defined and justified; and
demanding but realistically approachable. Prospective assessments of the
guidelines (rather than evaluations of them ultimate impact) should use the
attributes as benchmarks for judging the soundness of the guidelines. The
focus should be on guidelines as a set rather than as isolated statements.
Creating a practical assessment instrument for AHCPR, based on
the attributes below, is one task of a second IOM project on practice
guidelines (Appendix C). In this first project, the committee has tried to
be sensitive to the challenges involved in moving from abstract concepts to
real applications.
The committee recommends that the agency look for the following
eight attributes, properties, or characteristics when it assesses the soundness
of guidelines. Relatedly, the agency should instruct its contractors and
expert panels lo adhere to procedures that will produce guidelines in
keeping with these attributes.
VALIDII Y: Practice guidelines are valid if, when followed, they lead to the
health and cost outcomes projected for them, other things being equal. A
prospective assessment of validity will consider the projected health out-
comes and costs of alternative courses of action, the relationship between
the evidence and recommendations, the substance and quality of the sci-
entific and clinical evidence cited, and the means used to evaluate the
evidence.
RELL\BILITY/REPRODUCIBILITY: Practice guidelines are reliable and
reproducible (1) if given the same evidence and methods for guidelines
development another set of experts would produce essentially the same
statements and (2) if given the same circumstances the guidelines are
interpreted and applied consistently by practitioners or other appropriate
parties. A prospective assessment of reliability may consider the results of
independent external reviews and pretests of the guidelines.
CLINICAL APPLICABILITY: Practice guidelines should be as inclusive of
appropriately defined patient populations as scientific and clinical evidence
OCR for page 101
CONCLUSIONS AND RECOMMENDATIONS
101
and expert judgment permit, and they should explicitly state the populations
lo which statements apply.
CLINICAL FLEXIBILITY: Practice guidelines should identify the specifi-
cally known or generally expected exceptions to their recommendations.
CLARITY: Practice guidelines should use unambiguous language, define
terms precisely, and use logical, easy-to-follow modes of presentation.
MULTIDISCIPLINARY PROCESS: Practice guidelines should be devel-
oped by a process that includes participation by representatives of key
affected groups. Participation may include solving on panels that develop
guidelines, providing evidence and viewpoints to the panels, and reviewing
draft guidelines.
SCHEDULED REVIEW: Practice guidelines should include statements
about when they should be reviewed to determine whether revisions are
warranted, given new clinical evidence or changing professional consensus.
DOCUMENTATION: The procedures followed in developing guidelines,
the participants involved, the evidence used, the assumptions and raho-
nales accepted, and the analytic methods employed should be meticulously
documented and descnbed.
The stringency of these attributes, especially taken together, is well
recognized. Realistically, neither existing guidelines nor those likely to be
developed by the agency in the foreseeable future will "score well" on
all eight properties simultaneously; indeed, near-perfect scores may always
lie in the realm of aspiration rather than attainment. Nevertheless, the
committee wishes to emphasize the importance of worldng toward practice
guidelines that exhibit these characteristics.
_ , ,
RECOMMENDATIONS: IMPLEMENTATION AND EVALUATION
The main work of disseminating and applying guidelines will be in
private hands, but as OBRA 89 recognizes, the agency cannot and should
not be divorced from implementation. Evaluation will be a major agency
concern. The committee's discussions centered on how the processes of
implementation and evaluation can reinforce and extend the eight attributes
defined above.
The Forum should instruct its expert panels and contractors to keep
the challenges and difficulties of implementation and evaluation in mind as
they develop guidelines. For instance, the tension between extraordinarily
detailed, complex, or sophisticated guidelines and those that can be readily
translated into medical review criteria, or into documents understandable
by the average patient, has to be recognized and dealt with dunng the
OCR for page 102
102
CLINICAL PRACTICE GUIDELINES
development process, not after the fact. Keeping implementation and
evaluation in mind means, among other things, the following.
The credibility of the development process, the participants, and
the scientific grounding of guidelines must be clear to intended users.
· A truly multidisciplinary approach to guidelines development will
facilitate acceptance and use of guidelines by members of the groups
represented and by other, secondary target groups.
· Guidelines should be specific, comprehensive, and flexible enough
to be useful in the varied settings and circumstances of everyday medical
practice and in the evolving programs to assess the appropriateness of care
provided in these settings.
Guidelines language, logic, and symbols should be easy to fol-
low and unambiguous, so that movement from guidelines statements to
educational tools, review criteria, or other instruments is unimpeded.
· The guidelines should specie what information about the clinical
problem, the patient's circumstances and preferences, and the delivery
setting should be recorded to permit later evaluation of the appropriateness
of care Judged against criteria generated from the guidelines).
The Forum can underscore its intent to examine critically and improve
its processes and products In at least three ways. First, it should ask its
expert panels for feedback on the strengths and weaknesses of the proce-
dures followed. Second, the Forum should pretest the guidelines either on
a pilot basis in a real delivery setting or on a set of prototypical cases, or
using both methods. Third, it should try to evaluate the effectiveness of
intermediate steps (for example, formatting, disseminations that are nec-
essary if guidelines are to have their intended effects on health practices,
outcomes, and costs. Each of these steps can be part of a learning process
for the Forum and for other interested parties.
Attention to implementation and evaluation must start during the pro-
cess of guidelines development; these aspects of the guidelines effort should
not be dealt with sequentially. Moreover, many parties must contribute to
these efforts, and the processes recommended and adopted over the years
will evolve as sponsors and participants become more experienced. Today
there is no "one best way" to develop guidelines, although preferable-and
unacceptable approaches can be identified. Finally, the effectiveness of
any guidelines in reaching desired COSt and quality goals cannot be assumed.
Useful evaluations of impact will take a real commitment of resources.
This report distinguishes between the implementation of the guidelines
program called for in OBRA 89 and the implementation of the guidelines
themselves. AHCPR and its Forum have the primely responsibility for
the former, but the latter will depend largely on private individuals and
organizations and will inevitably be harder to coordinate, monitor, and
OCR for page 103
CONCLUSIONS AND RECOMMENDATIONS
103
evaluate. The committee's discussion of implementation in Chapter 4 fo-
cuses primarily on issues involved in guidelines implementation. However,
the discussion of definitions and attributes in Chapters 2 and 3 speaks to
two aspects of program implementation: the committee's work with expert
panels and its assessment of the guidelines they produce.
The committee also distinguishes between two kinds of evaluation.
One is the evaluation of medical practice. Such evaluations may employ
clinical trials and other outcomes-oriented methods, or they may rely
on medical review criteria and similar tools that compare actual practice
with recommended practice. The other kind of evaluation, considered in
Chapter 4, asks about the Impact of guidelines on the quality, effectiveness,
and cost of care.
The implementation of guidelines is a diffuse, difficult-to-track process
that will depend on many factors besides the quality and credibility of the
guidelines. These factors include the resources devoted to the different
steps in implementation, the incentives for use of the guidelines, and the
availability of supportive data systems. Different users will have different
objectives, and strategies for meeting particular objectives will vary in
their cost-effectiveness and manageability for the parties involved. That
is, different objectives may call for different choices among (1) formats
(physical layout and logic), (2) dissemination media, and (3) administrative
supports for users of guidelines in the form of computer-based information
systems, periodic feedback, or standard operating procedures.
The committee also discussed a few broad policy issues related to
the use of medical review criteria and other evaluation tools derived from
practice guidelines. First, consistent with the recommendations of the 1989
IOM report on private-sector utilization management and observations of
the 1990 IOM Medicare quality assurance report, review criteria should
be public with respect to their content and their development process.
This does not require that subsequently developed software and other
administrative tools be public, although public monies should not be used
to generate, in the first instance, proprietary products.
Second, when criteria are used to assess quality of care, deny payment
for specific services, or take similar steps, an appeals process must be pro-
vided. This process must be clearly described to patients and practitioners
and be free from unreasonable complexity, delay, or other barriers. The
attributes identified by this committee- particularly those of clinical appli-
cability and flexibility combined may help reduce the need for appeals.
The point is that individual, idiosyncratic cases will surely surface, no matter
how complex and comprehensive the guidelines are, and these eventualities
can best be met with an appeals process.
A third issue relating to implementation of medical review programs
is the burden on patients and practitioners. Review organizations should
OCR for page 104
104
CLINICAL PRACTICE GUIDELINES
make their review activities as manageable and nonintrusive as possible for
all those affected by the guidelines. Such organizations are one contributor
to the perceived and real "hassle factor" in medical care, which grows
out of burgeoning demands by payers and others for more information
and justification for services delivered or proposed. The Forum needs
to be sensitive to this issue during the guidelines development phase;
it also needs to work with HCFA and other organizations to minimize
negative effects from poor translation of otherwise good guidelines into
review criteria, from unduly stnugent application of such criteria, or both.
As noted previously, early involvement of provider groups and respected
representatives of review organizations, such as the Medicare PROs, is
desirable as one means of enhancing the later manageability of guidelines.
Fourth, review organizations should, insofar as possible, provide con-
structive information and feedback to physicians and other clinicians. This
material, and its mode of presentation, should aim to improve practice
rather than punish missteps, a view consistent with the overall strategy
proposed by the IOM in its Medicare quality assurance study (1990~. Re-
gardless of their form, guidelines will not be success) al if they are perceived
(correctly or not) as vehicles solely for the external control of an obdurate
professional community, and the Forum needs to be sensitive to this point.
With respect to evaluation, the study committee believes that the
OBRA 89 timetable is unrealistic. The Secretary of Health and Human
Services is due to report on the impact of the first three guidelines by
January 1, 1993. It is unlikely that these guidelines would have measurable
effects on health care or costs that quickly, and even if they did, it is unlikely
that appropriate data on patient outcomes and program costs would be
available and analyzed. Instead of a full-fledged evaluation, the agency can
more reasonably be expected to provide a report on its evaluation plan, the
steps being taken to implement the plan, and any preliminary evidence of
impact.
DIVERSITY IN CLINICAL PRACTICES AND GUIDELINES
In in discussions, the committee repeatedly returned to questions of
diversity in clinical practice and inconsistency among guidelines. Diversity
in clinical practice can be acceptable or unacceptable. It may be reasonable
when the scientific evidence to support different courses of care is uncertain.
In addition, some degree of diversity may be warranted by differences in
individual patient characteristics and preferences and variations in delivery
system capacities related to locale, resources, and patient populations.
However, even though practice variation based on scientific uncertainty
or differences in values may be acceptable, both science and values are
OCR for page 105
CONCLUSIONS AND RECOMMENDATIONS
105
open tO change. Thus, what is perceived as acceptable diversity in clinical
practice may change over tine.
Diversity in practice is unacceptable when it stems from poor practi-
tioner skills, poor management of delivery systems, ignorance, or deliberate
disregard of well-documented preferable practices. It should not be toler-
ated when it iS a self-se~ving disguise for bad practices that harm people or
waste scarce resources.
Guidelines can clarify what is acceptable and unacceptable variation in
clinical practice, but that clarification itself has limits that may lead different
groups to different and even inconsistent guidelines. Weak evidence is still
weak evidence, although the processes described in Chapter 3 should
allow the best use of whatever evidence is available. For example, the
more formal methods of analyzing and characterizing evidence can reduce
the opportunity for inconsistency arising from poor analysis of evidence.
Nonetheless, these methods can still leave room for differences of expert
opinion about such issues as whether a flaw in research design "matters"
or whether differences in results between two treatment alternatives are
"clinically important" or only "statistically significant."
Inconsistengy among guidelines can also arise from variations in val-
ues and tolerance of nsL People may simply differ in how they perceive
different health outcomes and how they judge when benefits enough out-
weigh harms to make a service worth providing. One way to approach this
kind of variation is for guidelines developers to try to establish practitioner
and patient attitudes toward different benefits and harms and then identity
what is known about the probabilities of those different outcomes. In some
cases, the developers of guidelines take the further step of applying their
own values, but others considering the guidelines later might look at the
same information and perhaps come tO different conclusions. Also, for
some services and clinical conditions, the developers of guidelines may
choose not to recommend one course of action but to lay out alternative
courses of treatment that may be appropriate, depending on, for example,
the preferences of a patient or the characteristics of a delivery setting or
community.
In sum, merely identifying inconsistencies in guidelines says nothing
about the legitimacy of those inconsistencies. Some inconsistencies may
arise from biased or inept development processes. Some may result from
reasonable differences in the interpretation of scientific evidence or in the
application of patient, practitioner, or social values. Other inconsistencies
may essentially disappear when the rationales for specific recommendations
are closely examined. The challenge is to determine which explanation
applies. Meticulous documentation of the evidence and rationales for
guidelines will make this determination easier.
OCR for page 106
106
CLINICAL PRACTICE GUIDELINES
CONCLUDING COMMENTS
As stated at the outset of this report, this committee believes that
AHCPR's practice guidelines effort has real potential to advance the state
of the art in this field, strengthen the knowledge base for health care
practice, and, hence, improve the appropriateness and effectiveness of
health care. The conditions for such success are demanding but not out
of reach. In particular, expectations for the agency and for practice
guidelines per se-must be realistic regarding timetables and results. All
parties concerned must act in good faith and keep the credibility and
accountability of their actions in mind. Strict attention to the scientific
rigor of the process is critical as is avoidance of premature closure on a
single method of guidelines development. Attention to implementation and
evaluation needs to be factored into the development process at an early
stage.
Fulfilling the agencies mission and potential will require heroic effort
from a small staff, serious commitment from participants in the expert
panels, and honest, practical support from the many involved and interested
parties. Many conceptual and practical issues remain to be confronted. The
undertaking will be strengthened if expectations are sturdily realistic, that
is, neither too naively optimistic nor too cynically pessimistic.
Representative terms from entire chapter:
guidelines development
