Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 27
2
Criteria for Topic Selection
Many clinical practices have not been scientifically evaluated. Even when
evaluations exist, they often have not been compiled in the form of guidelines
or other statements that are accessible and persuasive to clinicians, patients, and
others who make decisions about the prevention, diagnosis, and management of
clinical problems. Because the number of clinical topics for which guidelines
might be developed far exceeds the resources for such work, those who sponsor
the development of practice guidelines must set priorities to choose among the
many possible topics.
Key to any priority-setting process are the criteria for selecting topics. The
multiplicity and complexity of aspirations for practice guidelines are reflected in
the criteria for guidelines selection set forth in the legislation establishing the
Agency for Health Care Policy and Research (AHCPR) and in the agency's own
statements. This chapter examines these criteria and considers additional ones.
CURRENT CRITERIA FOR GUIDELINES TOPICS
As described in Chapter 1, organizations vary in the criteria they use to
select topics for guidelines. Most criteria fall into several broad categories:
prevalence of a clinical problem, cost, burden of illness, disagreement or
uncertainty about appropriate care, availability of evidence on which to base
judgments, potential impact of guidelines, and relevance to particular concerns
of the sponsor.
The 1989 and 1992 legislation creating AHCPR described several general
factors for its administrator to consider in selecting topics for guidelines
development Table 2.1~. It specified additional criteria for the first three
27
OCR for page 28
28
SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES
guidelines that the agency developed. In addition, the legislation stated that
AHCPR guidelines should be based on the best scientific evidence available.
The availability of scientific evidence related to a potential topic could thus be
viewed as an implicit criterion to be considered in topic selection.
TABLE 2.1 Legislative Criteria for Selection of Guidelines Topics for
AHCPR's Forum on Quality and Effectiveness in Health Care, from P.L. 102-
239 (1989) and P.L. 102-410 (1992)
General criteria: potential of guidelines
1. to improve methods for disease prevention
2. to improve methods of diagnosis, treatment, and clinical management for
the benefit of a significant number of individuals
3. to reduce clinically significant variations among physicians in the particular
services and procedures utilized in making diagnoses and providing treatments
4. to reduce clinically significant variations in the outcomes of health care
services and procedures
5. to reflect needs and priorities of the Title XVIII program (Medicare)
Additional criteria (1989) for initial set of three guidelines: clinical treatments or
conditions that
provided
6. account for a significant portion of expenditures under Title XVIII
7. have a significant variation in the frequency or the type of treatment
Additional criteria (1992) for three guidelines
8. prevention of conditions that account for significant national health
expenditures
SOURCE: 42 USC 299bl,3.
In a 1993 Federal Register notice, the agency listed several factors that
would be considered in selecting topics for guidelines (AHCPR, 1993~. Actually,
the notice set forth two slightly different lists at different places in the text. The
(combined) list, shown in Table 2.2, is drawn from the legislative language but
differs from it in some respects. Although the intent behind the changes is not
clear to the committee, several differences are worth noting. Specifically,
OCR for page 29
CRITERIA FOR TOPIC SELECTION
29
· The 1989 legislation, in discussing initial guidelines, refers to Title XVIII
(Medicare) expenditures, practice variations, and "needs and priorities," whereas
the notice refers to "specific needs of the Medicare and Medicaid populations."
Comment: The notice draws attention to the federal and state Medicaid program
1 itle X~, which covers many children and younger adults.
· The original legislation refers only to Medicare expenditures, whereas the
notice refers to "cost of the condition to all payers, including patients."
Comment: Again, the notice extends the selection criterion, this time from
Medicare to all payers.
· The legislation refers to "clinically significant variations among
physicians," but the notice does not include the reference to physicians.
Comment: By eliminating this qualification, the notice extends the criterion
implicitly to include all practitioners (e.g., nurses, dentists) and, potentially,
patients (whose decisions may also have an important effect on outcomes).
· The legislation refers to improvements "for the benefit of a significant
number of individuals," whereas the notice lists as a criterion "the number of
individuals affected by the condition" (emphasis added). Comment Both
statements relate to the prevalence of a problem, but the legislative language also
puts some emphasis on the potential for desired changes.
· The legislation refers to the extent to which guidelines would "improve
methods of diagnosis, treatment, and clinical management" or "reduce clinically
significant variations" in services and outcomes (emphasis added). The notice
picks up only the latter point. Comment: The concept of variation reduction is
prominent in the literature on continuous quality improvement, which seems to
heat variation as something to be eliminated unless proved valuable.
TABLE 2.2 Priority-Setting Criteria Specified By the Forum
1. Amenability of a particular condition to prevention.
2. Potential for reducing clinically significant [and unexplained] variations in the
prevention, diagnosis, management, or outcomes [of health services].
3. Adequacy of science-based evidence on which to develop a guideline.
4. Number of individuals affected by the condition.
5. Cost of the condition to all payers, including patients.
6. Specific needs of the Medicare and Medicaid populations.
SOURCE: AHCPR, 1993, Program Note.
In various documents, the Forum also mentions other, less explicit criteria
for considering guidelines topics. One such criterion is whether other
OCR for page 30
30
SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES
organizations have developed guidelines related to a nominated topic. Another
involves the fit between Forum resources and the estimated cost of developing
particular sets of guidelines.
In addition, AHCPR staff have convened practitioner groups to identify
potential topics within broad clinical areas (e.g., neurology). According to
Forum staff, the choice of these topic areas reflected the agency's interests in (1)
topics of concern to nursing and allied health practitioners and (2) topics
covering major organ systems. These interests constitute implicit criteria for
topic selection.
CURRENT CRITERIA FOR TECHNOLOGY ASSESSMENT
The Office of Health Technology Assessment (OHTA) has also published
its criteria for topic selection in the Federal Register (AHCPR, 1994~. The
criteria are nearly identical to those stated in the 1992 reauthorization legislation
Table 2.3~. (The 1989 legislation did not include specific criteria, although it
referred to a process of priority setting for selecting topics.)
The criteria listed by OHTA generally follow those set forth in the 1992
IOM report on priority setting for technology assessment (see Appendix A for
summary). Where they differ, they reflect language used in the 1992 legislation
reauthorizing AHCPR. First, the OHTA refers to the potential of an assessment
"to improve health outcomes or affect costs," as if one rather than two criteria
were involved or as if only one or the other criterion would be considered. In
the committee's view, potential to improve health outcomes and potential to
affect costs should be treated as distinct criteria and scored separately.
Second, the combined use of three criteria-prevalence, economic burden
("total . . . cost per case per year of managing a condition"), and aggregate cost
of use of a technology appears to "double count" prevalence as a criterion.
Third, the OHTA use of"mortality arid morbidity" as a criterion is conceptually
narrower than the model process's "burden of illness." The latter concept
encompasses functional status arid burdens on family members. In this
committee's view, that concept provides a broader and more appropriate
understanding of patient outcomes.
'These criteria were specified in the reauthorization legislation. The IOM model
process did not include aggregate cost as a separate criterion but treated it as the product
of two other criteria, prevalence and cost per person (IOM, 1992b).
OCR for page 31
CRITERIA FOR TOPIC SELECTION
31
TABLE 23 Criteria for Setting Priorities for Selecting OHTA Technology
Assessment Topics
1. The potential of an assessment to improve health outcomes or affect costs.a
2. The morbidity and mortality associated with the health conditions
3. Variations in current practice.
4. The economic burden posed by the prevention, diagnosis, treatment, and clinical
management of the health condition.C
5. Aggregate cost of the use of the technology.
6. The prevalence of the health condition.
7. The ability of the assessment to inform ethical, social, or legal issues.&
a Two criteria related to potential impacts on outcomes and costs are combined into a single
criterion.
b In the 1992 IOM report this criterion was, instead, the `'burden of illness" associated with
a condition, which was defined as '`the difference in quality-adjusted life expectancy (QALE)
between a patient who has the condition and receives conventional treatment and the QALE
of a person of the same age who does not have the condition" (IOM, 1992b, p. 61). In the
Federal Register notice, the OHTA definition of mortality and morbidity `'[equates] morbidity
to severity of illness [which is not defined], and mortality to death" (AHCPR, 1994, p.
19726~.
c This criterion was not included in the IOM report.
This criterion was included in the IOM report but not the legislation. The OHTA included
it in the proposed criteria list after consultation.
SOURCE: AHCPR' 1994.
DIFFERENCES IN CRITERIA FOR GUIDELINES
AND TECHNOLOGY ASSESSMENT
The priority-setting criteria as now stated by the Forum and the OHTA are
fairly similar. Several differences can, however, be noted.
Unlike the guidelines criteria, the technology assessment criteria do not
explicitly refer to the needs of the Medicare and Medicaid populations. OHTA
does, nonetheless, conduct the majority of its assessments in response to requests
from the Health Care Financing Administration (HCFA), which oversees the
Medicare and Medicaid programs.2 Provisions of the 1992 reauthorization
legislation called for OHTA to look beyond these responsibilities in selecting
2HCFA's interest in guidance for Medicare coverage decisions often leads to requests
for assessments of relatively obscure or low-prevalence technologies. For example, recent
assessments have focused on salivary electrostimulation in Sjogren's Syndrome and
implantation of the automatic cardiovertor-def~brillator.
OCR for page 32
32
SEITING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES
topics for technology assessment, but the impacts of these provisions are
unclear.3
Also in contrast to the Forum criteria, the OHTA criteria do not mention the
availability of scientific evidence. In fact, the OHTA notice in the Federal
Register explicitly says that "the availability of sufficient data" should not affect
"whether but how a technology assessment should be done." That is, if an
assessment is important but cannot be addressed through scientific evidence, then
the careful use of expert consensus may be preferable to avoiding the topic. This
position is consistent with that stated in the IOM's 1992 report on priority setting
for technology assessment (IOM, 1992b). One participant in the workshop
sponsored by the committee (see Appendix B) noted, for example, that an
argument against guidelines in areas with little or no relevant evidence is
essentially an argument in favor of random variation in practice.
Another difference between the Forum and OHTA criteria is that the latter
are somewhat more explicit about the relevance of expected health outcomes.
In addition, the latter criteria include three references to costs versus one
reference in the Forum criteria.
Also in contrast to OHTA, the Forum criteria make no mention of ethical,
social, or legal issues, nor does the legislation that created the Forum. However,
on the basis of its knowledge of AHCPR and the Forum, the committee
concluded that the Forum has implicitly considered the potential of a guideline
to inform ethical, legal, or social issues in selecting some past topics for
guidelines. For example, the selection of sickle-cell management is, in the
committee's view, evidence of Forum concern with a problem of special
importance to African-Americans. Similarly, in the list of possible guidelines
topics published in September 1993, the inclusion of genetic screening and
counseling during pregnancy undoubtedly reflects a mix of social, legal, and
ethical concerns.
TOWARD COMMON CRITERIA?
Is it reasonable for topic selection criteria to differ for the Forum and
OHTA? The committee concluded that, by and large, it is not. That is, both
organizations are part of an agency charged with the broad objective of
improving the quality, appropriateness, and effectiveness of health care services
by extending and linking scientific knowledge to improve clinical practice and
According to a recent GAO report (1994), about 60 percent of OHTA's assessments
have been performed at HCFA's request, with virtually all the remainder requested by the
Civilian Health and Medical Program of the Uniformed Services in the Department of
Defense.
OCR for page 33
CRITERIA FOR TOPIC SELECTION
33
health outcomes. Both are supposed to be sensitive to health care costs and the
specific needs of federally financed health problems, although only OHTA is
responsible for making recommendations related to reimbursement.
The committee concluded that the Forum and the OHTA should move
toward common criteria and definitions for priority setting. Where criteria differ,
the differences should reflect differences in each unit's responsibilities.
In assessing how existing Forum and OHTA criteria might be revised and
combined, the committee separated criteria into two categories: general and
agency specific. General criteria are those that would figure into almost any
organization's consideration of priorities. Agency-specific criteria reflect
legislative mandates or other special factors relevant to an organization's
particular missions. The committee concluded that the general criteria are the
most amenable to "scoring" and inclusion in a formal decision model such as
that discussed later in this chapter.
General Criteria
The committee recommended that six general criteria be applied in
considering topics for either guidelines development or technology assessment.
These criteria are
Prevalence of the clinical problem: number of affected persons per 1 ,000
persons in the general U.S. population.4
2. Burden of illness imposed by the problem: individual mortality,
morbidity, or functional impairment.
3. Cost of managing the problem: cost per person.
4. Variability in practice: significant differences in utilization rates for
prevention, diagnosis, or treatment options.
5. Potential of a guideline or assessment to improve health outcomes:
expected effect on health outcomes.
6. Potential of guideline or assessment to reduce costs: expected effect on
costs (to sponsoring organization, other relevant agencies, patients and families,
and/or society in general).
Although this list includes many current Forum and OHTA criteria, it
follows more closely the list set forth in the 1992 IOM report on priority setting.
A few inclusions and exclusions from the list warrant further discussion.
4In some instances, the Forum or OHTA may wish to consider the incidence of a
clinical problem (i.e., new cases per 100,000 in a year) if it perceives that an emerging
disease (e.g., AIDS) promises to become a serious issue for clinicians and patients.
OCR for page 34
34
SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES
Potential for Guidelines to Change Health Outcomes or Costs
In addition to considering the potential for guidelines to reduce significant
clinical variation, the Forum should explicitly consider the potential for a set of
guidelines to foster desired health or cost outcomes. Some of the current criteria
focus on the magnitude of a particular variable (e.g. cost of managing a condition
or burden of illness), but none reflect the potential for a set of guidelines to
change the variable (e.g., reduce costs by 5 percent). The potential for change
actually has two components: the expected magnitude of an effect and the
probability of achieving it.
Several things must happen if practice guidelines, technology assessments,
or similar recommendations are to help improve health, reduce costs, or achieve
some other desired result (Eddy, 1989~. First, a guideline development or
technology assessment process must produce conclusions with the potential to
change clinical practice. In general, if an assessment is likely to yield
inconclusive results, the topic would not be a good candidate for selection. In
some cases, however, an assessment might be warranted if equivocal findings
were expected to discourage overenthusiastic users of a technology.
Second, exposure to the guideline or assessment must actually cause some
practitioners to change the way they care for patients. Although numerous
guidelines are commonly followed by practitioners, a major criticism of practice
guidelines and similar statements is that they are too often ignored or only
weakly implemented (Eisenberg, 1986; Kibbe, Kaluzny, and McLaughlin, 1994~.
Some topics may be better candidates for assessment because the practice
changes they would recommend would be more consistent with existing
organizational incentives or resources.
Third, a change in practice induced by the implementation of guidelines
must change patient- or organization-relevant outcomes (e.g., health status). For
example, change must not be limited to physiological variables (e.g., blood
chemistry).
Although those selecting guidelines topics should consider the potential for
better outcomes, the Forum and other organizations may sometimes reasonably
proceed with guidelines when the short-term probability of change is in doubt
(e.g., resistance to change appears strong) but the scientific evidence is clear that
an alternative to current practice is preferable. Such guidelines may be regarded
as an initial phase of a long-tern effort to change behavior and improve
outcomes even if the short-tenn measurable impact of the guidelines is
expected to be minimal. The work to develop guidelines for vaginal birth
following a previous Caesarean birth (VBAC) is a case in point. The publication
of VBAC guidelines certainly did not have an immediate effect, but the
persistent effort to find effective ways to convince physicians to apply them is
evidence of their perceived importance (Lomas, 1991, 1993~.
OCR for page 35
CRITERIA FOR TOPIC SELECTION
Social, Ethical, and Legal Considerations
35
Left out are one of the 1992 IOM criteria and one of the criteria specified
for OHS A in the 1992 reauthorization legislation: the potential of the results of
an assessment to inform ethical, legal, and social issues. In the current
committee's view, this criterion is so multidimensional that it cannot be
reasonably captured in a single criterion score. It also is sufficiently subjective
that it may be better treated as an umbrella term for agency-specific concerns.
The committee agreed that consideration of such issues is reasonable but that
the rationale for focusing on specific social, ethical, or legal concerns should be
explicit and should not be divorced from consideration of other selection criteria.
For example, if the needs of the Medicare program or prevention objectives are
a high priority, then the explanation lies in legislative mandates. In contrast, a
focus on nursing or prenatal care warrants additional explanation. A later section
ofthis chapter considers how these kinds of considerations may be systematically
factored into a priority-setting process.
Availability of Evidence
The general criterion list also does not include the availability of scientific
evidence. The committee noted, however, that one of the comparative
advantages of AHCPR as a health services and outcomes research entity is that
it is better situated to mobilize expertise to analyze scientific evidence than many
other organizations. Its advantage is less clear in situations that require near-
exclusive reliance on expert opinion. The committee thus concluded that it
malces sense for AHCPR to capitalize on its strengths and to consider the
availability of evidence as one factor in selecting topics for the development of
practice guidelines. The committee acknowledged, however, that it sometimes
may not be possible to assess the availability of evidence until a full-scale
literature review is initiated.5
Moreover, it may sometimes be appropriate for the Forum to select a topic
for which little evidence is available. For example, a strong case might be made
in some cases that misleading claims of effectiveness for an intervention need to
be countered by practice guidelines. Act analogy would be the Food and Drug
Administration's statements on certain health claims for vitamin supplements.
5The American College of Physicians, for example, has started developing guidelines
on some topics only to discover that evidence was insufficient for the kinds of
conclusions that it seeks to reach. The U.S. Preventive Services Task Force identified the
availability of evidence as one topic selection criterion, but then concluded that it lacked
the resources to assess availability and explicitly factor it into topic rankings.
OCR for page 36
36
SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES
In any case, the available scientific evidence almost always is insufficient to
answer all important questions about a clinical problem, and guidelines can make
important contributions by identifying major gaps in clinical knowledge.
ADDITIONAL CRITERIA
Should additional criteria that are neither in the Foru~n's current criteria set
nor in the set just discussed be considered by the Forums Several criteria not
currently in the Forum's list were cited by some of the organizations described
in Chapter 1 or suggested by participants in the project workshop.
Two such criteria are the degree of controversy over a problem and the
amenability of a controversy to technical clarification. For example, when the
National Cancer Institute (ACT reexamined the issue of breast cancer screening
for women aged 40 to 49, it did so because there was significant clinical
controversy, one sign of which was contradictory guidelines for clinical practice
(NIH, 1 994b). In tackling controversial issues, however, public organizations run
the risk of antagonizing politically important interests, as NCI did with its
judgment that the evidence of benefit from mammography for younger women
. .
was 1nconc uslve.
Although wide variations in clinical practice may indicate disagreement or
controversy about appropriate clinical practice, consistent practice may also
create controversy if it is inappropriate. For example, NIH looked at perinatal
corticosteroids because their use in premature labor was so consistently low
despite good evidence of benefit (NIH, 1994a). Thus, as a criterion for
consideration in topic selection, controversy is distinct from the criterion of
practice variability. The committee viewed the existence of controversy and the
susceptibility of the controversy to clarification through review of scientific
evidence as reasonable factors for AHCPR to consider in the nomination and
final selection of topics for guidelines development. These criteria, however, do
not appear to be appropriate for inclusion in a formula-based ranking system
such as that proposed by OHTA.
Criterion Measurement
In general, agency officials and legislative staff should refer to the 1992
report's discussion of measurement issues, including the importance of using
comparable time horizons for comparing alternative clinical interventions (see
Appendix A). In future explanations of its criteria for topic selection, the Forum
should provide explicit conceptual definitions and quantitative or qualitative
measures for its criteria, even if it does not propose the development of explicit
criteria scores. This would help staff and other participants in the process clarify
OCR for page 37
CRITERIA FOR TOPIC SELECTION
37
the aspects of the criteria that are of most concern and search for relevant data.
A later section of this chapter returns to selected measurement issues related to
the relative importance of different criteria.
THE RELATIVE IMPORTANCE OF INDIVIDUAL CRITERIA
Criteria Weights
Unlike the OHTA and the IOM model processes, the Forum has not
assigned explicit weights to its criteria. Neither has it adopted a formula for
generating criteria and topic scores. If the Forum eventually moves toward the
OHTA process on the basis of that unit's favorable experience with the new
process, it will have to consider whether the weights assigned by that unit are
consistent with the Forum's objectives. The OHTA weights are generally similar
to those adopted in a pilot test of the model process.
The committee did not discuss criteria weights at length. It did note that
some analyses suggest that variations in weights have less impact on the results
of formal decision analyses than might be expected (vonWinterfeldt and
Edwards, 1986~. Weighting of all criteria equally may be reasonable.
Measurement Scales
This committee concluded that the 1992 IOM report specified a generally
sound methodology for setting priorities for technology assessment.
Nevertheless, the committee echoed some concerns raised in the original report
about the different scales for quantitative and qualitative criteria (IOM, 1992b).
Specifically, scores for qualitative criteria are constrained within a scale from 1
to 5, whereas scores for quantitative criteria (e.g., cost, prevalence) are not so
constrained. To address this problem, the scoring formula takes the natural
logarithm of each criterion score, which tends to reduce differences in the range
of scores between the quantitative and qualitative criteria. For example, if the
unit cost of a service was $9,000 and the subjectively judged potential for an
assessment to change costs was rated 3.2 on a scale of 1 to 5, the criterion
values after the natural logarithm of each value was taken would be 13.66 and
2.33. The difference in scores is greatly reduced, but the effect of scale
differences remains substantial.
The 1992 report acknowledged that the unit cost criterion may dominate the
final priority score, particularly if is also given a high weight (as it was in both
the IOM and OHTA weighting exercises). Experience with the OHTA's use of
the criterion scores, weights, and overall scoring formula (assuming OHTA
proceeds as planned) will provide analysts an opportunity to assess the effects
OCR for page 38
38
SEITING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES
of these elements on topic rankings. This experience combined with sensitivity
analyses (like that just described) may provide a better sense of how best to
handle this issue.
If the results appear too skewed toward the unit cost criterion, at least three
alternative scoring approaches are possible. One approach is to transform the
unconstrained scale for this criterion to a constrained scoring range (e.g., 1 to 5~.
A second approach is to rely on the final stage of policy approval of topics to
make adjustments, for example, to select topics not on the list for consideration.
A third approach is, as discussed below, to use procedures that "force" certain
types of topics onto the list for the final policy decision on the basis of some
other criterion.
Availability of Data
The more a priority-setting process emphasizes the use of empirical
information to assess alternative topics, the more vulnerable it may be to
overemphasizing variables that can be readily measured and, consequently, to
suboptimizing decisions. For example, as has often been acknowledged,
mortality is much more easily measured than morbidity, and a priority-setting
process that does not account for this may systematically underrate guidelines for
chronic conditions for which morbidity and not mortality is the more significant
outcome. To cite another example, as part of a cost-effectiveness analysis, it is
easier to determine the purchase price of alternative technologies (e.g., $100,000
versus $200,000) than to estimate accurately their expected years of use (e.g., 2
versus 5 years).
Although the committee recognizes that the components of some criteria do
not lend themselves to easy measurement, that difficulty is not grounds for
discarding the underlying concepts. The committee recommended that those
involved in priority setting identify clearly important unmeasured elements and
the possible effects of ignoring them. Some form of expert judgment may be
used to rate potential guidelines topics on key hard-to-measure variables, with the
ratings then used as surrogates for empirical data. In sum, the committee
concluded that it is not sufficient to simply acknowledge the problem; rather, the
potential for bias should be considered each time topics are scored and ranked
against each other.
PROCEDURES FOR APPLYING CRITERIA
In addition to considering specific selection criteria, the committee discussed
more systematic procedures for applying criteria in the Forum's topic selection
OCR for page 39
CRITERIA FOR TOPIC SELECTION
39
process. Criteria may be applied simultaneously or sequentially or both. In
addition, the application of criteria may involve a single track or multiple tracks
for nominating and evaluating topics. The following sections explain these
alternatives.
Simultaneous Application of Criteria
The model process that the IOM recommended in 1992 features an explicit,
simultaneous procedure for applying seven priority-setting criteria. The
procedure takes seven individual criterion scores for a topic (e.g., cost scores)
and manipulates them through a single formula to arrive at an overall score that
can be compared to other scores to derive a topic rank. That formula is
Priority Score = Wins + W21~S2 + . . . + W71nS7,
where W is the criterion weight, in is the natural logarithm of the criterion score,
and S is the criterion score itself.
More common than this formal approach is a process that relies on what is
sometimes called "global subjective judgment." Rather than score candidate
topics explicitly on each criterion and then aggregate the individual results to
arrive at an overall score, this process relies on participants to rank topics on the
basis of an implicit (mental) consideration of all the criteria taken together. This
is essentially the procedure followed by the Forum in obtaining topic rankings
in broad clinical areas from the practitioner panels it convened.
Sequential Application of Criteria
The IOM model process actually recommended the use of both the implicit
and the explicit processes described above. An expert panel using global
subjective judgment would "winnow" a large number of topics to a smaller
number, and that reduced set oftopics would be subjected to the explicit scoring
process.6 The rationale for this two-step approach is practical; resources do not
permit all nominated topics to be considered in depth. A preliminary screening
iS necessary.
60ne alternative for winnowing topics was to rank them using a single criterion, the
one that was weighted most heavily in the overall scoring formula. The topics that
ranked highest on that single criterion would then be scored again using the full set of
. .
criteria.
OCR for page 40
40
SETTING PRIORITIES FOR CLINICALPRACTICE GUIDELINES
In addition, an organization may wish or need to apply criteria in sequence
for other reasons. In particular, some selection criteria may not readily fit into
a single scoring formula. The Forum's legislative mandates are a case in point.
The 1989 requirements for three Medicare-related guidelines and the 1992
requirements for three prevention guidelines essentially preclude selection
procedures that would allow other criteria to outweigh the mandates completely.
Several examples of possible sequencing processes and their rationales are
described below.
Sequencing Example 1. Organization A wishes to select three topics for
guideline development based on a social, legal, or ethical consideration-
specifically, malpractice claims. The organization generally agrees with the
relatively low weight assigned to the social-legal-ethical criterion in both the
IOM and OHTA processes, but it expects that the other criteria will dominate the
rankings so that no malpractice-related topic will emerge from the process.
Thus, the organization decides to "force" the topic selection results it wants by
applying selection criteria in two steps. First, it restricts initial topic nominations
to clinical conditions or services associated with significant malpractice problems.
The nominated topics are then scored and ranked using the remaining criteria and
formula specified by IOM and OHTA. The top three candidates satisfy the
organization's objective of focusing its next three guidelines on malpractice
problems.
Sequencing Example 2. Before it chooses topics for guidelines
development, organization B wants to estimate and consider explicitly the likely
cost of developing specific guidelines. It goes through the model process for
winnowing and ranking topics first and then estimates the development costs for
the six highest-ranking topics. Given its budget for guidelines development, it
decides that it prefers to do the four topics ranked 1, 4, 5, and 6, which together
have the same estimated total production cost as the three topics ranked 1, 2, and
3.
Sequencing Example 3. Organization C wishes to ensure that all of its
topics relate to children's health and, more specifically, that at least one topic
deals with prevention and one with the management of a chronic disease. The
organization "forces" this result by asking that all topics be relevant to children
and by emphasizing its particular interest in prevention and chronic illness topics.
It then separately scores and ranks the topics in each of the two designated areas
and in the "all other" category before deciding which topics to select.
OCR for page 41
CRITERIA FOR TOPIC SELECTION
Single- Versus Multiple-Track Processes
41
Sequencing Example 1 above describes a single-track process for setting
priorities. That is, all potential topics are funneled through the same constrained
nominating procedure and criterion, the same winnowing procedure, and the
same final scoring procedure. Sequencing Example 2 also involves a single
track, one in which an additional criterion is applied to all high-ranking topics
after a ranked list is generated. Sequencing Example 3, however, describes a
multi-track process, one that has three separate funnels or channels for
nominating, winnowing, and ranking topics. Figure 2.1 illustrates this process.
Although not as explicit or rigid as Sequencing Example 3, the Forum's use
of practitioner panels to consider topics in broad clinical areas amounts to a
multi-track and sequential process for setting priorities. Legislative mandates
(Medicare priorities and prevention) designate additional topic tracks for special
consideration.
The committee agreed that it is reasonable and desirable for the Forum to
apply criteria in ways that ensure (or at least make it likely) that certain general
kinds of topics appear on the recommended list of topics for guideline
development. The committee recommended, however, that the Forum define
more explicitly its rationales for selecting clinical areas or other topic categories
for special consideration in its priority-setting process. This step will allow
interested parties to critique the rationales and contribute to long-term
improvement in the entire process, including better ways of establishing broad
topic categories for special attention.
REASSESSMENT OF EXISTING GUIDELINES
The discussion to this point has focused on the selection of new topics for
guidelines development and ignored decisions about the reassessment or updating
of existing guidelines. In general, guidelines should be reassessed with an eye
toward three objectives:
· incorporation of new knowledge;
· correction of errors of fact or interpretation; and
· alterations in content or format that improve the usefulness of the
guidelines.
The continuing advance of scientific knowledge provides the most
formidable challenge for those charged with developing and updating practice
guidelines. Indeed, the availability of new evidence is generally the only issue
raised in most discussions of updating. Regular literature searches are one way
OCR for page 42
42
SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES
All Other Topics
Topic
Nomination
Initial Screening
of Topics
Assessment of
Remaining Topics
Topics for Final
Consideration and
Decision
T
Ust 1
FIGURE 2.1 Multi-~ack process for selecting guidelines topics.
Topics in designated areas
1~
1
List 2 List 3
14~
1 ~
to identify new evidence. The solicitation of topics for guidelines development
or reassessment is another avenue.
Errors of fact or interpretation do not appear to be common in guidelines
such as those developed by AHCPR, but they are possible. Broadly viewed,
errors can arise for many reasons, including bias in the original panel, faulty
conceptualization of the problem, and even typographical mistakes. Although
every organization hopes to avoid such errors in the first place or to correct them
as drafts are reviewed and edited, the publication of guidelines is typically an
occasion for mistakes or misstatements to be identified. As individuals and
groups try to implement the guidelines, other errors may be discovered.
Implementation also provides an opportunity for users to identify ways in
which guidelines might be more useful. For example, although those who have
tried to implement the Forum's guidelines have found that they sometimes were
not specific enough to guide actual patient care decisions, they have praised those
guideline documents that have included instruments for patient assessment and
outcomes measurement (e.g., the Braden Scale for defining patient risk for
pressure ulcers). Users might identify well-tested tools that were overlooked in
a set of guidelines and that could be included in an update. Algorithm
presentation offers another area in which existing (and new) guidelines may be
improved, either by adding algorithms to guidelines that lacked them or by
revising those that were included. Although the committee heard considerable
OCR for page 43
CRITERIA FOR TOPIC SELECTION
43
disagreement about the utility of algorithms for clinicians, AHCPR panels should
be encouraged to use common formats and symbols when they do present
algorithms.
Depending on the nature of the text to be corrected or the content or format
improvements to be made, staffmay take care of updates or revisions without the
need for the initial development panel or a new one to be much involved, if at
all. (Distribution of corrections may be difficult and expensive, but the
committee did not consider these issues.) If the reason for updating guidelines
is new evidence, however, then decisions about revisions will likely require a
more extensive effort. The options include reconvening the original panel,
convening a new panel, or contracting with a private organization to undertake
the update.
How should the Forum decide when to invest resources in reassessing and
updating existing guidelines? One view suggests treating these topics like any
others and subjecting them to regular priority-setting procedures. Another view
is that the Forum has a particular responsibility once it publishes guidelines to
monitor and maintain them, and thus, these guidelines should have a special
status in the topic nomination and selection process. One way to provide such
a status is for the Forum to routinely include existing guidelines topics when it
solicits comments on proposed topics for guidelines development and to
designate a separate "track" for considering these comments, the results of
literature searches, and the established criteria. For the foreseeable future, the
committee concluded that the latter course was prudent.
OCR for page 44
Representative terms from entire chapter:
practice guidelines
