Click for next page ( 28


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 27
2 Criteria for Topic Selection Many clinical practices have not been scientifically evaluated. Even when evaluations exist, they often have not been compiled in the form of guidelines or other statements that are accessible and persuasive to clinicians, patients, and others who make decisions about the prevention, diagnosis, and management of clinical problems. Because the number of clinical topics for which guidelines might be developed far exceeds the resources for such work, those who sponsor the development of practice guidelines must set priorities to choose among the many possible topics. Key to any priority-setting process are the criteria for selecting topics. The multiplicity and complexity of aspirations for practice guidelines are reflected in the criteria for guidelines selection set forth in the legislation establishing the Agency for Health Care Policy and Research (AHCPR) and in the agency's own statements. This chapter examines these criteria and considers additional ones. CURRENT CRITERIA FOR GUIDELINES TOPICS As described in Chapter 1, organizations vary in the criteria they use to select topics for guidelines. Most criteria fall into several broad categories: prevalence of a clinical problem, cost, burden of illness, disagreement or uncertainty about appropriate care, availability of evidence on which to base judgments, potential impact of guidelines, and relevance to particular concerns of the sponsor. The 1989 and 1992 legislation creating AHCPR described several general factors for its administrator to consider in selecting topics for guidelines development Table 2.1~. It specified additional criteria for the first three 27

OCR for page 27
28 SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES guidelines that the agency developed. In addition, the legislation stated that AHCPR guidelines should be based on the best scientific evidence available. The availability of scientific evidence related to a potential topic could thus be viewed as an implicit criterion to be considered in topic selection. TABLE 2.1 Legislative Criteria for Selection of Guidelines Topics for AHCPR's Forum on Quality and Effectiveness in Health Care, from P.L. 102- 239 (1989) and P.L. 102-410 (1992) General criteria: potential of guidelines 1. to improve methods for disease prevention 2. to improve methods of diagnosis, treatment, and clinical management for the benefit of a significant number of individuals 3. to reduce clinically significant variations among physicians in the particular services and procedures utilized in making diagnoses and providing treatments 4. to reduce clinically significant variations in the outcomes of health care services and procedures 5. to reflect needs and priorities of the Title XVIII program (Medicare) Additional criteria (1989) for initial set of three guidelines: clinical treatments or conditions that provided 6. account for a significant portion of expenditures under Title XVIII 7. have a significant variation in the frequency or the type of treatment Additional criteria (1992) for three guidelines 8. prevention of conditions that account for significant national health expenditures SOURCE: 42 USC 299bl,3. In a 1993 Federal Register notice, the agency listed several factors that would be considered in selecting topics for guidelines (AHCPR, 1993~. Actually, the notice set forth two slightly different lists at different places in the text. The (combined) list, shown in Table 2.2, is drawn from the legislative language but differs from it in some respects. Although the intent behind the changes is not clear to the committee, several differences are worth noting. Specifically,

OCR for page 27
CRITERIA FOR TOPIC SELECTION 29 The 1989 legislation, in discussing initial guidelines, refers to Title XVIII (Medicare) expenditures, practice variations, and "needs and priorities," whereas the notice refers to "specific needs of the Medicare and Medicaid populations." Comment: The notice draws attention to the federal and state Medicaid program 1 itle X~, which covers many children and younger adults. The original legislation refers only to Medicare expenditures, whereas the notice refers to "cost of the condition to all payers, including patients." Comment: Again, the notice extends the selection criterion, this time from Medicare to all payers. The legislation refers to "clinically significant variations among physicians," but the notice does not include the reference to physicians. Comment: By eliminating this qualification, the notice extends the criterion implicitly to include all practitioners (e.g., nurses, dentists) and, potentially, patients (whose decisions may also have an important effect on outcomes). The legislation refers to improvements "for the benefit of a significant number of individuals," whereas the notice lists as a criterion "the number of individuals affected by the condition" (emphasis added). Comment Both statements relate to the prevalence of a problem, but the legislative language also puts some emphasis on the potential for desired changes. The legislation refers to the extent to which guidelines would "improve methods of diagnosis, treatment, and clinical management" or "reduce clinically significant variations" in services and outcomes (emphasis added). The notice picks up only the latter point. Comment: The concept of variation reduction is prominent in the literature on continuous quality improvement, which seems to heat variation as something to be eliminated unless proved valuable. TABLE 2.2 Priority-Setting Criteria Specified By the Forum 1. Amenability of a particular condition to prevention. 2. Potential for reducing clinically significant [and unexplained] variations in the prevention, diagnosis, management, or outcomes [of health services]. 3. Adequacy of science-based evidence on which to develop a guideline. 4. Number of individuals affected by the condition. 5. Cost of the condition to all payers, including patients. 6. Specific needs of the Medicare and Medicaid populations. SOURCE: AHCPR, 1993, Program Note. In various documents, the Forum also mentions other, less explicit criteria for considering guidelines topics. One such criterion is whether other

OCR for page 27
30 SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES organizations have developed guidelines related to a nominated topic. Another involves the fit between Forum resources and the estimated cost of developing particular sets of guidelines. In addition, AHCPR staff have convened practitioner groups to identify potential topics within broad clinical areas (e.g., neurology). According to Forum staff, the choice of these topic areas reflected the agency's interests in (1) topics of concern to nursing and allied health practitioners and (2) topics covering major organ systems. These interests constitute implicit criteria for topic selection. CURRENT CRITERIA FOR TECHNOLOGY ASSESSMENT The Office of Health Technology Assessment (OHTA) has also published its criteria for topic selection in the Federal Register (AHCPR, 1994~. The criteria are nearly identical to those stated in the 1992 reauthorization legislation Table 2.3~. (The 1989 legislation did not include specific criteria, although it referred to a process of priority setting for selecting topics.) The criteria listed by OHTA generally follow those set forth in the 1992 IOM report on priority setting for technology assessment (see Appendix A for summary). Where they differ, they reflect language used in the 1992 legislation reauthorizing AHCPR. First, the OHTA refers to the potential of an assessment "to improve health outcomes or affect costs," as if one rather than two criteria were involved or as if only one or the other criterion would be considered. In the committee's view, potential to improve health outcomes and potential to affect costs should be treated as distinct criteria and scored separately. Second, the combined use of three criteria-prevalence, economic burden ("total . . . cost per case per year of managing a condition"), and aggregate cost of use of a technology appears to "double count" prevalence as a criterion. Third, the OHTA use of"mortality arid morbidity" as a criterion is conceptually narrower than the model process's "burden of illness." The latter concept encompasses functional status arid burdens on family members. In this committee's view, that concept provides a broader and more appropriate understanding of patient outcomes. 'These criteria were specified in the reauthorization legislation. The IOM model process did not include aggregate cost as a separate criterion but treated it as the product of two other criteria, prevalence and cost per person (IOM, 1992b).

OCR for page 27
CRITERIA FOR TOPIC SELECTION 31 TABLE 23 Criteria for Setting Priorities for Selecting OHTA Technology Assessment Topics 1. The potential of an assessment to improve health outcomes or affect costs.a 2. The morbidity and mortality associated with the health conditions 3. Variations in current practice. 4. The economic burden posed by the prevention, diagnosis, treatment, and clinical management of the health condition.C 5. Aggregate cost of the use of the technology. 6. The prevalence of the health condition. 7. The ability of the assessment to inform ethical, social, or legal issues.& a Two criteria related to potential impacts on outcomes and costs are combined into a single criterion. b In the 1992 IOM report this criterion was, instead, the `'burden of illness" associated with a condition, which was defined as '`the difference in quality-adjusted life expectancy (QALE) between a patient who has the condition and receives conventional treatment and the QALE of a person of the same age who does not have the condition" (IOM, 1992b, p. 61). In the Federal Register notice, the OHTA definition of mortality and morbidity `'[equates] morbidity to severity of illness [which is not defined], and mortality to death" (AHCPR, 1994, p. 19726~. c This criterion was not included in the IOM report. This criterion was included in the IOM report but not the legislation. The OHTA included it in the proposed criteria list after consultation. SOURCE: AHCPR' 1994. DIFFERENCES IN CRITERIA FOR GUIDELINES AND TECHNOLOGY ASSESSMENT The priority-setting criteria as now stated by the Forum and the OHTA are fairly similar. Several differences can, however, be noted. Unlike the guidelines criteria, the technology assessment criteria do not explicitly refer to the needs of the Medicare and Medicaid populations. OHTA does, nonetheless, conduct the majority of its assessments in response to requests from the Health Care Financing Administration (HCFA), which oversees the Medicare and Medicaid programs.2 Provisions of the 1992 reauthorization legislation called for OHTA to look beyond these responsibilities in selecting 2HCFA's interest in guidance for Medicare coverage decisions often leads to requests for assessments of relatively obscure or low-prevalence technologies. For example, recent assessments have focused on salivary electrostimulation in Sjogren's Syndrome and implantation of the automatic cardiovertor-def~brillator.

OCR for page 27
32 SEITING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES topics for technology assessment, but the impacts of these provisions are unclear.3 Also in contrast to the Forum criteria, the OHTA criteria do not mention the availability of scientific evidence. In fact, the OHTA notice in the Federal Register explicitly says that "the availability of sufficient data" should not affect "whether but how a technology assessment should be done." That is, if an assessment is important but cannot be addressed through scientific evidence, then the careful use of expert consensus may be preferable to avoiding the topic. This position is consistent with that stated in the IOM's 1992 report on priority setting for technology assessment (IOM, 1992b). One participant in the workshop sponsored by the committee (see Appendix B) noted, for example, that an argument against guidelines in areas with little or no relevant evidence is essentially an argument in favor of random variation in practice. Another difference between the Forum and OHTA criteria is that the latter are somewhat more explicit about the relevance of expected health outcomes. In addition, the latter criteria include three references to costs versus one reference in the Forum criteria. Also in contrast to OHTA, the Forum criteria make no mention of ethical, social, or legal issues, nor does the legislation that created the Forum. However, on the basis of its knowledge of AHCPR and the Forum, the committee concluded that the Forum has implicitly considered the potential of a guideline to inform ethical, legal, or social issues in selecting some past topics for guidelines. For example, the selection of sickle-cell management is, in the committee's view, evidence of Forum concern with a problem of special importance to African-Americans. Similarly, in the list of possible guidelines topics published in September 1993, the inclusion of genetic screening and counseling during pregnancy undoubtedly reflects a mix of social, legal, and ethical concerns. TOWARD COMMON CRITERIA? Is it reasonable for topic selection criteria to differ for the Forum and OHTA? The committee concluded that, by and large, it is not. That is, both organizations are part of an agency charged with the broad objective of improving the quality, appropriateness, and effectiveness of health care services by extending and linking scientific knowledge to improve clinical practice and According to a recent GAO report (1994), about 60 percent of OHTA's assessments have been performed at HCFA's request, with virtually all the remainder requested by the Civilian Health and Medical Program of the Uniformed Services in the Department of Defense.

OCR for page 27
CRITERIA FOR TOPIC SELECTION 33 health outcomes. Both are supposed to be sensitive to health care costs and the specific needs of federally financed health problems, although only OHTA is responsible for making recommendations related to reimbursement. The committee concluded that the Forum and the OHTA should move toward common criteria and definitions for priority setting. Where criteria differ, the differences should reflect differences in each unit's responsibilities. In assessing how existing Forum and OHTA criteria might be revised and combined, the committee separated criteria into two categories: general and agency specific. General criteria are those that would figure into almost any organization's consideration of priorities. Agency-specific criteria reflect legislative mandates or other special factors relevant to an organization's particular missions. The committee concluded that the general criteria are the most amenable to "scoring" and inclusion in a formal decision model such as that discussed later in this chapter. General Criteria The committee recommended that six general criteria be applied in considering topics for either guidelines development or technology assessment. These criteria are Prevalence of the clinical problem: number of affected persons per 1 ,000 persons in the general U.S. population.4 2. Burden of illness imposed by the problem: individual mortality, morbidity, or functional impairment. 3. Cost of managing the problem: cost per person. 4. Variability in practice: significant differences in utilization rates for prevention, diagnosis, or treatment options. 5. Potential of a guideline or assessment to improve health outcomes: expected effect on health outcomes. 6. Potential of guideline or assessment to reduce costs: expected effect on costs (to sponsoring organization, other relevant agencies, patients and families, and/or society in general). Although this list includes many current Forum and OHTA criteria, it follows more closely the list set forth in the 1992 IOM report on priority setting. A few inclusions and exclusions from the list warrant further discussion. 4In some instances, the Forum or OHTA may wish to consider the incidence of a clinical problem (i.e., new cases per 100,000 in a year) if it perceives that an emerging disease (e.g., AIDS) promises to become a serious issue for clinicians and patients.

OCR for page 27
34 SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES Potential for Guidelines to Change Health Outcomes or Costs In addition to considering the potential for guidelines to reduce significant clinical variation, the Forum should explicitly consider the potential for a set of guidelines to foster desired health or cost outcomes. Some of the current criteria focus on the magnitude of a particular variable (e.g. cost of managing a condition or burden of illness), but none reflect the potential for a set of guidelines to change the variable (e.g., reduce costs by 5 percent). The potential for change actually has two components: the expected magnitude of an effect and the probability of achieving it. Several things must happen if practice guidelines, technology assessments, or similar recommendations are to help improve health, reduce costs, or achieve some other desired result (Eddy, 1989~. First, a guideline development or technology assessment process must produce conclusions with the potential to change clinical practice. In general, if an assessment is likely to yield inconclusive results, the topic would not be a good candidate for selection. In some cases, however, an assessment might be warranted if equivocal findings were expected to discourage overenthusiastic users of a technology. Second, exposure to the guideline or assessment must actually cause some practitioners to change the way they care for patients. Although numerous guidelines are commonly followed by practitioners, a major criticism of practice guidelines and similar statements is that they are too often ignored or only weakly implemented (Eisenberg, 1986; Kibbe, Kaluzny, and McLaughlin, 1994~. Some topics may be better candidates for assessment because the practice changes they would recommend would be more consistent with existing organizational incentives or resources. Third, a change in practice induced by the implementation of guidelines must change patient- or organization-relevant outcomes (e.g., health status). For example, change must not be limited to physiological variables (e.g., blood chemistry). Although those selecting guidelines topics should consider the potential for better outcomes, the Forum and other organizations may sometimes reasonably proceed with guidelines when the short-term probability of change is in doubt (e.g., resistance to change appears strong) but the scientific evidence is clear that an alternative to current practice is preferable. Such guidelines may be regarded as an initial phase of a long-tern effort to change behavior and improve outcomes even if the short-tenn measurable impact of the guidelines is expected to be minimal. The work to develop guidelines for vaginal birth following a previous Caesarean birth (VBAC) is a case in point. The publication of VBAC guidelines certainly did not have an immediate effect, but the persistent effort to find effective ways to convince physicians to apply them is evidence of their perceived importance (Lomas, 1991, 1993~.

OCR for page 27
CRITERIA FOR TOPIC SELECTION Social, Ethical, and Legal Considerations 35 Left out are one of the 1992 IOM criteria and one of the criteria specified for OHS A in the 1992 reauthorization legislation: the potential of the results of an assessment to inform ethical, legal, and social issues. In the current committee's view, this criterion is so multidimensional that it cannot be reasonably captured in a single criterion score. It also is sufficiently subjective that it may be better treated as an umbrella term for agency-specific concerns. The committee agreed that consideration of such issues is reasonable but that the rationale for focusing on specific social, ethical, or legal concerns should be explicit and should not be divorced from consideration of other selection criteria. For example, if the needs of the Medicare program or prevention objectives are a high priority, then the explanation lies in legislative mandates. In contrast, a focus on nursing or prenatal care warrants additional explanation. A later section ofthis chapter considers how these kinds of considerations may be systematically factored into a priority-setting process. Availability of Evidence The general criterion list also does not include the availability of scientific evidence. The committee noted, however, that one of the comparative advantages of AHCPR as a health services and outcomes research entity is that it is better situated to mobilize expertise to analyze scientific evidence than many other organizations. Its advantage is less clear in situations that require near- exclusive reliance on expert opinion. The committee thus concluded that it malces sense for AHCPR to capitalize on its strengths and to consider the availability of evidence as one factor in selecting topics for the development of practice guidelines. The committee acknowledged, however, that it sometimes may not be possible to assess the availability of evidence until a full-scale literature review is initiated.5 Moreover, it may sometimes be appropriate for the Forum to select a topic for which little evidence is available. For example, a strong case might be made in some cases that misleading claims of effectiveness for an intervention need to be countered by practice guidelines. Act analogy would be the Food and Drug Administration's statements on certain health claims for vitamin supplements. 5The American College of Physicians, for example, has started developing guidelines on some topics only to discover that evidence was insufficient for the kinds of conclusions that it seeks to reach. The U.S. Preventive Services Task Force identified the availability of evidence as one topic selection criterion, but then concluded that it lacked the resources to assess availability and explicitly factor it into topic rankings.

OCR for page 27
36 SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES In any case, the available scientific evidence almost always is insufficient to answer all important questions about a clinical problem, and guidelines can make important contributions by identifying major gaps in clinical knowledge. ADDITIONAL CRITERIA Should additional criteria that are neither in the Foru~n's current criteria set nor in the set just discussed be considered by the Forums Several criteria not currently in the Forum's list were cited by some of the organizations described in Chapter 1 or suggested by participants in the project workshop. Two such criteria are the degree of controversy over a problem and the amenability of a controversy to technical clarification. For example, when the National Cancer Institute (ACT reexamined the issue of breast cancer screening for women aged 40 to 49, it did so because there was significant clinical controversy, one sign of which was contradictory guidelines for clinical practice (NIH, 1 994b). In tackling controversial issues, however, public organizations run the risk of antagonizing politically important interests, as NCI did with its judgment that the evidence of benefit from mammography for younger women . . was 1nconc uslve. Although wide variations in clinical practice may indicate disagreement or controversy about appropriate clinical practice, consistent practice may also create controversy if it is inappropriate. For example, NIH looked at perinatal corticosteroids because their use in premature labor was so consistently low despite good evidence of benefit (NIH, 1994a). Thus, as a criterion for consideration in topic selection, controversy is distinct from the criterion of practice variability. The committee viewed the existence of controversy and the susceptibility of the controversy to clarification through review of scientific evidence as reasonable factors for AHCPR to consider in the nomination and final selection of topics for guidelines development. These criteria, however, do not appear to be appropriate for inclusion in a formula-based ranking system such as that proposed by OHTA. Criterion Measurement In general, agency officials and legislative staff should refer to the 1992 report's discussion of measurement issues, including the importance of using comparable time horizons for comparing alternative clinical interventions (see Appendix A). In future explanations of its criteria for topic selection, the Forum should provide explicit conceptual definitions and quantitative or qualitative measures for its criteria, even if it does not propose the development of explicit criteria scores. This would help staff and other participants in the process clarify

OCR for page 27
CRITERIA FOR TOPIC SELECTION 37 the aspects of the criteria that are of most concern and search for relevant data. A later section of this chapter returns to selected measurement issues related to the relative importance of different criteria. THE RELATIVE IMPORTANCE OF INDIVIDUAL CRITERIA Criteria Weights Unlike the OHTA and the IOM model processes, the Forum has not assigned explicit weights to its criteria. Neither has it adopted a formula for generating criteria and topic scores. If the Forum eventually moves toward the OHTA process on the basis of that unit's favorable experience with the new process, it will have to consider whether the weights assigned by that unit are consistent with the Forum's objectives. The OHTA weights are generally similar to those adopted in a pilot test of the model process. The committee did not discuss criteria weights at length. It did note that some analyses suggest that variations in weights have less impact on the results of formal decision analyses than might be expected (vonWinterfeldt and Edwards, 1986~. Weighting of all criteria equally may be reasonable. Measurement Scales This committee concluded that the 1992 IOM report specified a generally sound methodology for setting priorities for technology assessment. Nevertheless, the committee echoed some concerns raised in the original report about the different scales for quantitative and qualitative criteria (IOM, 1992b). Specifically, scores for qualitative criteria are constrained within a scale from 1 to 5, whereas scores for quantitative criteria (e.g., cost, prevalence) are not so constrained. To address this problem, the scoring formula takes the natural logarithm of each criterion score, which tends to reduce differences in the range of scores between the quantitative and qualitative criteria. For example, if the unit cost of a service was $9,000 and the subjectively judged potential for an assessment to change costs was rated 3.2 on a scale of 1 to 5, the criterion values after the natural logarithm of each value was taken would be 13.66 and 2.33. The difference in scores is greatly reduced, but the effect of scale differences remains substantial. The 1992 report acknowledged that the unit cost criterion may dominate the final priority score, particularly if is also given a high weight (as it was in both the IOM and OHTA weighting exercises). Experience with the OHTA's use of the criterion scores, weights, and overall scoring formula (assuming OHTA proceeds as planned) will provide analysts an opportunity to assess the effects

OCR for page 27
38 SEITING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES of these elements on topic rankings. This experience combined with sensitivity analyses (like that just described) may provide a better sense of how best to handle this issue. If the results appear too skewed toward the unit cost criterion, at least three alternative scoring approaches are possible. One approach is to transform the unconstrained scale for this criterion to a constrained scoring range (e.g., 1 to 5~. A second approach is to rely on the final stage of policy approval of topics to make adjustments, for example, to select topics not on the list for consideration. A third approach is, as discussed below, to use procedures that "force" certain types of topics onto the list for the final policy decision on the basis of some other criterion. Availability of Data The more a priority-setting process emphasizes the use of empirical information to assess alternative topics, the more vulnerable it may be to overemphasizing variables that can be readily measured and, consequently, to suboptimizing decisions. For example, as has often been acknowledged, mortality is much more easily measured than morbidity, and a priority-setting process that does not account for this may systematically underrate guidelines for chronic conditions for which morbidity and not mortality is the more significant outcome. To cite another example, as part of a cost-effectiveness analysis, it is easier to determine the purchase price of alternative technologies (e.g., $100,000 versus $200,000) than to estimate accurately their expected years of use (e.g., 2 versus 5 years). Although the committee recognizes that the components of some criteria do not lend themselves to easy measurement, that difficulty is not grounds for discarding the underlying concepts. The committee recommended that those involved in priority setting identify clearly important unmeasured elements and the possible effects of ignoring them. Some form of expert judgment may be used to rate potential guidelines topics on key hard-to-measure variables, with the ratings then used as surrogates for empirical data. In sum, the committee concluded that it is not sufficient to simply acknowledge the problem; rather, the potential for bias should be considered each time topics are scored and ranked against each other. PROCEDURES FOR APPLYING CRITERIA In addition to considering specific selection criteria, the committee discussed more systematic procedures for applying criteria in the Forum's topic selection

OCR for page 27
CRITERIA FOR TOPIC SELECTION 39 process. Criteria may be applied simultaneously or sequentially or both. In addition, the application of criteria may involve a single track or multiple tracks for nominating and evaluating topics. The following sections explain these alternatives. Simultaneous Application of Criteria The model process that the IOM recommended in 1992 features an explicit, simultaneous procedure for applying seven priority-setting criteria. The procedure takes seven individual criterion scores for a topic (e.g., cost scores) and manipulates them through a single formula to arrive at an overall score that can be compared to other scores to derive a topic rank. That formula is Priority Score = Wins + W21~S2 + . . . + W71nS7, where W is the criterion weight, in is the natural logarithm of the criterion score, and S is the criterion score itself. More common than this formal approach is a process that relies on what is sometimes called "global subjective judgment." Rather than score candidate topics explicitly on each criterion and then aggregate the individual results to arrive at an overall score, this process relies on participants to rank topics on the basis of an implicit (mental) consideration of all the criteria taken together. This is essentially the procedure followed by the Forum in obtaining topic rankings in broad clinical areas from the practitioner panels it convened. Sequential Application of Criteria The IOM model process actually recommended the use of both the implicit and the explicit processes described above. An expert panel using global subjective judgment would "winnow" a large number of topics to a smaller number, and that reduced set oftopics would be subjected to the explicit scoring process.6 The rationale for this two-step approach is practical; resources do not permit all nominated topics to be considered in depth. A preliminary screening iS necessary. 60ne alternative for winnowing topics was to rank them using a single criterion, the one that was weighted most heavily in the overall scoring formula. The topics that ranked highest on that single criterion would then be scored again using the full set of . . criteria.

OCR for page 27
40 SETTING PRIORITIES FOR CLINICALPRACTICE GUIDELINES In addition, an organization may wish or need to apply criteria in sequence for other reasons. In particular, some selection criteria may not readily fit into a single scoring formula. The Forum's legislative mandates are a case in point. The 1989 requirements for three Medicare-related guidelines and the 1992 requirements for three prevention guidelines essentially preclude selection procedures that would allow other criteria to outweigh the mandates completely. Several examples of possible sequencing processes and their rationales are described below. Sequencing Example 1. Organization A wishes to select three topics for guideline development based on a social, legal, or ethical consideration- specifically, malpractice claims. The organization generally agrees with the relatively low weight assigned to the social-legal-ethical criterion in both the IOM and OHTA processes, but it expects that the other criteria will dominate the rankings so that no malpractice-related topic will emerge from the process. Thus, the organization decides to "force" the topic selection results it wants by applying selection criteria in two steps. First, it restricts initial topic nominations to clinical conditions or services associated with significant malpractice problems. The nominated topics are then scored and ranked using the remaining criteria and formula specified by IOM and OHTA. The top three candidates satisfy the organization's objective of focusing its next three guidelines on malpractice problems. Sequencing Example 2. Before it chooses topics for guidelines development, organization B wants to estimate and consider explicitly the likely cost of developing specific guidelines. It goes through the model process for winnowing and ranking topics first and then estimates the development costs for the six highest-ranking topics. Given its budget for guidelines development, it decides that it prefers to do the four topics ranked 1, 4, 5, and 6, which together have the same estimated total production cost as the three topics ranked 1, 2, and 3. Sequencing Example 3. Organization C wishes to ensure that all of its topics relate to children's health and, more specifically, that at least one topic deals with prevention and one with the management of a chronic disease. The organization "forces" this result by asking that all topics be relevant to children and by emphasizing its particular interest in prevention and chronic illness topics. It then separately scores and ranks the topics in each of the two designated areas and in the "all other" category before deciding which topics to select.

OCR for page 27
CRITERIA FOR TOPIC SELECTION Single- Versus Multiple-Track Processes 41 Sequencing Example 1 above describes a single-track process for setting priorities. That is, all potential topics are funneled through the same constrained nominating procedure and criterion, the same winnowing procedure, and the same final scoring procedure. Sequencing Example 2 also involves a single track, one in which an additional criterion is applied to all high-ranking topics after a ranked list is generated. Sequencing Example 3, however, describes a multi-track process, one that has three separate funnels or channels for nominating, winnowing, and ranking topics. Figure 2.1 illustrates this process. Although not as explicit or rigid as Sequencing Example 3, the Forum's use of practitioner panels to consider topics in broad clinical areas amounts to a multi-track and sequential process for setting priorities. Legislative mandates (Medicare priorities and prevention) designate additional topic tracks for special consideration. The committee agreed that it is reasonable and desirable for the Forum to apply criteria in ways that ensure (or at least make it likely) that certain general kinds of topics appear on the recommended list of topics for guideline development. The committee recommended, however, that the Forum define more explicitly its rationales for selecting clinical areas or other topic categories for special consideration in its priority-setting process. This step will allow interested parties to critique the rationales and contribute to long-term improvement in the entire process, including better ways of establishing broad topic categories for special attention. REASSESSMENT OF EXISTING GUIDELINES The discussion to this point has focused on the selection of new topics for guidelines development and ignored decisions about the reassessment or updating of existing guidelines. In general, guidelines should be reassessed with an eye toward three objectives: incorporation of new knowledge; correction of errors of fact or interpretation; and alterations in content or format that improve the usefulness of the guidelines. The continuing advance of scientific knowledge provides the most formidable challenge for those charged with developing and updating practice guidelines. Indeed, the availability of new evidence is generally the only issue raised in most discussions of updating. Regular literature searches are one way

OCR for page 27
42 SETTING PRIORITIES FOR CLINICAL PRACTICE GUIDELINES All Other Topics Topic Nomination Initial Screening of Topics Assessment of Remaining Topics Topics for Final Consideration and Decision T Ust 1 FIGURE 2.1 Multi-~ack process for selecting guidelines topics. Topics in designated areas 1~ 1 List 2 List 3 14~ 1 ~ to identify new evidence. The solicitation of topics for guidelines development or reassessment is another avenue. Errors of fact or interpretation do not appear to be common in guidelines such as those developed by AHCPR, but they are possible. Broadly viewed, errors can arise for many reasons, including bias in the original panel, faulty conceptualization of the problem, and even typographical mistakes. Although every organization hopes to avoid such errors in the first place or to correct them as drafts are reviewed and edited, the publication of guidelines is typically an occasion for mistakes or misstatements to be identified. As individuals and groups try to implement the guidelines, other errors may be discovered. Implementation also provides an opportunity for users to identify ways in which guidelines might be more useful. For example, although those who have tried to implement the Forum's guidelines have found that they sometimes were not specific enough to guide actual patient care decisions, they have praised those guideline documents that have included instruments for patient assessment and outcomes measurement (e.g., the Braden Scale for defining patient risk for pressure ulcers). Users might identify well-tested tools that were overlooked in a set of guidelines and that could be included in an update. Algorithm presentation offers another area in which existing (and new) guidelines may be improved, either by adding algorithms to guidelines that lacked them or by revising those that were included. Although the committee heard considerable

OCR for page 27
CRITERIA FOR TOPIC SELECTION 43 disagreement about the utility of algorithms for clinicians, AHCPR panels should be encouraged to use common formats and symbols when they do present algorithms. Depending on the nature of the text to be corrected or the content or format improvements to be made, staffmay take care of updates or revisions without the need for the initial development panel or a new one to be much involved, if at all. (Distribution of corrections may be difficult and expensive, but the committee did not consider these issues.) If the reason for updating guidelines is new evidence, however, then decisions about revisions will likely require a more extensive effort. The options include reconvening the original panel, convening a new panel, or contracting with a private organization to undertake the update. How should the Forum decide when to invest resources in reassessing and updating existing guidelines? One view suggests treating these topics like any others and subjecting them to regular priority-setting procedures. Another view is that the Forum has a particular responsibility once it publishes guidelines to monitor and maintain them, and thus, these guidelines should have a special status in the topic nomination and selection process. One way to provide such a status is for the Forum to routinely include existing guidelines topics when it solicits comments on proposed topics for guidelines development and to designate a separate "track" for considering these comments, the results of literature searches, and the established criteria. For the foreseeable future, the committee concluded that the latter course was prudent.

OCR for page 27