Appendix C
Clinical Practice Guideline Appraisal Tools



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 213
Appendix C Clinical Practice Guideline Appraisal Tools 213

OCR for page 213
TABLE C-1 Examples of Tools for Measuring the Quality of Clinical Practice Guidelines Development 214 Tool/Authors Purpose Tool A Provisional Instrument for To provide an explicit method Each of the tool’s seven attributes subsumes a number of Assessing Clinical Practice for examining the soundness of dimensions. Each dimension is assigned a “yes/no” score (the Guidelines clinical practice guidelines (CPGs) dimension is or is not represented in the guideline). If yes, an and to encourage their systematic associated rating of satisfactory, conditionally satisfactory, or IOM (1992) development. The committee unsatisfactory (degree to which the dimension is satisfied) is developed this instrument to be given. If the dimension is not represented (i.e., “no” response), used as an education tool, a self- then a judgment of whether the omission is unimportant, minor, or assessment tool, or a means of major is made. If most responses to the questions are “satisfactory” judging guidelines before their (or “unimportant omissions”), one might reasonably conclude that adoption. such a guideline would be sufficient for most clinical situations. Alternatively, if most responses were unsatisfactory (or major omissions), one would probably argue that the guideline needed to be revised before it could be used effectively. The seven attributes include 1. Clinical applicability a. The guideline document describes the patient populations to which the guidelines are meant to apply. b. The guideline document discusses complex clinical problems for the population(s) covered by the guidelines. c. The guideline document gives a rationale for excluding patient population(s). 2. Clinical flexibility a. The guideline document provides specific information about situations in which clinical exceptions might be made in applying the guidelines.

OCR for page 213
b. The guideline document provides specific information about nonclinical situations in which exceptions might be made in applying the guidelines. c. The guideline document discusses the role of patient preferences as they relate to healthcare decisions in the particular case that the guidelines cover. d. The guideline document describes how patient preferences were taken into account during the guideline development process. 3. Reliability/reproducibility a. The guidelines were subjected to independent review by experts or outside panels. b. The guideline document explains the lack of independent review. c. The guidelines were pretested in some manner. 4. Validity: Definition and evaluation questions a. The guideline document specifically describes the method(s) used to collect (i.e., identify and retrieve) the scientific evidence on which recommendations are based. b. The guideline document gives adequate references or citations to the sources of information used in developing the guidelines. c. The guideline document discusses in general terms the strength of the scientific evidence on which recommendations are based. d. The guideline document explicitly rates the strength of evidence. continued 215

OCR for page 213
TABLE C-1 Continued 216 Tool/Authors Purpose Tool e. If a formal method of synthesis is used to combine the scientific evidence quantitatively or otherwise to develop summary outcome measures that reflect the strength of the scientific evidence, then the guideline document explicitly describes the method. f. Given that a formal method of synthesis is used to combine the scientific evidence quantitatively or otherwise to develop summary outcome measures, the guideline document explicitly reports the results of that synthesis. g. If formal expert or group judgment techniques are used to reach professional consensus, then the guideline document explicitly describes the techniques. h. Given that expert or group judgment method(s) are used to reach professional consensus, the guideline document explicitly gives information about the strength of professional consensus. i. The guideline document provides a qualitative description of the health benefits that are expected from a specific health practice. j. The guideline document provides a qualitative description of the potential harms or risks that may occur as a result of a specific health practice. k. The guideline document provides quantitative information or estimates about the health benefits to be expected as a result of a specific health practice.

OCR for page 213
l. The guideline document projects health benefits or outcomes in terms of additional life expectancy or similar measures, such as Quality-Adjusted Life Years (QUALYs). m. The guideline document provides a quantitative description of the potential harms or risks that may occur as a result of a specific health practice. n. The guideline document provides a qualitative description of the health costs or expenditures that are expected from a specific health practice. o. The guideline document provides a quantitative description of the health costs or expenditures that are expected from a specific health practice. p. If health benefits are projected in terms of additional life expectancy or similar measures, such as QUALYs, then the cost per unit of each identified benefit is estimated. q. Generally, the estimates of the benefits, harms, and costs are consistent with the strength of the evidence presented in the guideline document. r. The guideline document explicitly discusses the strength of the scientific evidence on which each major recommendation is based. s. Each major recommendation is consistent with the estimated benefits, harms, and costs of the service or intervention (and thus with the strength of evidence). t. The guideline document identifies other sets of guidelines that deal with the same clinical condition, technology, or topic. u. The guideline document identifies possible conflicts among existing guidelines and the reasons for them. continued 217

OCR for page 213
TABLE C-1 Continued 218 Tool/Authors Purpose Tool 5. Clarity a. The guidelines describe the health condition to be prevented, detected, or treated in unambiguous terms. b. The guidelines describe the options for management of the health condition (i.e., the health practice and its alternatives) in unambiguous terms. c. If the guidelines give major recommendations, each is written in unambiguous terms. d. Recommendations are comprehensive, insofar as the evidence permits, and recommendations that might be expected are given. e. Recommendations are consistent (do not conflict with each other). f. The guideline document uses clear headings, indexes, lists, flow charts, or other devices to identify major topics discussed. g. The guideline document has a summary or abstract that accurately reflects the methods, content, and recommendations of the entire document. h. Each user of the guideline document can easily find each major recommendation. 6. Scheduled review a. The guideline document gives a specific date for the scheduled review, gives other information concerning the procedure by which a scheduled review might be done, or gives a sunset or expiration date.

OCR for page 213
7. Multidisciplinary process a. Persons with appropriate clinical and methodological disciplines participated in developing the guideline document—that is, a multidisciplinary approach was followed. b. The guideline document explicitly notes any potential biases and/or conflicts of interests of the panel members, or states that biases and conflicts of interest were discussed among panel members or otherwise taken into account. c. Overall, potential biases and/or conflicts of interest appear to be adequately balanced or otherwise accounted for in the guideline development process. The guideline process describes the methods used to solicit views of interested parties not on the guidelines development panel and to present those views to the members of the panel. Appraisal of Guidelines To assess the quality of CPGs: AGREE consists of 23 key items organized in 6 domains. Each for Research & Evaluation how well a guideline is developed domain is intended to capture a separate dimension of guideline (AGREE) and reported. It does not assess quality. Each item is rated on a 4-point Likert scale ranging from the clinical content or quality of 4 (strongly agree) to 1 (strongly disagree). Domain scores are Cluzeau, F.A., J. Burgers evidence of the recommendations. calculated by summing up all the scores of the individual items and an international in a domain and by standardizing the total as a percentage of the group of researchers maximum possible score for that domain. The six domain scores from 13 countries. are independent and should not be aggregated into a single quality score; there is no threshold score that separates a “good” from a AGREE (2001) “not good” guideline. However, there is a section at the end for an overall assessment of the guideline, with a series of options (strongly recommend, recommend, would not recommend, and unsure). continued 219

OCR for page 213
TABLE C-1 Continued 220 Tool/Authors Purpose Tool AGREE II AGREE II represents an update of the original tool to improve its Brouwers et al. (2010) usability and methodological properties, namely its validity and reliability. AGREE II also includes 2 final overall assessment items that require the appraiser to make overall judgments of the practice guideline and consider how they rated the 23 items. AGREE II contains the same six domains as the original AGREE; changes to individual attributes of the domains are highlighted below in italics. Scope and purpose 1. The overall objective(s) of the guideline is (are) specifically described. 2. The clinical question(s) covered by the guideline is (are) specifically described. AGREE II: The health question(s) covered by the guideline is (are) specifically described. 3. The patients to whom the guideline is meant to apply are specifically described. AGREE II: The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described. Stakeholder involvement 4. The guideline development group includes individuals from all the relevant professional groups. 5. The patients’ views and preferences have been sought. AGREE II: The views and preferences of the target population (patients, public, etc.) have been sought. 6. The target users of the guideline are clearly defined. 7. The guideline has been piloted among target users. AGREE II: Deleted item. Incorporated into user guide description of item 19.

OCR for page 213
Rigor of development 8. Systematic methods were used to search for evidence. AGREE II: No change. Renumber to 7. 9. The criteria for selecting the evidence are clearly described. AGREE II: No change. Renumber to 8. a. AGREE II: New item number 9. The strengths and limitations of the body of evidence are clearly described. 10. The methods used for formulating the recommendations are clearly described. 11. The health benefits, side effects, and risks have been considered in formulating the recommendations. 12. There is an explicit link between recommendations and the supporting evidence. 13. The guideline has been externally reviewed by experts prior to its publication. 14. A procedure for updating the guideline is provided. Clarity and presentation 15. The recommendations are specific and unambiguous. 16. The different options for management of the condition are clearly presented. AGREE II: The different options for management of the condition or health issue are clearly presented. 17. Key recommendations are easily identifiable. 18. The guideline is supported with tools for application. AGREE II: The guideline provides advice and/or tools on how the recommendations can be put into practice. Domain changes from clarity of presentation to applicability, and renumbered to 19. continued 221

OCR for page 213
TABLE C-1 Continued 222 Tool/Authors Purpose Tool Applicability 19. The potential organizational barriers in applying the recommendations have been discussed. AGREE II: The guideline describes facilitators and barriers to its implementation. Change in order from 19 to 18. 20. The potential cost implications of applying the recommendations have been considered. AGREE II: The potential resource implications of applying the recommendations have been considered. 21. The guideline presents key review criteria for monitoring and/or audit purposes. AGREE II: The guideline presents monitoring and/or auditing criteria. Editorial independence 22. The guideline is editorially independent from the funding body. AGREE II: The views of the functioning body have not influenced the content of the guideline. Conflicts of interest of guideline development members have been recorded. AGREE II: Competing interests of guideline development group members have been recorded and addressed. Cluzeau’s “Appraisal A critical appraisal instrument A checklist containing 37 items categorized into 3 conceptual Instrument” that assesses whether guideline dimensions: rigor of development (validity and reproducibility), developers have minimized biases context and content (reliability, applicability, flexibility, and Cluzeau et al. (1999) inherent in creating guidelines and clarity), and application (implementation, dissemination, and addresses requirements for effective monitoring strategies). The specific questions are answered yes, no, dissemination and implementation. or unsure, which are then given values of 1, 0, and 0 respectively.

OCR for page 213
A guideline dimension score is obtained by calculating the mean of the appraiser’s scores, and is then expressed as a percentage of the maximum possible score for that dimension in order to compare scores across the three dimensions. There is also a global qualitative assessment of the guidelines, with the options (strongly recommended, recommended, and not recommended) as a measure for overall quality. Dimension one: Rigor of development process 1. Is the agency responsible for the development of the guidelines clearly identified? 2. Was external funding or other support received for developing the guidelines? 3. If external funding or support was received, is there evidence that potential biases of the funding body(ies) were taken into account? 4. Is there a description of the individuals (e.g., professionals, interest groups—including patients) who were involved in the guideline development group? 5. If so, did the group contain representatives of all key disciplines? 6. Is there a description of the sources of information used to select the evidence on which the recommendations are based? 7. If so, are the sources of information adequate? 8. Is there a description of the method(s) used to interpret and assess the strength of evidence? continued 223

OCR for page 213
TABLE C-1 Continued 224 Tool/Authors Purpose Tool 9. If so, is (are) the method(s) for rating the evidence adequate? 10. Is there a description of the methods used to formulate the recommendations? 11. If so, are the methods satisfactory? 12. Is there an indication of how the views of interested parties not on the panel were taken into account? 13. Is there an explicit link between the major recommendations and the level of supporting evidence? 14. Were the guidelines independently reviewed prior to publication/release? 15. If so, is explicit information given about the methods and how the comments were addressed? 16. Were the guidelines piloted? 17. If so, is explicit information given about the methods used and the results adopted? 18. Is a date mentioned for reviewing or updating the guidelines? 19. Is the body responsible for the reviewing and updating clearly indentified? 20. Overall, have the potential biases of guideline development been adequately addressed? Dimension two: Context and content 1. Are the reasons for developing the guidelines clearly stated? 2. Are the objectives of the guidelines clearly defined?

OCR for page 213
3. Is there a satisfactory description of the patients to which the guidelines are meant to apply? 4. Is there a description of the circumstances (clinical or nonclinical) in which exceptions might be made in using the guidelines? 5. Is there an explicit statement of how the patient’s preferences should be taken into account in applying the guidelines? 6. Do the guidelines describe the condition to be detected, treated, or prevented in unambiguous terms? 7. Are the different possible options for the management of the condition clearly stated in the guidelines? 8. Are the recommendations clearly presented? 9. Is there an adequate description of the health benefits that are likely to be gained from the recommended management? 10. Is there an adequate description of the potential harms or risks that may occur as a result of the recommended management? 11. Is there an estimate of the costs or expenditures likely to incur from the recommended management? 12. Are the recommendations supported by the estimated benefits, harms, and costs of the intervention? Dimension three: Application of guidelines 13. Does the guideline document suggest possible methods for dissemination and implementation? 14. (National guidelines only) Does the guideline document identify key elements that need to be considered by local guideline groups? continued 225

OCR for page 213
TABLE C-1 Continued 226 Tool/Authors Purpose Tool 15. Does the guideline document specify criteria for monitoring compliance? 16. Does the guideline document identify clear standards or targets? 17. Does the guideline document define measureable outcomes that can be monitored? Hayward et al.’s “Structured A structured format for abstracts The instrument includes eight topics essential to CPG Abstracts of CPGs” of articles describing CPGs to help developmental reporting. Format for structured abstracts of CPGs readers assess the applicability, Hayward et al. (1993) importance, and validity of any 1. Objective: The primary objective of the guideline, guideline. including health problem and the targeted patients, providers, and settings. 2. Options: The clinical practice options considered in formulating the guideline. 3. Outcomes: Significant health and economic outcomes considered in comparing alternative practices. 4. Evidence: How and when evidence was gathered, selected, and synthesized. 5. Values: Disclosure of how values were assigned to potential outcomes of practice options and who participated in the process. 6. Benefits, harms, and costs: The type and magnitude of benefits, harms, and costs expected for patients from guideline implementation.

OCR for page 213
7. Recommendations: Summary of key recommendations. 8. Validation: Report of any external review, comparison with other guidelines, or clinical testing of guideline use. Sponsors: Disclosure of the persons who developed, funded, or endorsed the guideline. Shaneyfelt’s “Methodological A tool to assess methodological The 25-item instrument uses a yes or no format to measure Standards” quality of the development and adherence to three broadly grouped quality criteria. A guideline’s reporting of clinical practice score reflects the total number of standards it satisfies, that is, Shaneyfelt et al. (1999) guidelines in the peer-reviewed “yes” responses (0–25). medical literature. Standards of guidelines development and format 1. Purpose of the guideline is specified. 2. Rationale and importance of the guideline are explained. 3. The participants in the guideline development process and their areas of expertise are specified. 4. Targeted health problem or technology is clearly defined. 5. Targeted patient population is specified. 6. Intended audience or users of the guideline are specified. 7. The principal preventive, diagnostic, or therapeutic options available to clinicians and patients are specified. 8. The health outcomes are specified. 9. The method by which the guideline underwent external review is specified. 10. An expiration date or date of scheduled review is specified. continued 227

OCR for page 213
TABLE C-1 Continued 228 Tool/Authors Purpose Tool Standards of evidence identification and summary 11. Method of identifying scientific evidence is specified. 12. The time period from which evidence is reviewed is specified. 13. The evidence used is identified by citation and referenced. 14. Method of data extractions is specified. 15. Method for grading or classifying the scientific evidence is specified. 16. Formal methods of combining evidence or expert opinion are used and described. 17. Benefits and harms of specific health practices are specified. 18. Benefits and harms are quantified. 19. The effect on healthcare costs from specific health practices is specified. 20. Costs are quantified. Standards on the formulation of recommendations 21. The role of value judgments used by the guideline developers in making recommendations is discussed. 22. The role of patient preferences is discussed. 23. Recommendations are specific and apply to the stated goals of the guideline. 24. Recommendations are graded according to the strength of the evidence. Flexibility in the recommendations is specified. SOURCES: AGREE (2001); Brouwers et al. (2010); Cluzeau et al. (1999); Hayward et al. (1993); IOM (1992); Shaneyfelt et al. (1999).

OCR for page 213
229 APPENDIX C REFERENCES AGREE. 2001. Appraisal of guidelines for research & evaluation (AGREE) instrument. Brouwers, M. C., M. E. Kho, G. P. Browman, J. S. Burgers, F. Cluzeau, G. Feder, B. Fervers, I. D. Graham, S. E. Hanna, J. Makarski, and for the AGREE Next Steps Consortium. 2010. Development of the AGREE II, part 2: Assessment of validity of items and tools to support application. CMAJ 182(10):E472–E478. Cluzeau, F. A., P. Littlejohns, J. M. Grimshaw, G. Feder, and S. E. Moran. 1999. Devel - opment and application of a generic methodology to assess the quality of clinical guidelines. International Journal for Quality in Health Care 11(1):21–28. Hayward, R. S. A., M. C. Wilson, S. R. Tunis, E. B. Bass, H. R. Rubin, and R. B. Haynes. 1993. More informative abstracts of articles describing clinical practice guide - lines. Annals of Internal Medicine 118(9):731–737. IOM (Institute of Medicine). 1992. Guidelines for clinical practice: From development to use. Edited by M. J. Field and K. N. Lohr. Washington, DC: National Academy Press. Shaneyfelt, T., M. Mayo-Smith, and J. Rothwangl. 1999. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 281:1900–1905.

OCR for page 213