Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Documentation and Use of Evidence in the Consensus Conference Process* Gerard Bre art The purpose of a consensus development conference, as defined by the U.S. Office of Medical Applications of Research (OMAR) of the National Institutes of Health (NIH), is to evaluate the available scientific information on a biomedical technology and to produce a consensus statement that advances understanding of the technology or issue in question and that will be useful to health professionals and the public at large (OMAR, NIH, 19881. The consensus state- ment should reflect the available scientific evidence. Critics have highlighted the shortcomings of the consensus devel- opment process and considered the consensus statement to be more of a compromise of divergent viewpoints than an in-depth analysis of the existing research evidence (Ahrens, 19X5; Levitt and Polish, 1988; Lomas, 1986; Oliver, 1985~. Jacoby (1988) states that: The quality and quantity of data available for specific questions at these recent conferences still varied considerably. The separate elements of the consensus statements, however, did not always reflect this variability in how strongly conclusions were stated, resulting in inappropriately strong conclu- sions. These two different opinions on the origin and nature of consen- sus statements aBude to the fact that, in some circumstances, there *The author acknowledges, with thanks, Sharon R. Baratz for revising the maIlu- script. 23
24 CONSENSUS DEVELOPMENT may be a gap between the available data (consensus process origin) and the consensus statement (consensus process result). The aim of this paper is to outline the elements that may explain this gap. This discontinuity between the available evidence and the final statement may be due to the consensus process itself or to the general deci- sion-making process for clinical practice. THE CONSENSUS PROCESS In order to ensure that the final statement is based primarily on scientific evidence, the whole consensus process has to be oriented toward this objective. The following three conditions must be met: I. Scientific evidence must exist and must be clearly and com- pletely presented to the panel in an easily accessible and understand- able form. 2. Members of the panel must be ready to base their decisions on scientific evidence. Panelists must not have a strong opinion on the topic before the conference. Panelists must be given criteria for the evaluation of scientific studies. 3. The environment (actual practices, outside pressures) must al- low room for decisions based on scientific data. These three conditions have to be kept in mind throughout the organization of the following stages of the consensus development process: · preparation of the process · review of the existing evidence · selection of the panel · selection of the speakers . formulation of recommendations. Preparation of the Process At least two conditions must be satisfied for the consensus state- ment to be based on scientific evidence: (~) the existence of a rea- sonable body of research data and (2) variability in practice patterns conceming the proposed topic. Given insufficient data, the consensus statement may only be a compromise of contradictory expert opinions. The relationship be-
DOCUMENTATION AND USE OF FYIDENCE 25 tween current practice for a technology or clinical condition and the consensus development conference is of particular importance. If few practitioners diverge from the standard procedure, it is likely that the consensus statement will reflect the general opinion, even in the absence of good evidence. On the other hand, if practices vary considerably, it will be clear to the- panel that choices have to be made. The pane} should be prepared and willing to make choices on a scientific basis. One such an example has been observed in Can- ada for the consensus conference on cesarean birth (Battista and Fletcher, 198X). The importance of topic selection implies that the preparation of a consensus development conference should include a survey on actual practice as opposed to opinion on practice. The objective of the survey would be to identify problem areas and the potential for evidence to aid in finding solutions. This "need assess- ment" phase was an integral part of the Canadian conference (Lo- mas, 19861. Review of the Existing Evidence To guarantee that the review of the literature will be complete and systematic, a specific individual should be assigned to this task. This staff person should prepare a summary of each main paper using a standard presentation that explicitly states the criteria for evaluation of the quality of the studies. This individual should also prepare a synthesis of the literature using the technique of meta- analysis Whenever applicable. The review should include published papers and unpublished re- suits as well information on studies in progress. (Unpublished stud- ies may tend to have negative findings more often than published ones do.) The review should consider articles related to efficacy or effectiveness of the therapeutic or preventive method evaluated and papers that address other criteria for assessments, for example: · actual practices in the field and consequences for the health care system of any modification in practice · direct or indirect adverse effects of the evaluated procedure. The last point is particularly important as evaluations of "early detection procedures" by noninvasive technologies, generally regarded as safe, should include consideration of the consequences of false- positive or false-negative diagnoses.
26 Selection of the Pane! CONSENSUS DEVELOPMENT The panel should be as neutral as possible, and OMAR guidelines should be carefully followed: A. Balanced representation on the consensus pane} is crucial because a range of expertise on the panel is important to the panel's ability to deal with varied scientific material presented, and a diver- sified panel enhances the credibility of the consensus statement. B. Panel members must be thoughtful, able to weigh evidence, and capable of collaborative work. C. Panelists should have no vested interest in the technology being reviewed. D. The size of panels has varied from 9 to 16 members; 12 or 13 is a reasonable working group. E. The pane! should contain balanced representation from var'- ous sectors of professional and community life and should not be professionally identified with advocacy or promotional positions with respect to the consensus topic (OMAR, NIH, 19881. In addition, it may be useful to give the panel basic articles con- cerning the evaluation of the quality of data. Selection of the Speakers Speakers should be selected for their scientific expertise. Confer- ence planning committees should provide precise information to speakers concerning the topic they have to address. Speakers who present reviews on the topic should be asked to include aU of the opposing data and interpretations in their presentations. Speakers should receive information on the evaluation of the quality of pub- lished data. Speakers who present their own data should receive specific recommendations on the format for presenting the method- ology of the research. Formulation of Recommendations The members of the panel are asked to base their recommenda- tions on scientific evidence. It may be easier to comply with this rule if members of the pane] are asked to explicitly support their recommendations by data from the literature (even if this evidence
DOCUME~ATION ED USE OF WID=CE 27 is not included in the final statement) and if members of the pane} use standard sentences as a grading scale for recommendations. In their study of the periodic health examination, Battista and Fletcher (1988) proposed the following scale. A. There is good evidence to support the recommendation that the condition should be specifically considered in a periodic health examination. B. There is fair evidence to support the recommendation that the condition should be specifically considered in a periodic health ex- amination. C. There is poor evidence regarding the consideration of the con- dition in a periodic health examination, and recommendations may be made on other grounds. D. There is fair evidence to support the recommendation that the condition be excluded from consideration in a periodic health ex- amination. E. There is good evidence to support the recommendation that the condition should be excluded from consideration in a periodic health examination. SCIENTIFIC EVIDENCE AND CLINICAL PRACTICE The discussion on how to increase the use of scientific evidence in a consensus development process can also be broadened to the discussion of the relation between scientific evidence and use of innovative procedures. Surveys conducted in France in the field of perinatology (i.e., on ultrasound examinations, electronic fetal moni- toring, and prescription of beta-mimetics) give an example of this relation (Blondel et al., 1989; Breart, 1984; Ringa et al., 1986, 1989~. Of the doctors who provided antenatal care, 99 percent used rou- tine ultrasound screening for every pregnant woman. When there was no complication, 2 percent of the respondents said that they prescribed only one ultrasound examination, 47 percent prescribed two, 45 percent prescribed three, and 6 percent prescribed four or more. Overall, 96 percent of the respondents considered that im- provement in diagnosis and in pregnancy outcome were reasons for obstetric ultrasound (the latter was most frequently classified as the primary reason); 65 percent of the doctors mentioned demand by the mother and the safety of the procedure as reasons for ultrasound;
28 CONSENSUS DElfELOPMENT most of the time these reasons were ranked at the third or fourth level in the list of reasons (Blonde! et al., 1989~. Only ~ of 455 people who attended deliveries did not use elec- tronic fetal monitoring during labor. This screening was mostly applied to every woman: one percent of the respondents used it for high-risk women only. This pattern of practice was established be- fore 1975 by 39 percent of the obstetricians, between 1975 and 1979 by 37 percent, and after 1980 by 24 percent. Most of the obstetri- cians gave earlier detection of fetal distress as the first reason for their using the procedure. Overall, 99 percent of the respondents said that they used beta- mimetics for the prevention of preterm delivery. When asked under what circumstances they prescribed these drugs, 74 percent of the doctors said that they administered oral beta-mimetics prophylacti- cally, 63 percent used intravenous or oral beta-mimetics when they detected signs of premature maturation of the cervix, 76 percent prescribed them for women with ruptured membranes, and 49 per- cent prescribed them when the cervix was dilated 4 cm or more. The principal reason given by 65 percent of the obstetricians for using beta-mimetics was the tocolytic effect of these drugs; 38 per- cent indicated delay of delivery as a primary concern. Two other reasons (better compliance with rest and their psychological effect) were less often quoted, and they were mostly the third and fourth reasons for using these drugs. Thirty-eight percent of the doctors had not changed their practice concerning the use of beta-mimetics since 1980, Il percent had prescribed them more often, and S} per- cent had prescribed them less often (Blonde} et al., 1989~. Side effects of the treatment were more frequently reported as reasons for decreasing beta-mimetic use (90 percent) than was the lack of effectiveness of these drugs (74 percent), and the former was more frequency mentioned as the first reason. Improvement in neo- natal care and the existence of other treatments to prevent preterrn delivery were rarely given as the principal reason. Upon analysis of the evidence drawn from the clinical trials (Breart, 1984; King et al., 198S; Ringa et al., 1986, 1989), it is clear that the actual practices have not taken into account the results of the ran- domized controlled trials that did not favor extensive use of any of these procedures or that were published after use of the procedures had spread. However, these results do not indicate that use of any new tech- nology is not based upon scientific evidence. A systematic review
DOCUMENTATION AND USE OF EVIDENCE 29 of all articles published on ultrasound examination in four obstetr~- cal journals read by French obstetricians was performed for the pe- riod from 1979 to 1984. Of the 182 papers identified, 137 were re- lated to the diagnostic assessment of ultrasound, 14 concerned the effects on medical practice, and only 9 papers studied the effects of ultrasound scanning on health. Most of the 137 papers on diagnostic assessments of ultrasound came out in favor of the procedure. Many articles have been published on the topic; thus, the decision to use this technology seems to have been based on scientific publications. However, these publications addressed only the first level of evalu- ation, namely, the diagnostic value of the procedure, whereas an epidemiologist would have expected a more complete evaluation. According to the data presented here, it can be said that the deci- sion to practice routine ultrasound examination has been made on a theoretical basis, rather than on proof of its efficacy and effective- ness. The same holds true for the diffusion of beta-mimetics for a variety of clinical conditions; the propagation of the drugs was based on their ability to stop uterine contractions and not on the proof of their efficacy in reducing the preterm birth rate (the original purpose of these compounds). Similar results have also been observed for electronic fetal monitoring (Breart, 1984~. The surveys conducted in France also revealed that the decision to modify any given practice seems to be based on possible side effects rather than on doubts concerning its effectiveness. There- fore, if clinical practice is mainly based on "theoretical" considera- tion, as well as on side effects, this must be taken into consideration in the consensus development process. To take into account the first point, it may be useful to advise the members of the consensus development pane} to clearly distinguish between theoretical consid- erations and actual proof. For this task, the pane! should receive information about the theoretical basis for the effectiveness of the proposed intervention. This summary should include presentations of causal pathways in a manner similar to that proposed by Battista and Fletcher (1988) for preventive practices (Figure I). The presentation of the theoretical basis as well as the causal pathways should be accompanied by a list of questions concerning the efficacy and the effectiveness of the proposed interventions at each step. For the instance of early detection of high serum choles- terol levels, the ultimate purpose is to prevent the occurrence of coronary heart disease (CHD). The intermediate steps along the causal pathway are the ability of the detection procedure to identify
30 Asymptomatic individuals Early detection of high serum cholesterol levels High serum cholesterol levels 5 Dietary counseling and/or pharmacologic treatment Lowering of serum cholesterol levels 4 3 FIGURE 1 Early detection of high serum cholesterol levels. SOURCE: Batti.sta and Fletcher (1988~. CONSENSUS DElfELOPMENT Prevention of coronary heart disease (CHD) individuals with high cholesterol levels (step 1), the ability of di- etary counseling and/or pharmacologic treatment to lower serum cho- lestero} levels (step 2), and the successful prevention of CHD result- ing from the lowering of serum cholesterol levels (step 3~. The most important link here (step 4), is the efficacy or effectiveness of di- etary counseling and/or pharmacologic treatment. The utility of early detection depends upon how well these intermediate steps lead to prevention of CHD in asymptomatic individuals with high serum cholesterol levels (step 5~. A very careful search of published and unpublished results should be performed to address the issue of side effects. Since the theoreti- cal beneficial effects of a given procedure are taken into considera- tion in the consensus development process, the theoretical side ef- fects have to be considered with the same weight. The problem of the use of scientific evidence to make decisions is not unique to consensus development, as it is a general problem for
DOCUMENTATION AND USE OF EVIDENCE 31 decision making in clinical practice. Better use of scientific evi- dence in Me consensus development process may come from under- standing and consideration of the process of clinical decision mak- ing by practitioners. REFERENCES Ahrens, E.H. 1985. The diet heart question in 1985: Has it really been settled? The Lancet i:l085-1087. Battista, R.N., and S.W. Fletcher. 1988. Making recommendations on preventive practices: Methodological issues. American Journal of Preventive Medicine 4~4~(Suppl.~:53-67. Blondel, B., V. Ringa, and G. Breart. 1989. The use of ultrasound examinations, intrapartum fetal heart rate monitoring and beta-mimetic drugs in France. British Journal of Obstetrics and Gynaecology 96:44-51. Breart, G. 1984. Un probleme de decision en perinatologie: l'Enregistrement continu du coeur foetal pendant le travail. Revue d'Epidemiologie et de Sante Publique 32:34-39. Jacoby, I. 1988. Evidence and consensus. Journal of the American Medical Asso- ciation 259 (20~:3039. King, J.F., M.J.N.C. Keirse, A. Grant, and I. Chalmers. 1985. Tocolysis. The case for and against. Pp. 199-200 in Preterm Labor and its Consequences, R.W. Beard and F. Sharp, eds. London: Royal College of Obstetricians and Gynaecolo- gists. Levitt, S.H., and R.A. Polish. 1988. The consensus statements on adjuvant chemo- therapy in breast cancer: Fact or artifact. American Journal of Oncology 11:73- 76. Lomas, J. 1986. The consensus process and evidence dissemination. Canadian Medical Association Journal 134:1340-1341. Lomas, J., G.M. Anderson, M.W. Enkin, E. Vayda, R. Roberts, and B. MacKinnon. 1988. The role of evidence in the consensus exercise, results from a Canadian consensus exercise. Journal of the American Medical Association259~20~:3001- 3005. Office of Medical Applications of Research, National Institutes of Health. 1988. Guidelines for the selection and management of consensus development confer- ences. Bethesda, Md. Pamphlet. Oliver, M.F. 1985. Consensus or non-consensus conferences on coronary heart disease. Lancet i:l087-1089. Ringa, V., B. Blondel, and G. Breart. 1986. Evaluation des traitements contre la prematurity. Pp. 191-198 in Control and Management of Parturition, C. Sureau, P. Blot, D. Cabrol, et al., eds. Paris: Institut National de la Sante et de la Recher- che Medicale-John Libbey. Ringa, V., B. Blondel, and G. Breart. 1989. Ultrasound in obstetrics: Do the pub- lished evaluative studies justify its routine use? International Journal of Epi- demiology 18:489-497.
