Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Format and Conduct of Consensus Development Conferences: A Multination Comparison* Elizabeth A. McGlynn, Jacqueline Kosecoff, and Robert H. Brook INTRODUCTION The purpose of this paper is to compare the methods used in nine countries to organize and conduct consensus development confer- ences on scientific issues related to the delivery of medical care. Our comments focus primarily on the format and conduct of these conferences. However, we also briefly discuss the context, selec- tion, and scope of topics; the role of evidence; and the dissemination of results. We confine our remarks on these latter topics to descr~p- tive statements that illuminate issues relevant to a discussion of process. We address the process by which consensus development confer- ences are conducted because the approach taken may influence the results, or the acceptability of the results, either intentionally or unintentionally. Thus, at each step of the process it is important to consider whether the mechanics of conducting the conference are enhancing or detracting from achieving its goals and objectives. To the extent that the mechanism itself interferes with the intended outcomes, changes in the approach are warranted. *This paper is to be published in a forthcoming issue of the International Jour- nal of Technology Assessment in Health Care, Cambridge University Press, New York. 32
FORMAT AND CONDUCT METHODS 33 We set out to examine the process by which consensus develop- ment conferences are conducted in Canada, Denmark, Finland, The Netherlands, Norway, Sweden, Switzerland, the United Kingdom, and the United States. Although most of these countries began with the mode} used in the United States by the National Institutes of Health (NIH), many important variations have been introduced (Casparie and van Everdingen, 1985; Bennett, 1985; Perry, 1987, 19XX; Stocking, 1985; yang, 1986~. The 10 programs considered are: I. The Canadian Task Force on the Periodic Health Examina- tion (Canada I) 2. The Canadian Research Group (Canada II) 3. The Danish Medical Research Council and the Danish Hospi- tal Institute 4. The Medical Research Council of the Academy of Finland 5. National Organization for Quality Assurance in Hospitals in The Netherlands 6. The Norwegian Institute for Hospital Research and the Nor- wegian National Research Council 7. The Swedish Planning and Rationalization Institute for the Health and Social Services 8. The Swiss Institute of Public Health 9. The King's Fund Forum in the United Kingdom 10. The U.S. National Institutes of Health Office of Medical Ap- plications of Research (OMAR). The consensus development conference is a complex entity, and researchers have undertaken scientific evaluations of the conferences and their impact in their respective countries (Calitorp, 1988; Johnsson, 1988; Kanouse et al., 1989; Lomas et al., 1988; Worunan et al., 1988~. An international comparison is challenging because there may be as much or more variation within a particular country regarding how individual conferences are conducted as there is among countries (Andreasen, 1988~. For example, more than 75 such con- ferences have been held by NIH in the United States, and although the mode} was basically the same, implementation certainly varied across the conferences. We have attempted to use the most recent data available to represent each country in order to capture any
34 CONSENSUS DEVELOPMENT changes that have taken place over time in the conduct of the confer- ences. We used three sources of information to prepare this paper: I. Profiles prepared by representatives from each country for the June 1989 workshop on international consensus development for medical technology assessment organized by the Institute of Medi- cine 2. Published articles and consensus statements related to such conferences 3. Comments received after the conference from representatives of each country. We divided the consensus development process into four stages for our investigation: I. Context of the consensus development process 2. Prepane} process 3. Pane} composition 4. Consensus pane! meeting. The context within which consensus conferences are conducted encompasses the nature of the audience, the topics considered, and how the topics are selected. The prepare} process includes all ac- tivities that are required to stage the consensus conference, such as selecting the chair, panel members, and presenters as well as prepa- ration of background information. Pane} composition includes the type of panelists, their qualifications, and the process by which they are selected. The consensus panel meeting stage describes the ac- tivities at the actual conference, such as the use of public forums and private sessions, the type of information considered in arriving at consensus, and the group process by which consensus is actually achieved. RESULTS In this section, we describe how each of the countries described in this paper conducts consensus conferences. We have organized the section around the four major areas investigated: context, prepanel process, pane} composition, and the consensus panel meeting. Within each area we addressed several questions. Tables ~ through 4 sum- marize our findings. i
FORMAT AND CONDUCT Context 35 We begin by examining the context in which consensus develop- ment conferences are conducted because the format of conferences may differ depending on the intended audience for the conference and what issues are addressed. The results are displayed in Table 1. Audience The potential audience for any of these conferences is fairly broad. In most cases, countries indicate that a variety of different groups are included in the target audience. We, however, tried to determine which audience was the primary target for each country in order to detect differences in the focus of planning and conference efforts. Six of the programs consider health professionals to be the audience for their consensus conferences (Canada ~ and II, Finland, The Neth eriands, Switzerland, and the United States). Within those programs, most seem to be addressing physicians who are involved in direct patient care; however, academic-based physicians and research scien- tists are also targeted. Two countries (Norway and Sweden) consider national health authorities, such as health planners and political decision makers, as well as health professionals to be the primary audience for the conference. Denmark and the United Kingdom indicate that the public and health professionals are their primary audiences; most countries suggest that the public may be interested in the findings. Switzerland is the only country that included third- party payers in its audience list. Issues The scope of issues addressed in any one conference varies con- siderably, and most countries appear to have a fairly broad perspec- tive. Countries typically indicate that they intend to address a vari- ety of issues, including safety, effectiveness, efficacy, appropriate- ness, service requirements, economics, political and social concerns, and needs for future research. The evidence that is available in any particular area limits the issues that can be addressed. Most consen- sus development conferences address issues of effectiveness and efficacy (the strength of the clinical literature), while few address economic or cost issues (rarely addressed in clinical or epidemio-
36 CONSENSUS DEVELOPMENT TABLE 1 Context in Which Consensus Development Conferences Are Conducted Context United States HUH) Canada I Who ~ the audience for consensus product? What iggucs does panel address? Who selects topics? No. of topics Health professionals; public Benefits; appropriate use of technology; future research Instituteg within NIH and OMAR 75 Practicing primary . . . care physicians; medical educators; policymakers; general public Appropriateness of procedures to prevent adverse outcomes; focus is on the condition rather than the procedure; effectiveness (benefit > harm?) or, if no information, efficacy Task Force on Periodic Health Examination 1
FORMAT AND CONDUCT Canada ~ Denmark Finland Practicing physicians Appropriateness benefit ~ risk Investigators from participating universities (caster, Univ. of Toronto, Univ. of Briush Columbia) 2 6 3 37 Public; health planners; health professionals Effecuiveness; safety; adoption level; economic, organi7~honal, and ethical implications Subcommittee of Danish Research Council Health care providers; researchers; public, providers of agricultural products (cholesterol) Safety; efficacy; effectiveness; . . service requirements; adoption level; economic and other (e.g., ethical) . .. . unpllcanals Medical Research Combs Table I continues
38 TABLE 1 Continued CONSENSUS DE~ELOPMEl1T Context The Netherlands Norway Sweden Who is the audience Clinicians; National health Politicians; for consensus hospital authorities; administrators; product? administrators; hospital owners; educators; policymakers health professionals planners; public Whatissues does Primary: safety, Effectiveness; Safety; panel address? efficacy, efficacy; efficacy; effectiveness; risks; effectiveness; also: cost- costs; cost/economic effectiveness, consequences: implications; service requirements organizational, minor role for psychologicaVsocial, ethical, legal, ethical social implications Who selects Scientific Council Technology Representatives from topics? of COO Assessment Medical Research Committee Council and Swedish Planning end Rationalization Institute of Health Services form a steering committee No. of topics 22 2 9
FORMAT AND CONDUCT Switzerland Physicians; health care administrators and decision makers third-part~r payers Safety; effectiveness; substitution versus replacement; appropriate uses; cost Planning committee 39 United Kingdom NationaUlocal policymakers; . . . c 1nlclans; health professionals; public Scientific merits of technology for different patients; cost; safety; effectiveness; legal, social, service, ethical . . . imp. Cations King's Fund steering group including . . . G mans, professional bodies, representatives of Dept. of Health 7 Degree of Variation among Countries Six focus on health professionals; one on public two on national politics or planners; United Kingdom considers all potential audience groups to be equally important Scope varies: most have global perspectives; few seem likely to truly address efficacy; four examine appropriateness; one examines whether technology is a replacement or ~ . substitute Most combine government and specialty society input (except United States and Canada II) 140+
40 CONSENSUS DEVELOPMENT logic studies). Four programs (Canada I and II, Switzerland, and the United States) address appropriateness of technology use. Topic Selection Most countries rely upon a combination of government and spe- cialty societies to select topics for consensus development confer- ences. The exceptions are the United States and Canada II. The United States relies upon staff within the government-funded NIH to select topics, but does not formally involve relevant U.S. medical specialty societies (Office of Medical Applications of Research, National Institutes of Health, 1988~. Canada lI selects topics based on interests of the principal investigators and the availability of fund- ing to explore various technologies; subsequent to topic selection, the researchers generally seek government funding and specialty society support. In most countries, topic selection comes from a standing committee responsible for technology assessment activities (e.g., Scientific Council of CBO in The Netherlands, Research Coun- cils in Denmark and Norway). Topics The variation in topic selection within any country depends in part on how many consensus conferences have been conducted. In the United States and The Netherlands, which have conducted more than 75 and 27 conferences, respectively, there is considerable vari- atic~n in the tulles of topics selected try r In general, across ad countries, four types of topics have been addressed: treatment of conditions (e.g., otitis media, schizophrenia), diagnostic procedures (e.g., de- tection of breast cancer, diagnosis of venous thrombosis), therapeutic procedures (e.g., adjuvant chemotherapy for breast cancer, coronary artery bypass surgery), and planning issues (e.g., need for asylum for the mentally ill, impact of routine HTEV-TII [human T-lym- photropic virus type ITI] antibody testing on the safety of the blood supply). Conferences have also considered technologies at varying stages of development, from new or emerging to established and outmoded. The stage of development is important both because it has implica- tions for the availability of evidence and because the degree of con-
FORMAT AND CONDUCT 41 troversy surrounding the technology may be related to its stage of development. Prepane' Process Although most countries spend between six months and one year planning for the consensus development conference, different types of activities are undertaken during that time, as shown in Table 2. Responsibility for Planning Three different types of groups have responsibility for planning the consensus conferences. In seven countries (Denmark, Finland, The Netherlands, Norway, Sweden, Switzerland, and the United Kingdom), planning is done by a special working group generally appointed by the committee responsible for selecting the topic. In two countries (Canada and the United States), staff are responsible for preparing for the conference. In the United States, employees of the relevant bureaus, divisions, or institutes within NTH conduct planning activities. In Canada II, planning is done by the academic- based research team. In one country (Canada D, planning is done by the consensus pane! itself. Review of Literature Most of the conferences rely upon oral presentations of scientific evidence as the basis for making judgments about the technology under consideration. In most countries, information is sent ahead of time to panel members to prepare them for the conference. For example, in the United Kingdom, pane] members receive introduc- tory textbook-type information (necessary for the lay members of the panel) as well as a comprehensive set of readings, drawn from a computer search of all relevant literature, and abstracts prepared by the expert speakers who will present information at the actual meet- ing. Canada ~ and Il and Switzerland prepare a formal synthesis of the literature in advance of the pane} meeting. For Canada I, which arrives at a consensus through several iterations of papers, the syn- thesis of the literature provides a starting point for developing the
42 CONSENSUS DEKELOPMENT TABLE 2 Types of Activities Undertaken during He Prepanel Process P'epanel Process United States (OH) Canada I Time spent in 12-15 months Ongoing process preparation for panel meeting? Who is responsible N1H staff: Task force members for planning end OMAR coordinator, preparation? bureau, institute, or division (BE)) coordinator, representatives from sponsor agencies Extent and nature of Bibliography and Original reports of literature review? paper abstracts sent clinical trials or in advance; epidemiologic studies; sometimes background quality of evidence reports prepared or is graded; expert summaries of expert opinion used state of science; if no evidence is BE coordinator detennines available scope of literature search Is there a review of For sane topics, Some efforts made to current clinical cum nt uses are measure patient practice (e.g., summarized compliance; efficacy, current working on rules for uses of die using data on technology)? efficacy (e.g., decision analysis techniques) Are patient outcome Both intermediate data used in and final outcomes planning? are considered; disease prevention is main outcome; examining broader range of outcome measures (e.g., quality of life) Are commendations or Specific questions prepared for panel in advance? Conference questions developed by planning committee (four to six questions with one pertaining to future research); structure of questions allows answers to be drawn strictly from scientific literature Part of background paper which is prepared by panel member and then revised over time
FORMAT AND CONDUCT 43 Canada II Denmark Finland 12 months Project staff responsible for preparation of materials and meeting planning Synthesis of literature; all relevant research examined; mutually conclusive indications for use of technology developed Dunug needs assessment phase, presented to panel before conference by analysts; describe claims data loo some extent in development of indications Indications for use of procedure are rated by panelists in advance of meeting; formal consensus questions prepared in advance based on needs assessment phase 6-10 months Planning committee: head of expert group, head of questioning panel, head of secretariat, one or two topic specialists; appointed by DMRC subcommittee Abstracts of papers to be presented are sent in advance; no mention of formal literature review For some topics, current uses are summarized Planning committee formulates main conference questions 8-12 months Planning C~ninee is nominated by Medical Research Council Data are presented by speakers and include reviews, controlled trials, epidemiologic studies, and state of science; abstracts of papers are sent to panelists in advance Cunent clinical practice used only . . Once In F .anmug phase to develop . . prep Plenary responses to questions; efficacy considered throughout Prepared responses to questions was attempted Ally once (for OhtiS media) Table 2 continues i
44 TABLE 2 Continued CONSENSUS DEVEWPME~T Prepanel Process The Netherlands Norway Sweden Time spent in preparation for panel meeting? 9-12 months 12 months 8-12 months Who is responsible All planning and Technology for planning and preparation? preparation done by the working group in collaboration with CBO staff; 5-10 meetings Assessment ~ittee is established by National Research Council Specially appointed working group Extent and nature of Syllabus is compiled; Randomized Reviews, RCTs, Literature review? experts present controlled teals epidemiologic evidence; few (RCTs) and studies, expert literature reviews compilation presentations; have been done; of scientific abstracts sent in evidence graded evidence (abstracts advance using Sackett of expert method; background testimony) papers use research plus clinical experience Is there a review of Ad hoc study before Experts asked to current clinical conference to present data on practice (e.g., document spread actual state of - efficacy, current and use of technology, practice uses of the cost, and health technology)? effects Are patient outcome Efforts being made Where available Not explicitly data used in to introduce planning? decision analysis techniques into conferences in the preparatory phase Are recommendations Draft consensus Yes or specific questions prepared for panel in advance? statement is prepared in advance and sent to conference participants No
FORMAT AND CONDUCT Switzerland United Kingdom Degree of Variation among Countries 14 months Planning committee drawn from Swiss Public Health Institute, Federal Office of Social Insurances, appropriate medical associations, and Sickness Funds · . . Assomatlon Synthesis of bibliographic references; full text of expert reports to panel four weeks prior to conference; experts given guidelines on how to synthesize literature. Data on magnetic resonance imaging was collected for one year through a registry Partially, although no validation or controlled trial data available 9-12 months Planing committee nominated by King's Fund Reading material sent to panel before the conference includes introductory texts, actual literature, and 1,000 word abstracts from speakers Small surveys done where relevant; experts asked to present data Where available Most seem to take about a year Two have staff; seven have working groups; one has a consensus panel Three have formal synthesis; three have oral testimony; one has bibliography; seven have abstracts sent in advance; one has reading material Six rely on expelt testimony; one asks panelists to use experience; one uses review in planning conference; four conduct special surveys Six use outcomes to some degree (usually based on availability); Ihe Netherlands working on incorporating patient outcome data through decision analysis techniques Specific questions Advance preparation Five have prepared 6 months of questions or recommendations in before conference; statement advance; six have draft of statement specifically questions in advance; prepared 2 weeks disallowed two have no advance before conference preparation; one has patient indications 45
46 CONSENSUS DEVELOPME~ consensus statement. Canada ~ has the most formal approach to rating the relationship between the recommendations made and the scientific merit of the literature on which those recommendations are based (Battista and Fletcher, 1988~. Those recommendations based on clinical trials receive the highest grade, whereas those recom- mendations based on the opinions of respected medical authorities alone are given a lower grade. Canada IT combines the NTH consensus development conference approach with a mode} developed for the RAND Corporation/Uni- versity of California at L`os Angeles (UCL~A) Health Services Utili- zation Study (Chassin et al., 1987; Lomas et al., 19881. The latter approach uses a synthesis of the literature as a starting point for developing mutually exclusive categories of patients who might be candidates for the technology under evaluation. These "indications" for technology use are then rated by the panel, using a two-tier Delphi approach, on a nine-point scale ranging from extremely ap- propriate (nine points) to extremely inappropriate (one point); val- ues in He midrange (four to six points) represent equivocal ratings of the use of the technology in those patients. Where no literature exists to inform the ratings of certain patient categories, expert opin- ion is used to expand the available information. In some cases, U.S. consensus development panels have had a synthesis prepared, but this varies from conference to conference. In eight countries (Denmark, Finland, The Netherlands, Norway, Sweden, Switzerland, the United Kingdom, and the United States), abstracts of papers that will be presented at the conference are sent to pane} members in advance. In general, there appears to be very little systematic effort to surrey the available literature and summarize the state of the science for the topic under consideration. Only infrequently is an attempt made to synthesize the literature and assess the scientific merit of the research. Further, in those countries that rely upon experts to present summaries or evaluations of scientific information, it is un- clear how the experts who will present testimony are selected and whether the group of experts fairly represents the range and distribu- tion of evidence in the literature. Most consensus statements do not refer to the literature upon which findings are based, which makes it difficult to determine whether and to what extent He literature has had an influence on the conference's conclusions and whether some literature is more influential than other. Most countries indicate that
FORMAT AND CONDUCT 47 they favor using results from randomized clinical trials and epidemi- ologic studies, but it is difficult to ascertain the extent to which this is true in actual practice. Review of Current Clinical Practice Most countries indicate that their consensus development confer- ences are intended to address issues related to the efficacy of certain medical technologies. Four countries (The Netherlands, Norway, Switzerland, and the United Kingdom) conduct special surveys to collect data on current uses of the technology. In Finland, current clinical practice experience was used once (for otitis media) in the planning phase to develop preliminary responses to questions; how- ever, this practice is not the rule. Experts are relied upon in six countries (Canada IT, Denmark, The Netherlands, Sweden, the United Kingdom, and the United States) to provide information on current practice. An evaluation of the U.S. experience, however, demon- strated that conference topics and subsequent recommendations ad- dressed issues, such as the discontinuation of Halstead radical mas- tectomies, that were largely obsolete in clinical practice prior to the actual conference (Kosecoff et al., 1987~. Canada ~ is working on decision-making rules for using data on efficacy and is considering how decision analysis might be used to expand the information avail- able with which to make decisions about technologies. Canada lI uses analyses of claims data for information on current uses. Can- ada IT also uses the experience of individual panel members to rep- resent what is occurring in clinical practices. In rating indications for the use of the technology, pane} members are asked to draw on their own experience and make ratings based on how they would approach treatment for each type of patient seen in their own prac- tice (Lomas et al., 1988~. Sometimes the experiences of panel members also result in changes to the structure or form of the indi- cations (Park et al., 19861. Patient Outcome Data Although many countries indicate that they assess the appropri- ateness and outcomes of use for a technology, the definitions of appropriate and outcome are not always explicit. Possible approaches to defining appropriateness include cost-benef~t, cost-effectiveness,
48 CONSENSUS DEVELOPMENT and nsk-benef~t analyses. In each case, information about outcomes experienced by different types of patients is necessary for a formal evaluation of appropnateness. Only Canada ~ has made an explicit effort to include data on patient outcomes in the decision-making process; both intermediate and final outcomes are considered, but the prevention of disease is the primary final outcome. Canada I is also examining the introduction of broader outcome measures (e.g., quality of life) into its consensus development process (Battista and Fletcher, 1988~. Canada I! uses a nsk-benefit definition of appropr~- ateness for structuring and rating indications for use of the technol- ogy Comas et al., 19SX). The Netherlands has indicated that efforts are under way to introduce decision analysis techniques into the conference planning process and, because the consensus develop- ment pane} itself also does the planning, such efforts would cer- tainly have the opportunity to affect pane} deliberations (Klazinga et al., 1987~. Other countries indicate that they use such data when they are available, but the extent to which this is done and the effect of such information on the consensus development process remains unclear. Advance Preparation of Recommendations or Questions Because of the time pressures surrounding the actual consensus development conference, we were interested in the extent to which recommendations or questions are prepared in advance in draft form as a starting point for pane} deliberations. In five countries (Canada IT, Denmark, Norway, Switzerland, and the United States), questions to be addressed by the pane} are prepared in advance. This provides the focus for the conference and presumably affects the selection of experts who will present evidence at the conference. In Finland, prepanel drafting of answers to questions was attempted only once to help manage a particularly controversial debate regarding treat- ment of otitis media. In Canada IT, as previously described, a for- mal set of indications is created. In four countries (Canada I, The Netherlands, Norway, and Switzeriand), some form of draft consen- sus statement is prepared in advance and is made final during the conference. Canada ~ uses an entirely different approach from the rest of the countries; it relies on a permanent pane} that is respon- sible for an ongoing process of writing papers evaluating the use of certain preventive measures and diagnostic procedures used in peri-
FORMAT AND CONDUCT 49 odic health examinations (Battista and Fletcher, 1988~. The panel writes and revises papers (the equivalent of consensus statements) based on deliberations at regularly scheduled pane} meetings; as long as two years may be spent on the preparation of these state- ments. In The Netherlands, panelists receive the draft consensus statement in advance of the meeting. In two countries (Sweden and the United Kingdom), no advance preparation of questions or rec- ommendations is undertaken; in the United Kingdom, such advance work is specifically disallowed. Pane' Composition The acceptability of consensus development conference recom- mendations depends to some degree on the qualifications of pane! members. We examined variations in the panel size, the process of selection, and qualifications of the panel chair and members. The results are displayed in Table 3. Pane! Size Panels range in size from 9 to 18 members. Panel size is an interesting consideration given the implications for the group pro- cess required to accomplish the task at hand. Groups of 18 will probably function differently than groups of 9; the likelihood of everyone participating equally in the decision-making process de- clines as the number of panelists increases. On the other hand, larger groups enhance the range of viewpoints that can be taken into account in considenng complex issues. Pane! Chair In general, the planning committee selects the pane} chair. The exception to this is in Sweden, where no chair is selected; leadership is shared by different persons drawn from the expert group (i.e., the leadership rotates depending on the issue under discussion). In most countries, the qualifications of panel chairs are fairly general, in- cluding the individual's stature as a scientist and leadership abili- ties. An evaluation of the process of selecting panel chairs in the United States suggests that this is done informally and is based on staff familiarity with individuals who are currently conducting re-
so CONSENSUS DEVELOPMENT TABLE 3 Consensus Development Conference Panel Composition Panel Composition United States (NIH) Canada I Number of panelists Who selects the panel chair? Background of the panel chair? Who selects panel members? Background of the panel members? 9-16 Director of sponsoring BID with advice from the planning committee Stature in field · . . as pnyslc~an or scientist; skills in conducting meetings; no established . . position an technology under consideration Planning conunittee Researchers; health professionals; epidemiologists/ biostatisticians; public (ethicists, economists, lawyers, theologians, consumers); moved from balanced to neutral panels Clinical epidemiologists from family practice, internal medicine, psychiatry, pediatrics, infectious disease, nursing. Allhave academic affiliations; balanced
FORMAT AND CONDUCT 51 Canada ~ Denmark Finland 10 12-16 16 Principal Subcor'unittee of Planning committee investigator Danish Medical Research Council Pr~xninent medical Distinguished doctor in specialty not related to technology being assessed Planning committee Seven physicians (five Physicians, joumalists, obstetricians, one general other academics, practitioner, one neonatologist); politicians, patient generalists and specialists; academic and community practice; usersk~onusers of technology, regional representative; three nonphysicians (one epidemiologist, one lawyer, one consumer); balanced researcher, skill in conducting meetings Planning committee Researchers; . . . c. ~ ens; representatives; none are working with technology being assessed; neutral other health care professionals; lay representatives (social scientists, editors) Table 3 continues
52 TABLE 3 Continued CONSENSUS DEVELOPMENT Panel Canposition The Netherlands Norway Sweden Number of panelists Who selects the panel chair? 1~15 Scientific Council of CBO 1~12 National Research Council's technology assessment committee Background of the Scientific expert Member of the panel chair? plying committee Who selects panel Scientific Council members? of CBO in Background of the panel members? 1~18 Leadership is shared bydifferentpersons generally drawn from the expert group Technology Steering committee Assessment consultation with Committee chair Scientific association representatives, experts. May add: nursing, genera;! practitioners, physiotherapists Half medical experts; half nonmedical experts (health economists, ethicists, lawyers, joumalists, health administrators, health politics); neutral Half medical experts; half from health, care profession economists, policy, epidemiologists, · . . aammlstrators, patients
FORMAT AND CONDUCT 53 Switzerland United Kingdan Degree of Variation among Countnes 14 Planning committee Skill in leading ~ . . group aeC~SlOn process; strong knowledge in specific technology area Pluming committee Researchers in field; health professionals (specialists and those in related fields); epidemiologists; health care . . . a=lstrators; third-party payers; balanced 12 King's Fund Creditable persons in different specialty (e.g., neurosurgeon for coronary artery bypass panel) King's Fund and planning committee Two specialists in topic; half medical (not technology users); half nonmedical (judicial model); neutral 9-18 In general, planning committee; Sweden does not have a chair Scientific stature; leadership ability (not really clear other than this) Panel selected generally by planning committee All use a combination of scientists, . . F Scans, and lay people; three balanced, four neutral, three unknown
54 CONSENSUS DEVELOPMENT search in the topic area (Wortman et al., 19X81. In the United States, in addition to stature and leadership requirements, the panel chair must have no established position on the technology under consid- eration. In one conference in Me United Kingdom, the panel chair was deliberately selected from a specialty that does not use the tech- nology being evaluated (e.g., a neurosurgeon chaired the coronary artery bypass panel). Denmark also selects an individual from a nonuser physician specialty to chair the panel. Pane! Members The pane} members are also generally selected by the planning committee, and it appears that the selection process is not system- atic, but rather relies upon the people with whom the planning com- mittee is familiar. In small countries, the informal approach may produce the same list of potential members as a more formal pro- cess. In all countries, the panels are composed of both scientists and laypeople. Generally, the panel distribution is half scientist and half laypeople. In the Canada IT panel on cesarean section, there were five obstetrician-gynecologists, one general practitioner, one neona- tologist, one epidemiologist, one lawyer, and one consumer (or 70 percent physicians and 30 percent other). Nonmedical experts tend to include health economists and policymakers, epidemiologists, administrators, and patients. In the United Kingdom, among the medical panelists, two were specialists in the topic area and the other four were nonusers of the technology. There are two approaches to the composition of panels: balanced and neutral. Balanced panels are designed to encompass the range of opinions in the field; experts representing different viewpoints are brought together on the pane} itself. The balanced panel has the advantage of well-informed members who can engage in an exchange of opinion. The disadvantage of balanced panels is that it may be difficult to find meaningful middle ground. A neutral panel is com- posed of people who do not have a stated opinion on the technology being evaluated; the neutral pane} is more like a jury who will weigh the evidence and come to a decision. The advantage of the neutral pane} is that its judgments will presumably be based upon the evi- dence presented, rather than being influenced by preexisting opin- ions or experiences of panelists. The disadvantage, particularly for conferences concerned with complex technologies, is that it may be
FORMAT AND CONDUCT 55 problematic to educate the pane} sufficiently to allow for informed judgments. Three programs endeavor to use balanced panels (Can- ada ~ and IT and Switzeriand). The rest of the countries have neutral panels or do not indicate their approach on this dimension. The United States has changed over time from balanced panels to neutral panels because of experiences with balanced panel meetings "degen- erating into unconstructive debate" (]acoby, 1985~. Consensus Pane! Meeting The processes for the conduct of panel meetings exhibit perhaps the least variation overall. Most countries have adopted the fonnat used by NIH with little concrete variation (Table 4~. Product In nine programs, the consensus panel is charged with producing a written statement of consensus on the issues addressed in the topic area. In Canada ~ the product is practice guidelines; in Canada II the product includes a consensus statement as well as ratings of indications for use of the procedure (e.g., ces are en section) in differ- ent types of patients. It does not appear that any country has de- tailed guidelines concerning the content and/or specificity of the consensus statement. The U.S. guidelines, for example, "encourage panelists to strive for statements that (~) recommend concrete, spe- cific actions; (2) differentiate patients into subclasses when appro- priate; and (3) offer didactic advice to the clinician on precise tech- niques that should be used" (Kanouse et al., 1989; MuBan and Jacoby, 19X5~. The level of detail in the statements we reviewed varies considerably (both between and within statements). Statements (with the exceptions of Canada ~ and Il) tend to make lithe or no reference to the literature on which the findings are based, and the types of recommendations vary from general to specific. Meeting Time Most countries follow the U.S. mode! of conferences meeting times, which last two and a half days, during which the pane} writes or modifies the consensus statement in an all-night session starting on the second day. The exceptions are Canada I (which relies upon
56 TABLE 4 Consensus Panel Meedng Processes CONSENSUS DEVELOPMENT Consensus Panel Meeting United States Em) Canada I What is the final Consensus statement; Synthesis of scientific product? press conference evidence; How long does the panel meet? Extent of public forums; number of audience members? Private panel sessions? How is consensus defined? Who makes the final decision? Are formal votes taken? What criteria are used for making decisions? Two and a half days (including nights) One to two days of evidence presented in public; audience: 20~700 Meet night before public formn; begin drafting statement after first day of evidence Panel, but the audience may offer suggestions No None How is disagreement ~ some cases a handled? minority report was prepared (rarely dme) recommendations for . .. . practice policies (guidelines for practice) Four to eight iterations over one to two years; one hour discussion ... . per condition per meeting None All work done in . . private meeting Task force No Value of scientific evidence based on predetermined . . cntena Not clear Sensitivity to group Not clear Not clear process issues?
FORMAT AND CONDUCT 57 Canada II Denmark Finland Indications rated for awropnateness, inappn~iateness, and equivalent uses of technology and consensus statement Two and a half days Presentations on unpublished data from 11 experts All work done in private meeting to produce an interim consensus statement Consensus statement Three days Public sessions on all three days; audience may ask questions; audience: 150 Expert group and panel meet night before; panel meets evening of first day; consensus statement prepared in closed session Unanimity Panel Questioning panel Panel Yes No No Benefit > risk None None Companson of Public debate in Not clear ratings indicates newspapers and level of agreanent professional journals after dissemination of results Voting is done Not clear Not clear anonymously; opinions known before panel chair can ensure these are brought up Consensus statement Two and a half days (including nights) Two days of plenary sessions with medical community, expert witnesses, public; audience: 15~160 One to two meetings before conference is held; meet night before plenary session Table 4 continues
58 TABLE 4 Continued CONSENSUS DEVELOPMENT Consensus The Netherlands Norway Panel Meeting Sweden What is the final Consensus statement product? How long does the One to two days panel meet? (including nights) Extent of public forums; Practitioners and number of audience public; members? audience: 150-1,000 Consensus statement Consensus press conference statement Two and a half days Two and a half days (through night on last day if necessary) Physicians, experts and public audience: ~200 One and a half days for expert Private panel Experts meet several Private sessions to Panel starts writing sessions? times ahead of the write the consensus on afternoon of actual conference statement second day How is consensus Chair formulates Panel sets the rules defined? guidelines Who makes the final Audience is asked if Panel Panel presents at decision? they agree; final press conference decision is with panel Are formal votes No No No taken? What criteria are None None None used for making decisions? How is disagreement Points of Not clear Press conference handled? disagreement are mentioned in consensus statement text Sensitivity to group Not clear Not clear Not clear process issues?
FORMAT AND CONDUCT 59 Switzerland United Kingdom Degree of Variation among Countries Consensus statement; summary presentation in of forum; press conference Two and a half days (no all night session) Consensus statement Three days, 21 hours; no overnight session Nine consensus statements; one practice guideline; one indication rated (plus consensus statement); many hold press conference to release statement Most are two and a half days and write through night Canada I is four to eight iterations over one to two years One day of evidence; Same as NIH; All except Canada I half day of discussion; audience: 200 rely on public foam audience: 150 Two half-day Executive session All write statements executive sessions; for writing in executive session; meet night before statement; some Canada II has plenary session revisions made after indications rated before public presentation and during meeting No formal voting; No clear definitions No clear definitions comments on draft Chairman Panel Panel makes decision in all but one case; public input in two; chairman decides in Switzerland No No No, except Canada II Value of scientific None evidence; confonnity with state of practice Detailed discussion; Not clear if no consensus, disagreement mentioned in final document Little experience handling Not clear problems in a nonhomogeneous group; panel did not question experts thoroughly Generally no . . . . . exp. lClt cntena used U.S., some minority reports; Denmark, public debate; most have press conferences; Canada II, ratings; Switzerland and lye Netherlands, mentioned in consensus statement
60 CONSENSUS DEVELOPMENT four to eight iterations of a paper over one to two years), Canada II (which relies upon rated indications from an iterative process using the RAND/UCLA modified Delphi method to form the basis for pane! recommendations), and The Netherlands (which meets 5 to 10 times before the final meeting at which the consensus statement is completed). Public Forums All programs except Canada I rely upon public forums during which evidence is presented. The audience size ranges from 150 to 1,000, with most audiences being about 200. In most cases, the audience may ask questions or make comments, although the degree to which this is done varies. It is not clear whether or not the audience contributes substantially to the process; rather, it appears that the audience is in attendance primarily to observe the proceed- ings. Private Pane! Sessions The bulk of work required to answer questions and to draft, re- vise, and complete the consensus statement is done in private ses- sions. Most panels meet the night before the conference begins to receive instructions about the process of the meeting itself and to hear We `'charge" to the panel (i.e., questions to be answered). In many cases, the questions to be addressed have already been made public (Norway). In Finland and The Netherlands, the panel con- venes prior to the actual conference. In Finland, one or two meet- ings are held; in The Netherlands, several meetings are held during the conference planning process. The familiarity of panelists with one another presumably affects the group process dynamics. This may be accomplished in some countries without preconference meet- ings because Me pane} members already know each other (Wortman et al., 1988). Definition of Consensus Little information exists on how each country defines consensus. The implied definition is unanimous agreement with the consensus
FORMAT AND CONDUCT 61 statement. Because unanimity is expected, there are few formal mechanisms available for dealing with disagreement (with the ex- ception of Canada II). The Netherlands and Sweden indicate that the chair and pane} (respectively) are responsible for formulating guidelines to define consensus, but whether that requirement intro- duces any more systematic consideration into the process is unclear. Most countries release consensus statements at press conferences and/or in journals, which presumably allows a forum for dissenters to present their opinions. In the United States, a minority report may be used if serious disagreement remains, although this option is rarely employed (Iacoby, INS). Switzerland and The Netherlands mention any remaining sources of disagreement in the consensus statement. Decision-Making Process In all but one country, the pane} makes the final consensus deci- sion; in Switzerland, the chair is responsible for final decisions. Public input is solicited in two countries (The Netherlands and the United States). No formal votes are taken in the process of writing the consensus statement. There are also no formal criteria for mak- ing decisions. In Canada I, there is a formal rating of the evidence using criteria that are predetermined, and the consensus statements are tied to this rating of the quality of evidence. Sensitivity to Group Process Issues There is little available evidence on the extent to which confer- ences are sensitive to group process issues. The size of many of the consensus panels suggests that, in the absence of concerted efforts, many panelists may not voice their opinions during discussion. The lack of formal decision-making criteria and formal voting suggests that undercurrents of dissent may go unrecognized. The practice of writing consensus statements in an aH-night session suggests that individuals may agree because they are simply too tired to continue disagreeing. The presentation of evidence only in oral form would seem to particularly disadvantage lay members or scientists in unre- lated fields who are unfamiliar with the literature and/or the science of the technology. This is likely to result in less participation in the
62 CONSENSUS DEVELOPMENT discussion by the generalists, potentially resulting in decision mak- ing that is biased by particularly knowledgeable or vocal pane} members. DISCUSSION Consensus development conferences are used in many countries to assess the uses of medical technologies. Most of the conferences are conducted using a variation of the mode] that was developed in the United States by NIH. In this paper we have described the pro- cesses by which such conferences are conducted and the extent of variation among several countries along a series of specific dimen- sions. In this section we discuss the major concerns that arose from our review. Link between Goals and Inputs Most countries indicate that they use consensus development con- ferences to address a variety of issues, including the safety, effec- tiveness, efficacy, appropriateness, and consequences (political, social, ethical) of medical care and technology. In most cases, however, there appears to be a gap between these goals and the information available upon which to base such judgments. Few countries undertake a systematic examination of the clinical or health services research literature in order to construct a synthesis of what is known from published sources about the use of a technol- ogy. If a synthesis is done, it rarely relies upon a formal meta- analysis or any quantitative process of combining evidence. There are several advantages to beginning the process of planning a consensus development conference by conducting a thorough lit- erature synthesis. First, the synthesis provides all panelists with a common starting point for discussions. Few individuals, even those who work with a technology in clinical practice or research settings, are completely informed or have synthesized and can remember all findings from the literature. Panelists may tend to remember the studies that support their respective positions and forget or disregard information that runs counter to their beliefs. The common ground provided by the synthesis may be particularly important for lay pan- elists or scientists from different disciplines; in most countries, about half the panelists are in such categories.
FORMAT AND CONDUCT 63 Second, the synthesis and the information about the state of clini- cal practice as it exists in a community setting provide "the facts" from which to build a set of questions to be addressed by a confer- ence. In one evaluation of the consensus development conference process in the United States, the authors found that at the beginning of the conference, panelists believed that the right questions were being asked about a technology, but that their opinions about the critical issues changed as evidence was presented during the course of the conference (Wortman et al., 1988~. Other research found that conferences focused on areas of clinical medicine believed to be in need of change when actual clinical practice had already been cor- rected (Kosecoff et al., 1987~. A synthesis of the literature provides a means of identifying not only what is known but also what is not known from published sources (Iacoby, 1988~. The missing pieces may be answered through presentations by experts based on research that has not yet been published or through a systematic evaluation of sources such as pa- tient medical records, use of decision analysis techniques, or expert opinion. In fact, we found that many countries conduct ad hoc surveys prior to a conference to collect data on current uses of the technology being evaluated. In addition, a synthesis, if done care- fully, can help combine conflicting information and focus the panel on the most critical concerns. For instance, the literature synthesis may reveal that studies supporting the use of a technology are based on data obtained from poorly conducted studies, while data that do not support technological use come from randomized controlled clini- cal trials. Third, the synthesis provides a systematic way of identifying the experts in the field. Previous evaluations have suggested that the selection of experts is not necessarily systematic (Wortman et al., 1988~. This is not meant to suggest that the informal network ap- proach results in the selection of worse (or even different) experts than a systematic approach, but that the informal approach is less scientifically defensible and has the potential for introducing unin- tended biases into the consensus development process. Finally, linking of each consensus statement to its supporting evi- dence would allow users of the consensus findings to understand the scientific bases for the recommendations and would make clear where judgments are based on clinical trials, epidemiological studies, ob- servational studies, or expert opinion. This might also enable panels
64 CONSENSUS DEVELOPMENT to make statements with varying degrees of certainty based on the science available to back the statement, which could, in turn, en- courage the discussion of controversial topics. Formalizing the Group Process Consensus development conferences appear to be run fairly infor- mally with respect to the criteria for making decisions, definition of consensus, formal voting or polling of pane} members, and handling disagreements. A more formal process might not change the out- comes substantially, but it would make the process more scientifi- cally defensible and replicable. An increase in the formality of the process might also allow for the devotion of more discussion time to controversial issues, if provisions are made for a range of consensus definitions. The group process might be enhanced if the discussion is divided into disagreements about underlying assumptions. For example, panelists could discuss the probabilities of positive and negative outcomes associated with the use of a technology versus the patient utilities associated with those outcomes. If a conference recom- mended doing procedure x in clinical situation y, the recommenda- tion should be based on achieving certain specified outcomes. If both good and bad outcomes result, it would be useful to know what weights patients place on those outcomes (e.g., how much risk of death a person is willing to take to gain a specified improvement in functioning). Finally, for those conferences that deal with economic or management issues, it should be made clear how these considera- tions modified the conference findings regarding clinical policy (e.g., the use of the technology is clinically acceptable but the financial implications are unacceptable). Although not stated explicitly, it appears that consensus is de- fined in most conferences as unanimity. It has been suggested that requiring unanimity may result in statements that represent the "lowest common denominator" of opinion. Allowing consensus to take on a meaning that is less restrictive than unanimity expands the range and type of issues that can be addressed. The most straightforward way of introducing this type of definition of consensus into the process is by taking formal votes throughout the development of the consensus statement. Voting also provides a mechanism that allows for disagreement without necessarily endangering the overall pro- cess.
FORMAT AND CONDUCT 65 Alternatively, polling could be used (as it is in juries where una- nimity is required) to ascertain how close the group is getting to full agreement. This type of polling throughout the process provides a means for ensuring that everyone can be heard. (The chair can ask those who disagree with a statement to state the source of their differences). Voting or polling can be done in two ways. The first is anony- mous (i.e., using secret ballots), where individuals are not identified with particular positions. The RAND/UCLA and Canada II confer- ences used this approach, allowing those who disagree some protec- tion from undue pressure to change a position. The second approach is public voting, which may be more intimidating from the stand- point of a group process, but which may allow the group to focus on unresolved problems and force dissenters to defend their positions. Further' voting can be done simply (i.e., yes or no) or on some scale that reflects the level of agreement or disagreement. Canada II, for example, uses the method developed at RAND/UCLA that asks panelists to rate indications for the use of a procedure on a nine point scale ranging from "extremely appropriate" to "extremely in- appropriate." Another approach is to have panelists rate their agree- ment with a particular statement or recommendation on, for example, a f~ve-point scale ranging from "strongly agree" to "strongly dis- agree." Obviously, yes/no votes provide a method that is simpler and perhaps easier to implement, to increase the certainty of adopt- ing or rejecting recommendations. The advantage of using scales is that they allow panelists somewhat more latitude in agreeing or dis- agreeing with recommendations. Further, this information can be used to suggest a hierarchy or degree of certitude of those recom- mendations that should be implemented immediately versus those recommendations that require further study or that do not have strong support from the panel. The RAND Coloration has developed soft- ware for quickly entering the votes (through secret ballot) and pro- cessing the ratings; thus, technology is available to allow for reason- ~ ~ · ~ . - ~ ~ ~ , ~ ~ , %,., anle implementation of the more complex approach. Writing the Consensus Statement Most countries appear to use the U.S. model when the panel writes the statement in an overnight session and presents the results the next morning. Although this has the benefit of producing a state- ment very promptly, it would seem to raise questions about the qual-
66 CONSENSUS DEVELOPMENT ity of the product. As sleep deprivation and clear thinking are rarely compatible, the danger arises that the consensus reflects the opin- ions of those with stamina rather than representing a thoughtful consideration and deliberation based on the evidence. Although the introduction of an additional day or two into the process raises im- plications concerning the availability of experts and increased costs, the gains in the quality of the consensus findings may be well worth the additional time. There may be alternative approaches for conserving the amount of available meeting time. Along with the literature review prepared in advance, some countries begin drafting the consensus statement before the meeting, giving panelists the opportunity to consider the issues and focus on those areas of particular disagreement or contro- versy. The Canada IT method developed at RAND/UCLA, for ex- ample, requires that panelists rate indications for the appropriate use of a procedure in advance using a modified Delphi method. Analy- sis of the first set of ratings indicates areas of potential disagree- ment, which are then given more time during the pane} meeting. By shifting some of the responsibility onto panelists to prepare for the conference, the use of the standard meeting time may be more effec- tively targeted to those activities for which the group process and interactions are important. One alternative is seen in The Nether- lands, where CBO staff help to complete the statements at a meeting that occurs one to three months after the consensus meeting devel- opment conference. This provides time for writing and reflection before concluding the process. CONCLUSIONS The processes by which consensus development conferences are conducted can affect the value and validity of the final product. Although the NTH mode} has been used extensively in other coun- tries, it has never been shown to produce replicable results or to be preferable to other models. Further, for all approaches, there is always the potential for improvement in the process. The continued use of consensus development conferences to achieve agreement on the state of the science for a particular technology suggests a need to enhance the inputs to the process and the methods by which issues and experts are identified. An improvement in the inputs to the process has several advantages and seems likely to enhance the final
FORMAT AND CONDUCT 67 product. Formalizing the process by which consensus conferences are conducted would enhance the credibility of the statements by making allowances for varying degrees of consensus and providing an outlet for disagreement that does not detract from the overall endeavor. Such formal mechanisms as voting or polling, whether done anonymously or not, may allow for more controversial issues to be considered. Finally,~ me current proclivity toward writing the consensus statement in an all-night session seems to be unnecessar- ily grueling and may have a negative effect on the final product. By improving inputs at the planning stages, it should be possible to develop schedules that allow for consideration of serious issues on a more humane time frame. REFERENCES Andreasen, P.B. 1988. Consensus conferences in different countries, aims and per- spectives. International Journal of Technology Assessment in Health Care 4:305- 308. Battista, R.N., and S.W. Fletcher. 1988. Making recommendations on preventive practices: Methodological issues. American Journal of Preventive Medicine 4~4~(Suppl.~:53-67. Calltorp, J. 1988. Consensus development conferences in Sweden: Effects on health policy and administration. International Journal of Technology Assessment in Health Care 4:75-88. Casparie, A.F., and J.J.E. van Everdingen. 1985. Consensus development confer- ences in The Netherlands. International Journal of Technology Assessment in Health Care 1 :905-912. Chassin, M.R., J. Kosecoff, R.E. Park, C.M. Winslow, K.L. Kahn, N.J. Merrick, J. Keesey, A. Fink, D.H. Solomon, and R.H. Brook. 1987. Does inappropriate use explain geographic variation in the use of health care services? A study of three procedures. Journal of the American Medical Association 258~18~:2533-2537. Jacoby, I. 1985. The consensus development program of the National Institutes of Health: Current practices and historical perspectives. International Journal of Technology Assessment in Health Care 1~2~:420432. Jacoby, I. 1988. Evidence and consensus. Journal of the American Medical Asso- ciation 259~20~:309. Jennett, B. 1985. First consensus development conference in United Kingdom: On coronary artery bypass grafting, commentary by the chairman of the conference. British Medical Journal 291:716-718. Johnsson, M. 1988. Evaluation of the consensus conference program in Sweden: Its impact on physicians. International Journal of Technology Assessment in Health Care 4:89-94. Kanouse, D.E., R H. Brook, J.D. Winkler, J. Kosecoff, S.H. Berry, G.M. Carter, J.P. Kahan, L. McCloskey, W.H. Rogers, C.M. Winslow, G.M. Anderson, L. Brodsley, A. Fink, and L. Meredith. 1989. Changing medical practice Trough technology
68 CONSENSUS DEVELOPMENT assessment: An evaluation of the NIH consensus development program. Santa Monica, Calif.:RAND Corporation. Klazinga, N.S., A.F. Casparie, and J.J.E. van Everdingen. 1987. Contribution of medical decision-making to consensus development conferences. Health Policy 8:339-346. Kosecoff, J., D.E. Kanouse, W.H. Rogers, L. McCloskey, C.M. Winslow, and R.H. Brook. 1987. Effects of the National Institutes of Health consensus development program and physician practice. Journal of the American Medical Association 258:2708-2713. Lomas, J., G.A. Anderson, M. Enkin, E. Vayda, R. Roberts, arid B. MacKinnon. 1988. The role of evidence in the consensus process, results from a Canadian consensus exercise. Journal of the American Medical Association 259~20~:3001- 3005. Mullan, F., and I. Jacoby, 1985. The town meeting for technology: The maturation of consensus conferences. Journal of the American Medical Association 254(B): 1068-1072. Office of Medical Applications of Research, National Institutes of Health. 1988. Guidelines for the selection and management of consensus development confer- ences. Bethesda, Md. Pamphlet. Park, R.E., C.M. Winslow, K.L. Kahn, N.J. Merrick, J. Keesey, A. Fink, D.H. Solomon, and R.H. Brook. 1986. Physician ratings of appropriate indications for six medical and surgical procedures. American Journal of Public Health 76~7~:766- 772. Perry, S. 1987. The NIH consensus development program a decade later. New England Journal of Medicine 317:485488. Perry, S. 1988. Consensus development: An historical note. International Journal of Technology Assessment in Health Care 4:481-484. Stocking, B. 1985. First consensus development conference in United Kingdom: On coronary artery bypass grafting, views of audience, panel, and speakers. British Medical Journal 291:713-716. yang, J. 1986. The consensus development conference and the European experi- ence. International Journal of Technology Assessment in Health Care 2:65-76. Wortman, P.M., A. Vinokur, and L. Sechrest. 1988. Do consensus conferences work.? A process evaluation of die NIH consensus development program. Jour- nal of Health Politics, Policy, and Law 13~3~(Fall):469498.
