Cover Image

PAPERBACK
$15.00



View/Hide Left Panel
Click for next page ( 33


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 32
Format and Conduct of Consensus Development Conferences: A Multination Comparison* Elizabeth A. McGlynn, Jacqueline Kosecoff, and Robert H. Brook INTRODUCTION The purpose of this paper is to compare the methods used in nine countries to organize and conduct consensus development confer- ences on scientific issues related to the delivery of medical care. Our comments focus primarily on the format and conduct of these conferences. However, we also briefly discuss the context, selec- tion, and scope of topics; the role of evidence; and the dissemination of results. We confine our remarks on these latter topics to descr~p- tive statements that illuminate issues relevant to a discussion of process. We address the process by which consensus development confer- ences are conducted because the approach taken may influence the results, or the acceptability of the results, either intentionally or unintentionally. Thus, at each step of the process it is important to consider whether the mechanics of conducting the conference are enhancing or detracting from achieving its goals and objectives. To the extent that the mechanism itself interferes with the intended outcomes, changes in the approach are warranted. *This paper is to be published in a forthcoming issue of the International Jour- nal of Technology Assessment in Health Care, Cambridge University Press, New York. 32

OCR for page 32
FORMAT AND CONDUCT METHODS 33 We set out to examine the process by which consensus develop- ment conferences are conducted in Canada, Denmark, Finland, The Netherlands, Norway, Sweden, Switzerland, the United Kingdom, and the United States. Although most of these countries began with the mode} used in the United States by the National Institutes of Health (NIH), many important variations have been introduced (Casparie and van Everdingen, 1985; Bennett, 1985; Perry, 1987, 19XX; Stocking, 1985; yang, 1986~. The 10 programs considered are: I. The Canadian Task Force on the Periodic Health Examina- tion (Canada I) 2. The Canadian Research Group (Canada II) 3. The Danish Medical Research Council and the Danish Hospi- tal Institute 4. The Medical Research Council of the Academy of Finland 5. National Organization for Quality Assurance in Hospitals in The Netherlands 6. The Norwegian Institute for Hospital Research and the Nor- wegian National Research Council 7. The Swedish Planning and Rationalization Institute for the Health and Social Services 8. The Swiss Institute of Public Health 9. The King's Fund Forum in the United Kingdom 10. The U.S. National Institutes of Health Office of Medical Ap- plications of Research (OMAR). The consensus development conference is a complex entity, and researchers have undertaken scientific evaluations of the conferences and their impact in their respective countries (Calitorp, 1988; Johnsson, 1988; Kanouse et al., 1989; Lomas et al., 1988; Worunan et al., 1988~. An international comparison is challenging because there may be as much or more variation within a particular country regarding how individual conferences are conducted as there is among countries (Andreasen, 1988~. For example, more than 75 such con- ferences have been held by NIH in the United States, and although the mode} was basically the same, implementation certainly varied across the conferences. We have attempted to use the most recent data available to represent each country in order to capture any

OCR for page 32
34 CONSENSUS DEVELOPMENT changes that have taken place over time in the conduct of the confer- ences. We used three sources of information to prepare this paper: I. Profiles prepared by representatives from each country for the June 1989 workshop on international consensus development for medical technology assessment organized by the Institute of Medi- cine 2. Published articles and consensus statements related to such conferences 3. Comments received after the conference from representatives of each country. We divided the consensus development process into four stages for our investigation: I. Context of the consensus development process 2. Prepane} process 3. Pane} composition 4. Consensus pane! meeting. The context within which consensus conferences are conducted encompasses the nature of the audience, the topics considered, and how the topics are selected. The prepare} process includes all ac- tivities that are required to stage the consensus conference, such as selecting the chair, panel members, and presenters as well as prepa- ration of background information. Pane} composition includes the type of panelists, their qualifications, and the process by which they are selected. The consensus panel meeting stage describes the ac- tivities at the actual conference, such as the use of public forums and private sessions, the type of information considered in arriving at consensus, and the group process by which consensus is actually achieved. RESULTS In this section, we describe how each of the countries described in this paper conducts consensus conferences. We have organized the section around the four major areas investigated: context, prepanel process, pane} composition, and the consensus panel meeting. Within each area we addressed several questions. Tables ~ through 4 sum- marize our findings. i

OCR for page 32
FORMAT AND CONDUCT Context 35 We begin by examining the context in which consensus develop- ment conferences are conducted because the format of conferences may differ depending on the intended audience for the conference and what issues are addressed. The results are displayed in Table 1. Audience The potential audience for any of these conferences is fairly broad. In most cases, countries indicate that a variety of different groups are included in the target audience. We, however, tried to determine which audience was the primary target for each country in order to detect differences in the focus of planning and conference efforts. Six of the programs consider health professionals to be the audience for their consensus conferences (Canada ~ and II, Finland, The Neth eriands, Switzerland, and the United States). Within those programs, most seem to be addressing physicians who are involved in direct patient care; however, academic-based physicians and research scien- tists are also targeted. Two countries (Norway and Sweden) consider national health authorities, such as health planners and political decision makers, as well as health professionals to be the primary audience for the conference. Denmark and the United Kingdom indicate that the public and health professionals are their primary audiences; most countries suggest that the public may be interested in the findings. Switzerland is the only country that included third- party payers in its audience list. Issues The scope of issues addressed in any one conference varies con- siderably, and most countries appear to have a fairly broad perspec- tive. Countries typically indicate that they intend to address a vari- ety of issues, including safety, effectiveness, efficacy, appropriate- ness, service requirements, economics, political and social concerns, and needs for future research. The evidence that is available in any particular area limits the issues that can be addressed. Most consen- sus development conferences address issues of effectiveness and efficacy (the strength of the clinical literature), while few address economic or cost issues (rarely addressed in clinical or epidemio-

OCR for page 32
36 CONSENSUS DEVELOPMENT TABLE 1 Context in Which Consensus Development Conferences Are Conducted Context United States HUH) Canada I Who ~ the audience for consensus product? What iggucs does panel address? Who selects topics? No. of topics Health professionals; public Benefits; appropriate use of technology; future research Instituteg within NIH and OMAR 75 Practicing primary . . . care physicians; medical educators; policymakers; general public Appropriateness of procedures to prevent adverse outcomes; focus is on the condition rather than the procedure; effectiveness (benefit > harm?) or, if no information, efficacy Task Force on Periodic Health Examination 1

OCR for page 32
FORMAT AND CONDUCT Canada ~ Denmark Finland Practicing physicians Appropriateness benefit ~ risk Investigators from participating universities (caster, Univ. of Toronto, Univ. of Briush Columbia) 2 6 3 37 Public; health planners; health professionals Effecuiveness; safety; adoption level; economic, organi7~honal, and ethical implications Subcommittee of Danish Research Council Health care providers; researchers; public, providers of agricultural products (cholesterol) Safety; efficacy; effectiveness; . . service requirements; adoption level; economic and other (e.g., ethical) . .. . unpllcanals Medical Research Combs Table I continues

OCR for page 32
38 TABLE 1 Continued CONSENSUS DE~ELOPMEl1T Context The Netherlands Norway Sweden Who is the audience Clinicians; National health Politicians; for consensus hospital authorities; administrators; product? administrators; hospital owners; educators; policymakers health professionals planners; public Whatissues does Primary: safety, Effectiveness; Safety; panel address? efficacy, efficacy; efficacy; effectiveness; risks; effectiveness; also: cost- costs; cost/economic effectiveness, consequences: implications; service requirements organizational, minor role for psychologicaVsocial, ethical, legal, ethical social implications Who selects Scientific Council Technology Representatives from topics? of COO Assessment Medical Research Committee Council and Swedish Planning end Rationalization Institute of Health Services form a steering committee No. of topics 22 2 9

OCR for page 32
FORMAT AND CONDUCT Switzerland Physicians; health care administrators and decision makers third-part~r payers Safety; effectiveness; substitution versus replacement; appropriate uses; cost Planning committee 39 United Kingdom NationaUlocal policymakers; . . . c 1nlclans; health professionals; public Scientific merits of technology for different patients; cost; safety; effectiveness; legal, social, service, ethical . . . imp. Cations King's Fund steering group including . . . G mans, professional bodies, representatives of Dept. of Health 7 Degree of Variation among Countries Six focus on health professionals; one on public two on national politics or planners; United Kingdom considers all potential audience groups to be equally important Scope varies: most have global perspectives; few seem likely to truly address efficacy; four examine appropriateness; one examines whether technology is a replacement or ~ . substitute Most combine government and specialty society input (except United States and Canada II) 140+

OCR for page 32
40 CONSENSUS DEVELOPMENT logic studies). Four programs (Canada I and II, Switzerland, and the United States) address appropriateness of technology use. Topic Selection Most countries rely upon a combination of government and spe- cialty societies to select topics for consensus development confer- ences. The exceptions are the United States and Canada II. The United States relies upon staff within the government-funded NIH to select topics, but does not formally involve relevant U.S. medical specialty societies (Office of Medical Applications of Research, National Institutes of Health, 1988~. Canada lI selects topics based on interests of the principal investigators and the availability of fund- ing to explore various technologies; subsequent to topic selection, the researchers generally seek government funding and specialty society support. In most countries, topic selection comes from a standing committee responsible for technology assessment activities (e.g., Scientific Council of CBO in The Netherlands, Research Coun- cils in Denmark and Norway). Topics The variation in topic selection within any country depends in part on how many consensus conferences have been conducted. In the United States and The Netherlands, which have conducted more than 75 and 27 conferences, respectively, there is considerable vari- atic~n in the tulles of topics selected try r In general, across ad countries, four types of topics have been addressed: treatment of conditions (e.g., otitis media, schizophrenia), diagnostic procedures (e.g., de- tection of breast cancer, diagnosis of venous thrombosis), therapeutic procedures (e.g., adjuvant chemotherapy for breast cancer, coronary artery bypass surgery), and planning issues (e.g., need for asylum for the mentally ill, impact of routine HTEV-TII [human T-lym- photropic virus type ITI] antibody testing on the safety of the blood supply). Conferences have also considered technologies at varying stages of development, from new or emerging to established and outmoded. The stage of development is important both because it has implica- tions for the availability of evidence and because the degree of con-

OCR for page 32
FORMAT AND CONDUCT 41 troversy surrounding the technology may be related to its stage of development. Prepane' Process Although most countries spend between six months and one year planning for the consensus development conference, different types of activities are undertaken during that time, as shown in Table 2. Responsibility for Planning Three different types of groups have responsibility for planning the consensus conferences. In seven countries (Denmark, Finland, The Netherlands, Norway, Sweden, Switzerland, and the United Kingdom), planning is done by a special working group generally appointed by the committee responsible for selecting the topic. In two countries (Canada and the United States), staff are responsible for preparing for the conference. In the United States, employees of the relevant bureaus, divisions, or institutes within NTH conduct planning activities. In Canada II, planning is done by the academic- based research team. In one country (Canada D, planning is done by the consensus pane! itself. Review of Literature Most of the conferences rely upon oral presentations of scientific evidence as the basis for making judgments about the technology under consideration. In most countries, information is sent ahead of time to panel members to prepare them for the conference. For example, in the United Kingdom, pane] members receive introduc- tory textbook-type information (necessary for the lay members of the panel) as well as a comprehensive set of readings, drawn from a computer search of all relevant literature, and abstracts prepared by the expert speakers who will present information at the actual meet- ing. Canada ~ and Il and Switzerland prepare a formal synthesis of the literature in advance of the pane} meeting. For Canada I, which arrives at a consensus through several iterations of papers, the syn- thesis of the literature provides a starting point for developing the

OCR for page 32
42 CONSENSUS DEKELOPMENT TABLE 2 Types of Activities Undertaken during He Prepanel Process P'epanel Process United States (OH) Canada I Time spent in 12-15 months Ongoing process preparation for panel meeting? Who is responsible N1H staff: Task force members for planning end OMAR coordinator, preparation? bureau, institute, or division (BE)) coordinator, representatives from sponsor agencies Extent and nature of Bibliography and Original reports of literature review? paper abstracts sent clinical trials or in advance; epidemiologic studies; sometimes background quality of evidence reports prepared or is graded; expert summaries of expert opinion used state of science; if no evidence is BE coordinator detennines available scope of literature search Is there a review of For sane topics, Some efforts made to current clinical cum nt uses are measure patient practice (e.g., summarized compliance; efficacy, current working on rules for uses of die using data on technology)? efficacy (e.g., decision analysis techniques) Are patient outcome Both intermediate data used in and final outcomes planning? are considered; disease prevention is main outcome; examining broader range of outcome measures (e.g., quality of life) Are commendations or Specific questions prepared for panel in advance? Conference questions developed by planning committee (four to six questions with one pertaining to future research); structure of questions allows answers to be drawn strictly from scientific literature Part of background paper which is prepared by panel member and then revised over time

OCR for page 32
58 TABLE 4 Continued CONSENSUS DEVELOPMENT Consensus The Netherlands Norway Panel Meeting Sweden What is the final Consensus statement product? How long does the One to two days panel meet? (including nights) Extent of public forums; Practitioners and number of audience public; members? audience: 150-1,000 Consensus statement Consensus press conference statement Two and a half days Two and a half days (through night on last day if necessary) Physicians, experts and public audience: ~200 One and a half days for expert Private panel Experts meet several Private sessions to Panel starts writing sessions? times ahead of the write the consensus on afternoon of actual conference statement second day How is consensus Chair formulates Panel sets the rules defined? guidelines Who makes the final Audience is asked if Panel Panel presents at decision? they agree; final press conference decision is with panel Are formal votes No No No taken? What criteria are None None None used for making decisions? How is disagreement Points of Not clear Press conference handled? disagreement are mentioned in consensus statement text Sensitivity to group Not clear Not clear Not clear process issues?

OCR for page 32
FORMAT AND CONDUCT 59 Switzerland United Kingdom Degree of Variation among Countries Consensus statement; summary presentation in of forum; press conference Two and a half days (no all night session) Consensus statement Three days, 21 hours; no overnight session Nine consensus statements; one practice guideline; one indication rated (plus consensus statement); many hold press conference to release statement Most are two and a half days and write through night Canada I is four to eight iterations over one to two years One day of evidence; Same as NIH; All except Canada I half day of discussion; audience: 200 rely on public foam audience: 150 Two half-day Executive session All write statements executive sessions; for writing in executive session; meet night before statement; some Canada II has plenary session revisions made after indications rated before public presentation and during meeting No formal voting; No clear definitions No clear definitions comments on draft Chairman Panel Panel makes decision in all but one case; public input in two; chairman decides in Switzerland No No No, except Canada II Value of scientific None evidence; confonnity with state of practice Detailed discussion; Not clear if no consensus, disagreement mentioned in final document Little experience handling Not clear problems in a nonhomogeneous group; panel did not question experts thoroughly Generally no . . . . . exp. lClt cntena used U.S., some minority reports; Denmark, public debate; most have press conferences; Canada II, ratings; Switzerland and lye Netherlands, mentioned in consensus statement

OCR for page 32
60 CONSENSUS DEVELOPMENT four to eight iterations of a paper over one to two years), Canada II (which relies upon rated indications from an iterative process using the RAND/UCLA modified Delphi method to form the basis for pane! recommendations), and The Netherlands (which meets 5 to 10 times before the final meeting at which the consensus statement is completed). Public Forums All programs except Canada I rely upon public forums during which evidence is presented. The audience size ranges from 150 to 1,000, with most audiences being about 200. In most cases, the audience may ask questions or make comments, although the degree to which this is done varies. It is not clear whether or not the audience contributes substantially to the process; rather, it appears that the audience is in attendance primarily to observe the proceed- ings. Private Pane! Sessions The bulk of work required to answer questions and to draft, re- vise, and complete the consensus statement is done in private ses- sions. Most panels meet the night before the conference begins to receive instructions about the process of the meeting itself and to hear We `'charge" to the panel (i.e., questions to be answered). In many cases, the questions to be addressed have already been made public (Norway). In Finland and The Netherlands, the panel con- venes prior to the actual conference. In Finland, one or two meet- ings are held; in The Netherlands, several meetings are held during the conference planning process. The familiarity of panelists with one another presumably affects the group process dynamics. This may be accomplished in some countries without preconference meet- ings because Me pane} members already know each other (Wortman et al., 1988). Definition of Consensus Little information exists on how each country defines consensus. The implied definition is unanimous agreement with the consensus

OCR for page 32
FORMAT AND CONDUCT 61 statement. Because unanimity is expected, there are few formal mechanisms available for dealing with disagreement (with the ex- ception of Canada II). The Netherlands and Sweden indicate that the chair and pane} (respectively) are responsible for formulating guidelines to define consensus, but whether that requirement intro- duces any more systematic consideration into the process is unclear. Most countries release consensus statements at press conferences and/or in journals, which presumably allows a forum for dissenters to present their opinions. In the United States, a minority report may be used if serious disagreement remains, although this option is rarely employed (Iacoby, INS). Switzerland and The Netherlands mention any remaining sources of disagreement in the consensus statement. Decision-Making Process In all but one country, the pane} makes the final consensus deci- sion; in Switzerland, the chair is responsible for final decisions. Public input is solicited in two countries (The Netherlands and the United States). No formal votes are taken in the process of writing the consensus statement. There are also no formal criteria for mak- ing decisions. In Canada I, there is a formal rating of the evidence using criteria that are predetermined, and the consensus statements are tied to this rating of the quality of evidence. Sensitivity to Group Process Issues There is little available evidence on the extent to which confer- ences are sensitive to group process issues. The size of many of the consensus panels suggests that, in the absence of concerted efforts, many panelists may not voice their opinions during discussion. The lack of formal decision-making criteria and formal voting suggests that undercurrents of dissent may go unrecognized. The practice of writing consensus statements in an aH-night session suggests that individuals may agree because they are simply too tired to continue disagreeing. The presentation of evidence only in oral form would seem to particularly disadvantage lay members or scientists in unre- lated fields who are unfamiliar with the literature and/or the science of the technology. This is likely to result in less participation in the

OCR for page 32
62 CONSENSUS DEVELOPMENT discussion by the generalists, potentially resulting in decision mak- ing that is biased by particularly knowledgeable or vocal pane} members. DISCUSSION Consensus development conferences are used in many countries to assess the uses of medical technologies. Most of the conferences are conducted using a variation of the mode] that was developed in the United States by NIH. In this paper we have described the pro- cesses by which such conferences are conducted and the extent of variation among several countries along a series of specific dimen- sions. In this section we discuss the major concerns that arose from our review. Link between Goals and Inputs Most countries indicate that they use consensus development con- ferences to address a variety of issues, including the safety, effec- tiveness, efficacy, appropriateness, and consequences (political, social, ethical) of medical care and technology. In most cases, however, there appears to be a gap between these goals and the information available upon which to base such judgments. Few countries undertake a systematic examination of the clinical or health services research literature in order to construct a synthesis of what is known from published sources about the use of a technol- ogy. If a synthesis is done, it rarely relies upon a formal meta- analysis or any quantitative process of combining evidence. There are several advantages to beginning the process of planning a consensus development conference by conducting a thorough lit- erature synthesis. First, the synthesis provides all panelists with a common starting point for discussions. Few individuals, even those who work with a technology in clinical practice or research settings, are completely informed or have synthesized and can remember all findings from the literature. Panelists may tend to remember the studies that support their respective positions and forget or disregard information that runs counter to their beliefs. The common ground provided by the synthesis may be particularly important for lay pan- elists or scientists from different disciplines; in most countries, about half the panelists are in such categories.

OCR for page 32
FORMAT AND CONDUCT 63 Second, the synthesis and the information about the state of clini- cal practice as it exists in a community setting provide "the facts" from which to build a set of questions to be addressed by a confer- ence. In one evaluation of the consensus development conference process in the United States, the authors found that at the beginning of the conference, panelists believed that the right questions were being asked about a technology, but that their opinions about the critical issues changed as evidence was presented during the course of the conference (Wortman et al., 1988~. Other research found that conferences focused on areas of clinical medicine believed to be in need of change when actual clinical practice had already been cor- rected (Kosecoff et al., 1987~. A synthesis of the literature provides a means of identifying not only what is known but also what is not known from published sources (Iacoby, 1988~. The missing pieces may be answered through presentations by experts based on research that has not yet been published or through a systematic evaluation of sources such as pa- tient medical records, use of decision analysis techniques, or expert opinion. In fact, we found that many countries conduct ad hoc surveys prior to a conference to collect data on current uses of the technology being evaluated. In addition, a synthesis, if done care- fully, can help combine conflicting information and focus the panel on the most critical concerns. For instance, the literature synthesis may reveal that studies supporting the use of a technology are based on data obtained from poorly conducted studies, while data that do not support technological use come from randomized controlled clini- cal trials. Third, the synthesis provides a systematic way of identifying the experts in the field. Previous evaluations have suggested that the selection of experts is not necessarily systematic (Wortman et al., 1988~. This is not meant to suggest that the informal network ap- proach results in the selection of worse (or even different) experts than a systematic approach, but that the informal approach is less scientifically defensible and has the potential for introducing unin- tended biases into the consensus development process. Finally, linking of each consensus statement to its supporting evi- dence would allow users of the consensus findings to understand the scientific bases for the recommendations and would make clear where judgments are based on clinical trials, epidemiological studies, ob- servational studies, or expert opinion. This might also enable panels

OCR for page 32
64 CONSENSUS DEVELOPMENT to make statements with varying degrees of certainty based on the science available to back the statement, which could, in turn, en- courage the discussion of controversial topics. Formalizing the Group Process Consensus development conferences appear to be run fairly infor- mally with respect to the criteria for making decisions, definition of consensus, formal voting or polling of pane} members, and handling disagreements. A more formal process might not change the out- comes substantially, but it would make the process more scientifi- cally defensible and replicable. An increase in the formality of the process might also allow for the devotion of more discussion time to controversial issues, if provisions are made for a range of consensus definitions. The group process might be enhanced if the discussion is divided into disagreements about underlying assumptions. For example, panelists could discuss the probabilities of positive and negative outcomes associated with the use of a technology versus the patient utilities associated with those outcomes. If a conference recom- mended doing procedure x in clinical situation y, the recommenda- tion should be based on achieving certain specified outcomes. If both good and bad outcomes result, it would be useful to know what weights patients place on those outcomes (e.g., how much risk of death a person is willing to take to gain a specified improvement in functioning). Finally, for those conferences that deal with economic or management issues, it should be made clear how these considera- tions modified the conference findings regarding clinical policy (e.g., the use of the technology is clinically acceptable but the financial implications are unacceptable). Although not stated explicitly, it appears that consensus is de- fined in most conferences as unanimity. It has been suggested that requiring unanimity may result in statements that represent the "lowest common denominator" of opinion. Allowing consensus to take on a meaning that is less restrictive than unanimity expands the range and type of issues that can be addressed. The most straightforward way of introducing this type of definition of consensus into the process is by taking formal votes throughout the development of the consensus statement. Voting also provides a mechanism that allows for disagreement without necessarily endangering the overall pro- cess.

OCR for page 32
FORMAT AND CONDUCT 65 Alternatively, polling could be used (as it is in juries where una- nimity is required) to ascertain how close the group is getting to full agreement. This type of polling throughout the process provides a means for ensuring that everyone can be heard. (The chair can ask those who disagree with a statement to state the source of their differences). Voting or polling can be done in two ways. The first is anony- mous (i.e., using secret ballots), where individuals are not identified with particular positions. The RAND/UCLA and Canada II confer- ences used this approach, allowing those who disagree some protec- tion from undue pressure to change a position. The second approach is public voting, which may be more intimidating from the stand- point of a group process, but which may allow the group to focus on unresolved problems and force dissenters to defend their positions. Further' voting can be done simply (i.e., yes or no) or on some scale that reflects the level of agreement or disagreement. Canada II, for example, uses the method developed at RAND/UCLA that asks panelists to rate indications for the use of a procedure on a nine point scale ranging from "extremely appropriate" to "extremely in- appropriate." Another approach is to have panelists rate their agree- ment with a particular statement or recommendation on, for example, a f~ve-point scale ranging from "strongly agree" to "strongly dis- agree." Obviously, yes/no votes provide a method that is simpler and perhaps easier to implement, to increase the certainty of adopt- ing or rejecting recommendations. The advantage of using scales is that they allow panelists somewhat more latitude in agreeing or dis- agreeing with recommendations. Further, this information can be used to suggest a hierarchy or degree of certitude of those recom- mendations that should be implemented immediately versus those recommendations that require further study or that do not have strong support from the panel. The RAND Coloration has developed soft- ware for quickly entering the votes (through secret ballot) and pro- cessing the ratings; thus, technology is available to allow for reason- ~ ~ ~ . - ~ ~ ~ , ~ ~ , %,., anle implementation of the more complex approach. Writing the Consensus Statement Most countries appear to use the U.S. model when the panel writes the statement in an overnight session and presents the results the next morning. Although this has the benefit of producing a state- ment very promptly, it would seem to raise questions about the qual-

OCR for page 32
66 CONSENSUS DEVELOPMENT ity of the product. As sleep deprivation and clear thinking are rarely compatible, the danger arises that the consensus reflects the opin- ions of those with stamina rather than representing a thoughtful consideration and deliberation based on the evidence. Although the introduction of an additional day or two into the process raises im- plications concerning the availability of experts and increased costs, the gains in the quality of the consensus findings may be well worth the additional time. There may be alternative approaches for conserving the amount of available meeting time. Along with the literature review prepared in advance, some countries begin drafting the consensus statement before the meeting, giving panelists the opportunity to consider the issues and focus on those areas of particular disagreement or contro- versy. The Canada IT method developed at RAND/UCLA, for ex- ample, requires that panelists rate indications for the appropriate use of a procedure in advance using a modified Delphi method. Analy- sis of the first set of ratings indicates areas of potential disagree- ment, which are then given more time during the pane} meeting. By shifting some of the responsibility onto panelists to prepare for the conference, the use of the standard meeting time may be more effec- tively targeted to those activities for which the group process and interactions are important. One alternative is seen in The Nether- lands, where CBO staff help to complete the statements at a meeting that occurs one to three months after the consensus meeting devel- opment conference. This provides time for writing and reflection before concluding the process. CONCLUSIONS The processes by which consensus development conferences are conducted can affect the value and validity of the final product. Although the NTH mode} has been used extensively in other coun- tries, it has never been shown to produce replicable results or to be preferable to other models. Further, for all approaches, there is always the potential for improvement in the process. The continued use of consensus development conferences to achieve agreement on the state of the science for a particular technology suggests a need to enhance the inputs to the process and the methods by which issues and experts are identified. An improvement in the inputs to the process has several advantages and seems likely to enhance the final

OCR for page 32
FORMAT AND CONDUCT 67 product. Formalizing the process by which consensus conferences are conducted would enhance the credibility of the statements by making allowances for varying degrees of consensus and providing an outlet for disagreement that does not detract from the overall endeavor. Such formal mechanisms as voting or polling, whether done anonymously or not, may allow for more controversial issues to be considered. Finally,~ me current proclivity toward writing the consensus statement in an all-night session seems to be unnecessar- ily grueling and may have a negative effect on the final product. By improving inputs at the planning stages, it should be possible to develop schedules that allow for consideration of serious issues on a more humane time frame. REFERENCES Andreasen, P.B. 1988. Consensus conferences in different countries, aims and per- spectives. International Journal of Technology Assessment in Health Care 4:305- 308. Battista, R.N., and S.W. Fletcher. 1988. Making recommendations on preventive practices: Methodological issues. American Journal of Preventive Medicine 4~4~(Suppl.~:53-67. Calltorp, J. 1988. Consensus development conferences in Sweden: Effects on health policy and administration. International Journal of Technology Assessment in Health Care 4:75-88. Casparie, A.F., and J.J.E. van Everdingen. 1985. Consensus development confer- ences in The Netherlands. International Journal of Technology Assessment in Health Care 1 :905-912. Chassin, M.R., J. Kosecoff, R.E. Park, C.M. Winslow, K.L. Kahn, N.J. Merrick, J. Keesey, A. Fink, D.H. Solomon, and R.H. Brook. 1987. Does inappropriate use explain geographic variation in the use of health care services? A study of three procedures. Journal of the American Medical Association 258~18~:2533-2537. Jacoby, I. 1985. The consensus development program of the National Institutes of Health: Current practices and historical perspectives. International Journal of Technology Assessment in Health Care 1~2~:420432. Jacoby, I. 1988. Evidence and consensus. Journal of the American Medical Asso- ciation 259~20~:309. Jennett, B. 1985. First consensus development conference in United Kingdom: On coronary artery bypass grafting, commentary by the chairman of the conference. British Medical Journal 291:716-718. Johnsson, M. 1988. Evaluation of the consensus conference program in Sweden: Its impact on physicians. International Journal of Technology Assessment in Health Care 4:89-94. Kanouse, D.E., R H. Brook, J.D. Winkler, J. Kosecoff, S.H. Berry, G.M. Carter, J.P. Kahan, L. McCloskey, W.H. Rogers, C.M. Winslow, G.M. Anderson, L. Brodsley, A. Fink, and L. Meredith. 1989. Changing medical practice Trough technology

OCR for page 32
68 CONSENSUS DEVELOPMENT assessment: An evaluation of the NIH consensus development program. Santa Monica, Calif.:RAND Corporation. Klazinga, N.S., A.F. Casparie, and J.J.E. van Everdingen. 1987. Contribution of medical decision-making to consensus development conferences. Health Policy 8:339-346. Kosecoff, J., D.E. Kanouse, W.H. Rogers, L. McCloskey, C.M. Winslow, and R.H. Brook. 1987. Effects of the National Institutes of Health consensus development program and physician practice. Journal of the American Medical Association 258:2708-2713. Lomas, J., G.A. Anderson, M. Enkin, E. Vayda, R. Roberts, arid B. MacKinnon. 1988. The role of evidence in the consensus process, results from a Canadian consensus exercise. Journal of the American Medical Association 259~20~:3001- 3005. Mullan, F., and I. Jacoby, 1985. The town meeting for technology: The maturation of consensus conferences. Journal of the American Medical Association 254(B): 1068-1072. Office of Medical Applications of Research, National Institutes of Health. 1988. Guidelines for the selection and management of consensus development confer- ences. Bethesda, Md. Pamphlet. Park, R.E., C.M. Winslow, K.L. Kahn, N.J. Merrick, J. Keesey, A. Fink, D.H. Solomon, and R.H. Brook. 1986. Physician ratings of appropriate indications for six medical and surgical procedures. American Journal of Public Health 76~7~:766- 772. Perry, S. 1987. The NIH consensus development program a decade later. New England Journal of Medicine 317:485488. Perry, S. 1988. Consensus development: An historical note. International Journal of Technology Assessment in Health Care 4:481-484. Stocking, B. 1985. First consensus development conference in United Kingdom: On coronary artery bypass grafting, views of audience, panel, and speakers. British Medical Journal 291:713-716. yang, J. 1986. The consensus development conference and the European experi- ence. International Journal of Technology Assessment in Health Care 2:65-76. Wortman, P.M., A. Vinokur, and L. Sechrest. 1988. Do consensus conferences work.? A process evaluation of die NIH consensus development program. Jour- nal of Health Politics, Policy, and Law 13~3~(Fall):469498.