Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Appendixes
A Background Paper for the Committee to Improve the NTH Consensus Development Program* INTRODUCTION The purpose of this paper is to offer the Committee to Improve the National Institutes of Health Consensus Development Program uniform background information on the National Institutes of Health (NTH) Consensus Development Program (CDP). The committee's report represents one component of a three-part examination of group judgment methods for assessing medical technologies being con- ducted by the Council on Health Care Technology. The following synthesis of available literature about the program is intended to provide a common base of understanding for the ex- amination of the NTH CDP. Following a brief history of the NIH CDP, a general overview of the planning and implementation of consensus development conferences is provided. The paper then reviews the available literature concerning selected major compo- nents of the CDP. Each component is addressed in the order that the NIH CDP is conducted. *This paper was prepared by Sharon R. Baratz, Council on Health Care Technol- ogy, Institute of Medicine. 41
42 ORIGIN OF THE NIH CONSENSUS DEVELOPMENT PROGRAM APPENDIX A Throughout the 1970s, the acceleration of technological innova- tion in medicine, accompanied by rising costs and increased con- cerns for the quality of care, generated extensive interest in ~echnol- ogy assessment. Donald Fredrickson, then director of NIH, thought that the biomedical research community had a significant role to play in the evaluation of new and existing health care technologies. In October of 197X, the NIH formally established the Office of Medical Applications of Research (OMAR) to act as a link between the medical research community, clinical physicians, and the public. The CDP was established as one component of the technology assessment and information-sharing activities of NTH (Perry and Kalberer, 1980~. Although variations have occurred, the following is a description of the general format of the CDP as it has typically been conducted in recent years. THE CONSENSUS DEVELOPMENT PROGRAM The CDP provides a forum for concerned individuals to evaluate medical technologies. Observers have noted that the three-day con- sensus development conference borrows processes from the scien- tific meeting, the judicial process, and the town meeting (Jacoby and Rose, 1986; Mulian and Jacoby, 1985~. In a consensus development conference, the panelists represent a jury who give the verdict after hearing the expert speakers give scientific testimony concerning the technological case at hand. The presentations by speakers are pub- lic, and members of the audience may comment and participate, similar to the proceedings of a town meeting. The entire planning and implementation process for a particular conference usually occurs over a period of 12 to 15 months, al- though the planning process has occurred over a period of as short as 5 months (Elliott, 1989). Each conference normally lasts over a period of three days. The original intention of the NTH CDP was to evaluate emerging technologies with potentially significant health impacts. Neverthe- less, even in the early years, panels have addressed technologies already in widespread use that had not been previously scrutinized for safety and effectiveness (Perry and Kalberer, 1980~. At present,
APPENDIX A 43 consensus development conferences investigate technologies that are new, established, or in widespread use, including some that may be obsolescent. Meetings have been held to reassess topics; for ex- ample, adjuv ant chemotherapy for breast cancer, originally addressed in 1980, was reexamined in 1985 (Goodman, 1988~. The topics of consensus development conferences have varied substantially since the inception of the program. Conferences tend to focus on a technology, e.g., electroconvuIsive therapy (1985) or magnetic resonance imaging (1987), or on a particular health prob- lem and the alternative technologies applied for diagnosis, treat- ment, or rehabilitation of these, e.g., travelers' diarrhea (198S) or adult urinary incontinence (1988~. In general, the bureaus, institutes, and divisions (BIDs) of NIH suggest consensus topics to OMAR for consideration. Topics are considered from other sources, including other Public Health Serv- ice agencies such as the Food and Drug Administration, the U.S. Congress, or organizations outside of government (Goodman, 1988~. OMAR uses the following criteria for the selection of conference topics: · The subject under consideration should have public health im- portance. The topic should affect or have broad application to a significant number of people. · There should be controversy surrounding biomedical/scientific aspects of the topic that would be clarified by the consensus ap- proach or a gap between current knowledge and practice that a con- sensus development conference might help to narrow. · The topic must have an adequately defined and available base of scientific information to answer the previously posed questions and to resolve the controversies insofar as possible. · The topic should be amenable to clarification on technical grounds, and the outcome should not depend mainly on the impres- sions or value judgments of panelists. Additional elements desirable for positive consideration of a consen- sus topic include health care cost impact, preventive impact, and public interest (OMAR, NTH, 19881. The topic selection process may take from two months to a year or more. A representative from the NIH Coordinating Committee on Assessment and Transfer of Technology (CCA=) usually par- ticipates in the initial evaluation of topics for consensus confer-
44 APPENDIX A ences. The CCA~ member is usually a BID senior staff person appointed by the BID director to "serve as an institutional spokes- person for the BID on issues that relate to the Committee. In addi- tion to being a member of the Committee, each BID representative shall serve as the primary focal point in his or her BID for commu- nication with OMAR and with other NIH units on all matters related to technology assessment and transfer" (NIH, 1983~. In the event that a topic under consideration is not suitable for a consensus de- velopment conference, OMAR and the BID may choose to conduct a workshop or host a scientific meeting to discuss the topic (Elliott, 1989). Once a topic has been suggested for a consensus development conference, the sponsoring BID nominates a BID coordinator who is knowledgeable in the area of science under consideration. The re- sponsibilities of the BID coordinator include chairing the planning meeting and representing the BID in managing the conference. A senior OMAR staff person is selected as the OMAR coordinator to work with the BID coordinator and other BID representatives in organizing He conference (OMAR, NIH, 1988~. OMAR's focus is on the consensus process, while the initiating BID's contribution concerns the scientific information required for the conference topic (Elliott, 1989~. The OMAR and BID coordinators, together with other appropriate BID and OMAR staff, meet to determine the topic's acceptability using the cr~tena listed above for a consensus develop- ment conference. After OMAR and a BID have agreed to act as lead sponsors of a particular consensus development conference, other BIDs and other agencies may agree to cosponsor the event (OMAR, NIH, 1988). The OMAR coordinator, the BID coordinator, and representatives from other sponsoring organizations meet to discuss He potential scope and date of the conference. The conference chairperson is selected at this time (OMAR, NTH, 1988~. The sponsoring BID and OMAR jointly select the chairperson (Elliott, 1989~. The chairper- son of the consensus development conference is chosen at an early stage in conference planning so that he or she can participate as a member of the planning committee and thus bridge the gap between conference planning and implementation (Elliott, 19891. The plan- ning committee is composed of NTH staff' including OMAR and BID staff, and outside experts from the research community who are not federal employees (OMAR, NTH, 1988~.
APPENDIX A 45 The planning committee has four major functions: "~1) to draft consensus questions, (2) to draft the conference program, (3) to rec- ommend conference speakers, and (4) to recommend consensus pane! members" (OMAR, NTH, 1988~. The consensus questions asked vary in number and type; in most instances, panels have addressed from four to six questions. The 1988 OMAR guidelines describe the drafting of questions to be addressed as follows: A. The agenda of a [consensus development conference] is structured around key questions posed to me pane} that serve to determine the scope and substance of the conference. B. Questions should be structured so that answers can be derived from scientific information and data presented by the speakers. The questions should not be phrased in a manner that requires responses dependent solely upon the subjective judg- ments or opinions of the panelists. C. Questions should be straightforward, concise, and con- structed so that it will be evident whether consensus has been achieved. D. Ordinarily, four to six questions are posed, including questions on efficacy, risks, clinical applications, and a final one on directions for future research (OMAR, NIH, lags). The planning committee plans the agenda. OMAR hires a support contractor to organize all conference logistics. The planning committee is also responsible for nominating the panel. OMAR conducts a literature search for all suggested panel- ists, including the chairperson, to ensure that individuals have no published positions concerning the conference topic (Elliott, 1989~. OMAR currently assembles panels with balanced representation comprising individuals with venous areas of expertise in order that the panel may better deal with the diverse material presented and that the panel's credibility may be enhanced (OMAR, NTH, 1988~. In selecting panelists, OMAR seeks individuals who are "thoughtful, able to weigh evidence, and capable of collaborative work" and who "have no vested interest in the technology being reviewed" (OMAR, NIH, 19X8~. According to OMAR, panels should include individuals involved in research in He field; heath professionals who are users of the technology; methodologists or evaluators such as epidemiolo- gists or biostatisticians; and public representatives such as ethicists, lawyers, theologians, economists, public interest group or voluntary
46 APPENDIX A health association representatives, consumers, and patients. Panel- ists should be residents of the United States and should not be fed- eral employees, to avoid the appearance of undue federal influence (OMAR, NIH, 19X8~. Speakers are experts on the chosen topic and present evidence, as appropriate, to the pane} on the safety, efficacy, effectiveness, and service requirements of the technology in question (Goodman, 1988~. OMAR guidelines describe the speaker's role as follows: A. Speakers should be selected for their scientific expertise and may include both clinical investigators and basic scientists as well as general authorities in the field. Where differences of scientific opinion exist, care should be exercised to include the presentation of opposing data and interpretations. B. Speakers should be asked to confine their presentations to the scientific topic that they have agreed to address and to be certain to present all relevant data and information. C. To prevent the appearance of bias, the planning commit- tee is encouraged not to include any of its members as speakers unless other experts are unavailable (OMAR, NTH, 1988~. At least one month in advance of the actual conference date, pan- elists receive abstracts of the speakers' presentations to prepare them for the consensus development conference. Speakers are also asked to bring photocopies of their slides to the consensus development conference for panelists. The BID coordinator is responsible for the supply of overview articles and other supplemental materials for the panelists prior to the consensus development conference. The BID coordinator meets with National Library of Medicine (NtM) staff in order to direct and detail the strategy for preconference information retrieval. Panelists receive a copy of this literature search approxi- mately three months prior to He actual conference date. The BID coordinator determines the criteria for the search, for example, how far back in years HEM staff should search and with which key words (Elliott, 1989~. Speakers and members of the audience receive con- ference materials, which include only speakers' presentation sum- maries, a conference agenda, and logistical information, at the first public meeting of the panel. The panel chair divides the pane} into subcommittees and desig- nates subcommittee chairs for each question several months before the conference. Each group is asked to prepare a proposed format to
APPENDIX A 47 answer one question, along with a draft outline of elements to be considered in answering the question. OMAR requests that this material not be in narrative form. Panelists use the infonnation they receive from OMAR to complete this task and submit the documents to the pane} chair at the first pane} meeting, which is held the day before the opening of the conference. This question assignment process allows panelists to specialize and focus on the relevant is- sues presented by particular speakers. During the preconference meeting, the panelists discuss the con- sensus development process and outline the consensus statement. Pane} members are responsible for writing the specific portion of the consensus statement that corresponds to the questions they were assigned prior to the conference. All panelists remain responsible for the statement as a whole, however, and thus should follow all presentations and deliberations in the conference. During the next two days panelists hear evidence from experts on the consensus topic and prepare several drafts of the consensus state- ment. On the morning of the first day, the director of OMAR deliv- ers the charge to the panel. The chairperson of the meeting opens the session and is responsible for conducting the presentations in an orderly and timely manner. Each speaker presents his or her expert knowledge on the consensus conference topic in a predetermined amount of time, usually between 15 and 30 minutes. In general, for every hour of presentations there is a discussion period of approxi- mately 30 minutes. The panelists are given the opportunity to ask questions for further information or for clarifications. Members of the conference audience (which have ranged from 200 to 1,100 indi- viduals, depending on the topic) and the attending speakers may then question the presenters and offer additional information. The chairperson seeks to ensure that the group follows the time schedule and does not digress from the consensus topic or questions. The public testimony from experts continues for one and a half days. After the first day's presentations, the panelists adjourn to draft the consensus statement. After the second day of presentations (which is a half day), the pane} again convenes in executive session to continue drafting the statement. At that time, the OMAR and BID coordinators are present to act as resources for the panel (e.g., to check statistics) and to ensure that the panel adheres to OMAR guidelines regarding the general content of the consensus statement (e.g., to ensure that direct recommendations for specific individuals
48 APPENDIX A or agencies are not included in the statement) (Elliott, 1989~. A representative from the Food and Drug Administration is usually available to answer the panel's questions on the regulatory status of the technology. The pane} remains in closed session the evening of the second day of the conference until consensus is reached and the statement is drafted. These sessions have continued into the early- morning hours of the third day. On the morning of the third day, the chairperson reads the draft consensus statement to the speakers and interested members of the public. Members of the audience may make comments or sugges- tions for changes in the statement. The panel adjourns again after the morning session to discuss the proposed alterations and draws up a final draft of the consensus statement. Following this, the pane] meets with the press in the afternoon to relate the group's conclu- sions. Panelists have approximately two weeks to contact OMAR to offer additional changes. For the most part, these later alterations have tended to be stylistic rather than substantive. Consensus development conferences sponsored by the National Institute of Child Health and Human Development (NICHD) vary from the format described above in several ways. Topics are re- viewed and analyzed by the NICHD Office of Planning and Evalu- ation according to OMAR criteria for consensus conference topic selection and the topic's relevance to the NICHD mission and re- search priorities. The planning committee for an NICHD/OMAR- sponsored conference chooses the chairperson, specifies the types of individuals for the panel, and identifies conference questions. The planning committee suggests the individuals who should present evidence as speakers. The final decision on conference speakers involves the planning committee, the panel, and NICHD and OMAR staff. To prepare for the first pane} meeting, panelists receive mate- rials from an extensive literature search done by HEM. The first meeting is held 12 to 18 months prior to the consensus development conference so that the pane} has time to develop a draft report on the chosen topic, including a list of important issues that need to be discussed during the meeting. The draft report documents the Knells analysis of the literature and includes complete references and an appendix of all materials presented at the conference. The draft report is circulated widely among interested organizations and indi- viduals, who are encouraged to submit written comments and/or give oral testimony at the consensus development conference. The public
APPENDIX A 49 consensus development conference consists of presentation by the panel of a summary of the draft report, invited comments, presenta- tion of additional data not previously considered, testimony from individuals and organizations, and a closed executive session for preparation of the consensus statement. After the conference the draft report is modified to coincide with the consensus statement and issued as a final report. The products of a NICHD/OMAR consensus development conference are a monograph and a consen- sus statement. The approximate cost per NICHD/OMAR consensus development conference is $275,000, whereas for OMAR consensus development conferences this figure is $ ~ 16,000 (Elliott, 1989~. REVIEW OF THE MAJOR COMPONENTS OF THE CONSENSUS DEVELOPMENT PROGRAM The structure of OMAR's CDP comprises the following steps: · topic selection · formation of the planning committee and selection of the chair- person · selection of specific conference questions, the consensus panel, and conference speakers · presentation and synthesis of conference data · development of consensus · preparation of initial and final consensus statements · dissemination of consensus information Many participants and observers have examined, commented upon, and made suggestions regarding improvement of the program. The substance of such reviews is summarized here, and is arranged by the components identified above. Topic Selection The effectiveness of the NIH CDP may be linked to the selection of appropriate topics at the best time for assessment. A study of physicians' knowledge, attitudes, and practices as a result of eight early conferences (1979-19X0) indicated that the choice of topics affects OMAR's ability to accomplish its information transfer goal. Kanouse et al. (1989) contended that the 1980 conference on cesar- ean sections was successful in reaching physicians, because it ad-
so APPENDIX A dressed "an aspect of care perceived to be problematic" and "per- formed a unique function in drawing together the accumulated knowI- edge on this topic and pointing out its practice implications." Other consensus development conferences were found to be less successful in reaching physicians. The researchers recommended that NTH "select topics only after a systematic review of the conditions under which the procedure to be covered by the conference is performed, including a determination as to whether physicians' and hospitals' practice presently conform with the recommendations that a pane] is likely to make" (Kanouse et al., 1989~. Political interests or pressures may have a substantial impact on the placement of a subject on the CDP agenda. Individuals involved in the Reye's syndrome conference reported that congressional con- cern in this area accelerated the adoption of the subject for a consen- sus development conference (Wortman et al., 1988~. The 1983 con- ference on liver transplantation has been cited as an instance of a politically, rather than scientifically, prompted conference (Markle and Chubin, 1987~. According to Markle and Chubin, the emotional aspect of a costly liver transplantation, especially when conducted for children, prompted extensive media coverage and congressional pressure to hold a consensus development conference. Markle and Chubin held that the staff of the National Institute of Arthritis, Dia- betes, and Digestive and Kidney Diseases and OMAR contended that the procedure was still experimental, given the absence of clini- cal data to substantiate stable positive outcomes after treatment. Liver transplantation was an experimental technology (i.e., the technology had not yet been adequately evaluated for safety and effectiveness in scientific studies) and not an emerging technology (i.e., the proce- cure had already passed through clinical trials, but was not yet widely adopted by clinicians). Markle and Chubin (1987) reported the fol- lowing: early 1980 Charles Lowe, then acting associate director of OMAR, re- quested that the National Institute of Arthritis, Metabolism, and Digestive Diseases (NIAMMD) supply him with an objective summary of the current state of the art of liver transplantation .... On April 7, 1980, NIAMMD staff responded to Lowe's request, providing a state-of-the-art assessment and con- cluding that ``despite recent improvements . . . hepatic transplantation is still largely an experimental procedure with relatively unpredictable outcome in individual cases." Nonetheless, the conference was held in June of 1983. At issue is the conflict between not selecting topics for which insufficient data
APPENDIX A 51 are available for reaching scientifically valid conclusions and the pressure to hold conferences on controversial issues (Institute of Medicine, 19X5~. Formation of the Planning Committee and Selection of the Chairperson Individuals, including representatives from the sponsoring agen- cies, the conference chairperson, and two or thme experts from out- side NIH, are invited by NTH staff to join the planning committee. The program planning committee is responsible for the definition of conference questions, formulation of a conference agenda, and nomi- nation of the speakers and panelists. The consensus development conference chairperson is appointed prior to the first meeting of the planning committee. According to OMAR, the chairperson of the consensus development conference is selected for his or her stature as a distinguished physician and scientist, nonadvocacy regarding the conference issues, and personal skills in chairing the open sym- posium portion of the conference and in leading the consensus pane} (OMAR, NTH, 19BX). Selection of the Specific Conference Questions, The Consensus Panel, and Conference Speakers For the most part, the conference questions posed by the planning committee have been found to be straightforward and concise. A study by Wortman and Vinokur (19X2) concluded that the questions tend to reflect the concerns of NTH staff. In their study, Wortman and Vinokur (1982) classified 12 sets of questions from conferences held from 1980 to 19X2 by content and found that the majority addressed the benefits and appropriate conditions for use of the tech- nology. Social issues were infrequently addressed; only 5 of 84 questions concerned economic, legal, and ethical inquires. This observation may be consistent with the stated guidelines of the NIH consensus development program, which indicate that The purpose of a [consensus development conference] is to evaluate the avail- able scientific information on a biomedical technology and to produce a Con- sensus Statement that advances understanding of the technology or issue in question (assessment) and that will be useful to health professionals and the public at large (transfer) .... Although many aspects of a technology under evaluation may be discussed, the primary focus of a [consensus development conference] is the technology's clinical applications (OMAR, NIH, 1988~.
52 APPENDIX A Controversial questions may be excluded from the conference agenda. Wo'iman et al. (1988) noted that the association between Reye's syndrome and aspirin was not a question posed for the con- sensus development conference for Reye's syndrome: A decision to eliminate a question that did not have a scientific answer at that time was acceptable to the institute planners involved, but could not be kept entirely out of the conference and on the basis of audience reactions and the concern of some panelists, a brief section had to be added to die consensus statement regarding the issue. Within a year, another U.S. Public Health Senice agency, the Centers for Disease Control, made a fairly definitive statement on the issue. Markle and Chubin (19X7) pointed out that the 1983 liver trans- plantation conference explicitly addressed only technical aspects of the technology, although important social, ethical, and economic issues shaped the conference deliberations: If liver transplantation was deemed an experimental procedure by the consen- sus panel, then third-party payment was unlikely. However, if it was classi- fied as a therapy with all the trappings of success, i.e.,, impressive survival rates and enhanced quality of life, then a government subsidy would be war- ranted. This emerged as a pivotal issue, yet was barely addressed at the conference. The result was that these issues became implicitly important, shaping the consensus statement in ways unrecognized, or at least not admit- ted by the participants themselves. The consensus statement on liver transplantation does not address reimbursement for this very expensive procedure; the existing and potential demand for the procedure; or how such demand could be met in terms of available transplant teams, facilities, and donor or- gans (Institute of Medicine, 1985~. A broader, more coordinative role to encompass wider social, legal, ethical, and economic assessment concerns in the U.S. Department of Health and Human Services was envisioned for another agency. Perry and Kalberer (1980), in a review of the first two years of the NTH CDP, noted that assessments of technologies occurred at two levels in He U.S. Department of Heady and Human Services: The NIH, representing interest and expertise in die biomedical sciences, has emphasized technical consensus development the assessment of scientific and medical aspects of the technology in question. Other agencies in this department also engage in evaluation activities on a smaller scale. The re- cently established National Center for Health Care Technology [NCHCT] is responsible for coordinating all the assessment activities in the department and has a mandate to concern itself with economic, legal, and ethical implica- tions as well as scientific and medical issues.
APPENDIX A 53 According to the former director of NCHCI, several features dis- tinguished the NCHCT from the NIH's CDP: 1. the NCHCT's role included the establishment of priorities for assessment among technologies; 2. the NCHCT was able to fund new areas of research identified and fund technology assessment activities; 3. assessments sponsored by the NCHCI were broad in scope, addressing economic and ethical concerns and the direct impact of the assessments on coverage by Medicare (Perry, 1982~. In 1981, budget cuts arrested NCHCl's activities. Markle and Chubin (1987) inferred that the NIH CDP "assumes that a strict separation of factual and value issues is possible, and further, that objective evidence compels experts to converge on the 'correct' decision." In contrast to concerns regarding the circumscribed scope of the program, some observers and participants have voiced frustration with the broad agenda of certain consensus development confer- ences. Four to six questions are usually addressed, and one of these normally pertains to suggestions regarding further research (Jacoby, 1985~. A panelist at the conference on the health implications of obesity charged that the group attempted to answer too many ques- tions (Henig, 19851. Henig (1985) indicates that the effectiveness of the consensus development conference may be limited if the plan- ning committee selects a wide scope of questions. Researchers at the RAND Corporation recommended that "consensus studies should focus on carefully defined problems that can be investigated in a timely and economical way" (Fink et al., 19X4~. The need to clearly define conference questions and focus the conference effort, on one hand, and the desire by some health professionals and members of the public to cover social, ethical, and legal implications of the application of a technology, on the other hand, raise the importance of appropriate attention being given to matters of scope and concern at all conferences. In addition to selection of conference questions, the planning committee nominates individuals to the panel. Over the course of the program, OMAR has assembled two general types of panels: the "balanced panel" and the "neutral panel" (Jacoby, 1985~. The bal- anced pane} is made up of representatives of different points of view on the technology in question. In this case, adversarial opinions are intended to carry He same weight, with equal numbers of advocates on each side. The neutral pane! consists of experts with no pub-
54 APPENDIX A fished opinion on the topic. In this case, individuals are presumed to be less committed to a particular position and thus, perhaps, more willing and able to consider the evidence with less bias. In recent years, the neutral panel has been used as discussions by balanced panels have been reported to tend to "degenerate into unconstructive debate" (lacoby, 1985~. Asch and Lowe (1984) recommended that the panel consist of representatives of their respective professions who are experts and thus would validate pane! recommendations. The size of panels depends upon the complexity of the problem presented, the number of diverse points of view and specialties required, and the resources available for the actual consensus development conference (Fink et al., 1984~. In recent years, panels have been neutral and therefore do not reflect diverse points of view, and the size of the pane! has not been limited by available resources (Elliott, 1989~. According to Wortman et al. (1988), panelists should be able to weigh evidence and to write clearly and concisely. The pane! begins to write the consensus statement during the one and a half days of presentations by the speakers. Given this limited amount of time for writing consensus statements, writing skills are vital to the effec- tiveness of the panel (Worunan et al., 1988~. Although the credibil- ity of consensus statements may depend upon the formation of ap- parently unbiased panels, critics of the consensus statement have protested that a pane] is "stacked in favor of the clinical status quo or NIH-supported recommendations" (Henig, 1985~. Conference speakers are selected on the basis of their expertise concerning the topic in question and may advocate particular posi- tions. Consumer representatives, lawyers, economists, and ethicists are not always represented at consensus development conferences. In general, the planning committee considers the relevance of these types of experts for a particular conference, given the limited speak- ing time available. An economist gave a presentation at the consen- sus conference on urinary incontinence (1988) in order to demon- strate the financial impact of the disability on the health care sys- tem. A report on the consensus development program noted that con- ference participants are drawn from a fairly narrow segment of the biomedical community and that panelists and speakers are often acquaintances of NIH staff members (Wortman et al., 1988~. Indi- viduals rarely participate in more than one consensus development
APPENDIX A 55 conference (Elliott, 1989). The panelists and speakers may all be drawn from among individuals who are closely affiliated with NTH; thus, they may tend to be involved in biomedical science rather than in policy or social science areas. NTH staff may have less contact and knowledge of the appropriate individuals outside of the medical research and practice communities. A former director of OMAR recommended a broader and more systematic search for conference participants (Perry, 1987~. A former OMAR officer suggested that "a 'separation of powers' be maintained among the triad of forces [planning committee mem- bers, speakers, and panelists] responsible for the final consensus product." lacoby (1985) held that this division of responsibilities among members of the panel, speakers, and the planning committee, throughout the consensus development conference program, would help ensure an "impartiality of thought and equality of influence." At present, members of the planning committee may participate in the consensus development conference as expert speakers but not as panelists, although this is discouraged by OMAR (Elliott, 1989~. Presentation and Synthesis of Conference Data Asch and Lowe (1984) noted that the validity of the CDP "capi- talizes on the ability and experience of NTH in mobilizing expertise, and in providing the data necessary for the formulation of authorita- tive statements." The panelists receive information on the consen- sus topic prior to and during the meeting. Kanouse et al. (19X9) hold that "the panel should be well-in- formed about both the current state of science and the current state of practice." Panelists receive a bibliography on the consensus topic, including the paper abstracts from the bibliographic citations, at least one month in advance of the scheduled conference. The BID coordi- nator for each conference determines the scope and the nature of the literature search to be conducted through NEM. Background reports may also be prepared, and individual experts may be commissioned to compile summaries of the state of the science. According to Perry (1987), "there have been charges that data provided to some panels have been incomplete, that they have been overinterpreted or misrepresented, [and] that recommendations have been made in the absence of supporting evidence." At present, OMAR instructs pan- els that they may qualify the recommendations they make in the
56 APPENDIX A consensus statement according to the strength of the evidence pre- sented (Elliott, 1989~. Levitt et al. (1988) evaluated five papers used by panelists in the 1985 consensus development conference on adjuv ant chemotherapy for breast cancer. The consensus statement favored the use of such treatment for breast cancer. The researchers found problems with the statistical analysis or in the performance of the randomized clini- cal trials in Me five papers. These problems made them question the positive findings on the use of adjuv ant chemotherapy. The re- searchers did not evaluate all of the articles used to formulate a consensus. Levitt et al. (1988) concluded: we feel that it is essential to ensure that a consensus of the use of these agents be based on careful evaluation not only of the conclusions, but also of the possible deficiencies of all the clinical trials anned at establishing the values of a certain treatment. Levitt et al. (1988) recommend that information retrieval for pan- elists should extend beyond a normal literature search, as the results of inconclusive or negative randomized clinical trials may not have been published. Panelists may extrapolate from the evidence presented at the con- ference when they write the consensus statement. The 1984 consen- sus development conference on lowering blood cholesterol to pre- vent heart disease produced a contested consensus statement. Ahrens (19XS) questioned the conference panel's recommendation of"a shift from the current typical American diet to one that is lower in total fat, saturated fat, and cholesterol" (NTH, 19XS) for men, women, and children aged two and older. Ahrens questioned whether the avail- able evidence warranted this recommendation and the safety and effectiveness of such a shift in diet for all Americans from age two. Ahrens (1985) points out that "the pane} leaned heavily on epi- demiological evidence and public health considerations" as opposed to evidence from clinical trials. The association between high blood cholesterol levels and coronary heart disease is seen in cross-popula- tion comparison studies and in studies of migrants, although clinical trials of drugs and diets to lower blood cholesterol may not have established a significant difference in patient outcome (Kolata, 1985~. Both Ahrens (1985) and Kolata (1985) criticized the panel's use of the Coronary Primary Prevention Trial (cPPr). The cPPr was a randomized clinical trial comparing two groups of high-risk males, in which one group received cholestyramine and the other a placebo.
APPENDIX A 57 Ahrens and Kolata question the results of the trial. Comparisons of the experimental and control groups showed "a 19% reduction in risk (P < .05) of the primary endpoint-definite coronary heart dis- ease (CHD) death and a 19% reduction in nonfatal myocardial in- farction- reflecting a 24% reduction in definite CHD death and a 19% reduction in nonfatal myocardial infarction .... The risk of death from all causes was only slightly and not significantly reduced in the [expenmentall cholestyramine group" (Lipid Research Clinics Program, 19X4~. The Lipids Research Clinic (LRC)-CPPT investi- gators note Rat "excess mortality in the ERC-CPPr cholestyramine group was confined to violent and accidental deaths. Since no plau- sible connection could be established between cholestyramine treat- ment and violent or accidental death, it is difficult to conclude that this could be anything but a chance occurrence" (Lipid Research Clinics Program, 1984~. Ahrens (1985) objects to the panel's "unjustifiable extrapolations" from clinical teals. Kolata (1985) and Ahrens (1985) questioned the panel's use of data on reduction of blood cholesterol in high-nsk males, in whom little or no great health improvement was obtained, to support the consensus statement's broad recommendation for all Americans over age two to adhere to a diet "generally consistent with the most recent recommendations of the American Heart Asso- ciation and the Atherosclerosis Study Group of the Inter-Society Commission on Heart Disease Resources" (NIH, 19851. In addition, Ahrens noted the absence of details on alternative diets for those who do attempt to lower blood cholesterol levels. Oliver (1985) suggested Hat the CDP was highly imperfect for evaluation of the effect of cholesterol reduction on coronary heart disease. He noted specifically the shortage of time for the consen- sus process, the bias against cholesterol on the part of the panel, and the forced process of consensus in a highly controversial subject. Oliver (1985) suggested alternative scenarios for technology as- sessment in order to better present and synthesize the available in- formation: · a learned body or government agency could establish a small group of experts to conduct an extended review of the literature and draft a report over the course of two years; · experts could meet for a week of full discussion and a more complete literature review;
58 APPENDIX A · a learned body or goverrunent agency could form a small stand- ing committee to address topical issues in health care technology (Oliver, l9XS). These scenarios suggest that expansion of the time allocated for reaching consensus and for the preparation of the consensus state- ment would improve the outcome of consensus development activi- ties (Oliver, 1985~. Development of Consensus Some participants, including chairpersons, have been frustrated by the limited amount of time available for reaching consensus. Jacoby (1985) contended that "there is always the dilemma of not wanting to impose constraints (of time or rigidity) while at the same time trying to assure a useful outcome (i.e., closure) within an allot- ted time." Fink et al. (1984) recommended that the level or type of consen- sus on any subject of debate should be defined in advance to ensure an efficient and agreeable settlement of disputes; for example, · On the final vote, any topic supported by at least X~O (a prede- termined percentage) of participants is adopted. · All topics are rated on a scale of ~ to 5, only chose topics receiving a mean rating of 3.0 or greater are accepted. · Any topic is dropped if it is vigorously opposed by at least X% (a predetermined percentage) of the participants (Fink et al., 1984~. Several consensus development conferences have experimented with decision models to help the panel explore the implications of the data presented by speakers (lacoby and Pauker, 19861. McNeil and Pauker (1984) describe decision analysis as an exercise in building models- models that can be examined, modified, inter- rogated, and rebuiltto provide insights into a choice in which outcomes are uncertain and risks are unavoidable. The underlying purpose of such an exercise would, on the surface, seem to be to arrive at the "correct" answer the optimal strategy. But that is only a small part of the picture. The greater purpose is to provide a decision maker with an explicit reproducible process that can help structure complex alternatives in a rational way, a process that allows the incorporation of expertise and information from a variety of expert consultants without abdicating the decision to any one of them.
APPENDIX A 59 Jacoby and Pauker (1986) hold that the decision analytic process should involve five steps, which they describe as follows: I. ad of the available data are used to develop carefully struc- tured, specific questions and to define the implications of the tech- nology in terms of a logical sequence of potential strategies and outcomes; 2. the data are used to estimate the probability of each outcome; 3. the data are used to estimate the utility of each outcome; 4. consensus panelists use the mode} as an aid to determine the most desirable strategies; and 5. strategies are tested under different assumptions (sensitivity analysis). Six consensus development conferences have made some use of this approach: diagnostic ultrasound in pregnancy (19841; postmeno- pausal estrogens in the prevention of osteoporosis (1984~; limb-spar- ing treatment of adult soft-tissue and osteogenic sarcoma (19X4~; registries for bone marrow transplantation (19XS); adjuv ant chemo- therapy for breast cancer (1985~; and the impact of routine HTLV- III (human T-lymphotropic virus type IlI) antibody testing on public health (1986~. Jacoby and Pauker (1986) presented Table A-1 to review the potential uses of decision analysis in consensus develop- ment. The panelists' use of the decision models has varied. Panel- ists followed the structured proposed model question areas for con- sideration to different degrees; some groups have relied upon the decision analysis in response to questions from the audience on the consensus statement (Iacoby and Pauker, 1986~; other groups have made little use of the decision analysis. One NTH BID coordinator noted that the decision analysis did not add to the process of consen- sus development (Lipman, 1989~. Jacoby and Pauker (1986) con- tend that the process should be refined and that planners should present information about the advantages and limitations of decision analysis in a more straightforward manner to better educate panel- ists. The consensus process may not facilitate the expression of alter- nate viewpoints. Henig (1985) related the criticism that the "consen- sus statement which almost by definition, leans toward the moder- ate, the mainstream, Me prudentcan't embrace minority views." Several consensus conferences have produced the statements with
60 TABLE A.l Decision Analysis in Consensus Development APPENDIX A Preconference During Conference Postconference Planning Selecting topics Formulating questions Identifying needed input Preparation Tuning panel in Focusing speakers Providing consistent data Evening before Identifying central issues Broadening panel's views During presentations Publication Modify model Incorporate new data . ncorporate new assumptions During deliberations Focus discussion Perform sensitivity analysis Integrate and interpret data Make projections During writing Focus consensus statement Clarify specifics During public discussion Provide support against criticism Permit justified modifications Document reasoning Provide baseline for future assessments SOURCE: Israel Journal of Medical Sciences, 1986, v. 22, p. 186. dissenting views (see, for example, the 1980 consensus development conference on cervical cancer screening). The majority of NTH con- sensus development conferences emerge with one statement accepted by Me panels. Preparation of Initial and Final Consensus Statements Early in the program, critics were concerned that consensus state- ments would be "bland generalities that represent the lowest com- mon denominator of a debate" (Rennie, 1981~. Kahan et al. (1988) performed a content analysis of the statements of 24 consensus de- velopment conferences between 1979 and 19X3. The researchers
APPENDIX A 61 recommended a didactic-style consensus statement, that is, one that is practical in orientation and offers detailed guidance for clinicians. The researchers hypothesize that concrete, differentiating recommen- dations (ones that suggest specific actions on the part of health care practitioners for different subclasses of patients) would influence physicians. Panels are currently asked to write a shorter section of conclusions and recommendations to emphasize the key points of the consensus statement. There have been two attempts to involve a science writer in the composition of the consensus statement. In one instance, the sci- ence writer was brought in by the program planning committee; in the other, the chairperson selected the science writer to join the panel in executive session. Both times, the practice was reportedly unsuccessful as the panelists involved were unwilling to submit to outside editing and interference (lacoby, 198X). Dissemination of Consensus Information According to OMAR, the dissemination of the results may now in- clude the following: A. Consensus development conferences usually receive consider- able attention from the medical media and general media at the time of their occurrence. This serves to focus attention on the topic and the statement of the panel. B. The OMAR Director of Communications and the BID Informa- tion Officer develop an information dissemination plan covering publicity for the conference and the strategy for distributing the Consensus Statement. C. The Consensus Statement is printed by OMAR and distributed routinely to a variety of Federal health agencies, health care organi- zations' and the directors of continuing education of American Hos- pital Association membership hospitals. Additionally, the Consensus Statement is sent to targeted individuals and organizations specified in the information dissemination plan. D. The Journal of the American Medical Association routinely publishes most of the Consensus Statements. Consensus Statements are also published by specialty journals in the area of the topic. E. OMAR places notices in numerous professional journals an- nouncing the availability of the Consensus Statement and inviting · 1nqulrye
62 APPENDIX A F. The publication of the Consensus Statement along with se- lected papers from a consensus development conference as a sympo- sium is also a possibility. Proceedings of several conferences have been published in this manner either as supplements to specialty journals or as a monograph. G. Summary videotapes and audiotapes of the conference may also be prepared and distributed. H. A summary of the statement is also prepared and sent to appro- priate specialty journals (OMAR, NTH, 1988~. References Ahrens, E.H., Ir. 1985. The diet-heart question in 1985: Has it really been setHed? The Lancet i:1085-1087. Asch, S.M., and C.U. Lowe. 1984. The consensus development program: Theory, process, and critique. Knowledge: Creation, Diffusion, Utilization 5~3~:369-385. Elliott, I. 1989. Office of Medical Applications of Research, per- sonal communication. Fink, A., I. Kosecoff, M. Chassin, and R.H. Brook. 1984. Consen- sus methods: Characteristics and guidelines for use. The Ameri- can Journal of Public Health 74~9~:979-983. Goodman, C., ed. 198X. Medical Technology Assessment Direc- tory. Washington, D.C.: National Academy Press. Henig, R.M. 1985. The process and provocations of consensus. Medical World News (October):96-1~ I. Institute of Medicine. 1985. Assessing Medical Technologies. Washington, D.C.: National Academy Press. Jacoby, I. 1985. The consensus development program of the Na- tional Institutes of Health: Current practices and historical per- spectives. International loumal of Technology Assessment in Health Care 2:420-432. Jacoby, I. 1988. Uniformed Services University of Health Sci- ences, personal communication. Jacoby, I., and S.G. Pauker. 1986. Technology assessment in health care: Group process and decision theory. Israel Journal of Medi- cal Science 22:~83-190. Jacoby, I., and M. Rose. 1986. Transfer of information and its impact on medical practice: The U.S. experience. International Joumal of Technology Assessment in Health Care 2~:107-~15.
APPENDIX A 63 Kahan, I.P., D.E. Kanouse, and I.D. Winkler. 1988. Stylistic var~- ations in National Institutes of Health consensus statements, 1979- 83. International Journal of Technology Assessment in Health Care 4~2~:289-304. Kanouse, D.E., R.H. Brook, I.D. Winkler, I. Kosecoff, S.H. Berry, G.M. Carter, I.P. Kahan, L. McCloskey, W.H. Rogers, C.M. Winslow, G.M. Anderson, L. Brodsley, A. Fink, and L. Meredith. 1989. Changing Medical Practice Through Technology Assess- ment: An Evaluation of the National Institutes of Health Consen- sus Development Program. Santa Monica, Calif.: The RAND Corporation. Kolata, G. 1985. Heart panel's conclusions questioned. Science 227:40-41. Levitt, S.H., R.A. Polish, D. Aeppli, and B. L`indgren. 1988. The consensus statements on adjuvant chemotherapy in breast cancer: Fact or artifact. American Journal of Oncology ~ :73-76. Lipids Research Clinics Program. 1984. The Lipid Research Clin- ics coronary primary prevention trial results. Journal of the American Medical Association 25~:351-364. Lipman, M. 19X9. Georgetown University, Vincent Lombardi Can- cer Center, personal communication. Markle, G.E., and D.E. Chubin. 1987. Consensus development in biomedicine: The liver transplant controversy. The Milbank Quar- terly 65:-24. McNeil, B.~., and S.G. Pauker. 19X4. Decision analysis for public health: Principles and illustrations. Annual Review of Public Health 5: 135-1 61. Mulian, F., and I. Jacoby. 1985. The town meeting for technology. Journal of the American Medical Association 254~:1068-1072. National Institutes of Health. 1980. Cervial Cancer Screening: The Pap Smear. Consensus Development Conference Summary 3~41. Washington, D.C.: U.S. Government Printing Office. National Institutes of Health. 1983. Charter of the NTH Coordinat- ing Committee on Assessment and Transfer of Technology. Be- thesda, Md.: National Institutes of Health. Photocopy. National Institutes of Health. 1985. Lowering Blood Cholesterol to Prevent Heart Disease. Bethesda, Md. Office of Medical Applications of Research, National Institutes of Health. 1988. Guidelines for the Selection and Management of Consensus Development Conferences. Bethesda, Md. Pamphlet.
64 APPENDIX A Oliver, M.F. 1985. Consensus or consensus conferences on coro- nar,, heart disease. The Lancet i:1087-1089. Perry, S. 1982. The brief life of the National Center for Health Care Technology. New England Journal of Medicine 307~17~:1095- 1100. Perry, S. 1987. The NTH consensus development program: A dec- ade later. New England Journal of Medicine 317:4X5-488. Perry, S., and I.T. Kalberer, Ir. 1980. The NTH consensus-develop- ment program and the assessment of health-care technologies: The first two years. New England Journal of Medicine 303:169-172. Rennie, D. 1981. Consensus statements. New England Journal of Medicine 304:665-666. Wortm an, P.M., and A. Vinokur. 1982. Evaluation of the National Institutes of Health Consensus Development Process, Phase I: Final Report. Ann Arbor: The Institute for Social Research, The Uni- versity of Michigan. Wortm an, P.M., A. Vinokur, and L. Sechrest. 1988. Do consensus conferences work? A process evaluation of the National Institutes of Health consensus development program. Journal of Health Politics, Policy and Law 13~3~:469-498.
