Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Dissemination and Impact of Consensus Development Statements Arnold D. Kaluzny Consensus development conferences were initiated in the United States at the National Institutes of Health (NIH) in 1977 as a method for developing a statement of accepted standards for clinical prac- tice either with regard to the use of a particular technology or the treatment of a particular disease or syndrome. Other countries have organized conferences, although their formats and purposes have changed over time and between countries. The purpose of this paper is to consider the main elements of the dissemination activities in various countries, programs, to consider different methods of diffu- sion, and to assess their impacts in various settings. The analysis presented is based on program profiles prepared by each country for the International Workshop on Consensus Development for Medical Technology Assessment (1989~. A comparative review of these profiles provides a unique opportunity to analyze international ef- forts in the dissemination of consensus recommendations and to suggest possible areas for further research. ELEMENTS OF DISSEMINATION ACTIVITIES While the elements of the dissemination process for any particular program vary by country, it is possible to compare these vis-a-vis barriers to diffusion identified by existing theory and research. Al- though the list could be quite extensive, attention will be given to 69
70 CONSENSUS DEVELOPMENT four factors with varying degrees of tractability that may directly influence dissemination activities: attributes of the technology un- der review and the resulting consensus statement, adopter character- istics, environmental constraints and incentives, and communication channels. Attributes The technology selected for review and the technological attri- butes considered are important determinants of dissemination. Un- fortunately, the literature on the diffusion of medical technology is often characterized by inconsistent findings and thus is often dis- counted in our overall effort to better manage me review process and the dissemination of state-of-the-art technology. In part, this inconsistency is accounted for by what some researchers (Downs and Mohr, 1976) have termed a unitary approach to developing in- novation theory. This means that all innovations, regardless of type and/or specific attributes, are considered as equal and subject to the same theory. Increasingly, it is recognized that both the rate and speed of adoption and diffusion are a function of the interaction of the type and attributes of the innovation with various adopter char- acteristics (FenneH and Warnecke, 1988~. Moreover, there is some evidence to suggest that both diffusion and adoption of various types of programs or technologies are not totally random. There appears to be a predictable order that the adopting unit follows as it tend to implement and/or adopt a particular activity (Fennel!, 1984~. Cer- tain types of technologies may be linked in such a way that the implementation of one tends to facilitate implementation of another. Thus, the selection of topics for consensus and the range of criteria upon which judgments are made are extremely important in the de- sign of a dissemination strategy. A review of consensus development conferences indicates that they take a fairly inclusive view of the types of technology subject to the consensus development process. While all programs specify criteria for inclusion, the technologies reviewed tend to be fairly eclectic and not subject to any apparent a priori strategy that would facilitate dissemination. Among the programs considered here, the one exception to this generalization is the Canadian Task Force on the Periodic Health Examination. The Canadian Task Force limits the focus of their evaluation and dissemination efforts to preventive
DISSEMINATION AND IMPACT 71 services offered to asymptomatic individuals within a primary care setting. This implicit programmatic theme may facilitate dissemina- tion and the subsequent adoption of consensus recommendations by physicians. Different countries assess a range of attributes for particular tech- nologies in the consensus development programs. As documented in the innovation literature (Fineberg, 1985; Scott, in press), these attributes influence diffusion of technology as wed as the ability to have an impact on physician practice. For example, technologies that have a favorable cost-benef~t ratio, are compatible with ongoing practice patterns, and are consistent with reimbursement policies will be disseminated more readily to potential adopters. Table 1 presents the various attributes upon which technologies are assessed according to three major groupings. The U.S. program and the Ca- nadian Research Group focus explicitly and almost exclusively on the effectiveness of technology. ~ . .. . ~ ~ . ~ . The Canadian Task Force and The Netherlands emphasize effectiveness, with secondary consideration given to the psychosocial, economic, ethical, and legal implications. The programs in Finland, Norway, Denmark, the United Kingdom, and Sweden appear to give equal weight to effectiveness as well as other attributes, including cost and service requirements. Technology and its associated attributes are important factors in the diffusion process, yet an equally critical element concerns the attributes of the consensus statement itself. How clear is the state- ment? Is it prescriptive or discursive, and does it provide concrete and specific actions or guidelines that physicians can follow in clini- cal practice? While there has been no systematic study of how the attributes of the consensus statement influence physician decision making, one U.S. study of the NIH consensus development process attempted to assess the attributes of the consensus statement (Kanouse et al., 1987~. The researchers selected 24 statements for analysis and found three dimensions for statement classification: discursive, didactic, and scholarly. Discursive statements tend to be long and abstract and contain few recommendations. . . . . ~ . . . Consensus statements ;narac~er~zea as a~aacuc otter cacaos practical and detailed guid- ance, while scholarly statements offer up-to-date descriptions of the scientific evidence bearing on a topic and devote more attention to detail than most statements. The critical issue is obviously the rela- tionship of these and other attributes to an actual change in physi- cian knowledge and behavior.
.~ - ~ a, a:, ·= 3 ~ 3 __- ~ S ~ o ~ a _ _ _ ~ S 3 i, e 0 ~ ~ ~ ~ ,. of_ ~- ~ ~ ~ 0 = ~ ;~ so ~ P 11 ~ ¢ !5 {, ~^ ~ , ~ ~ ~ ~ , ~ in, 6 ¢ 72 ~ TV ._ _ U. ._ ~ =._ 3 U. 3 o a 5 - 3 4! a; a, - ~ _ U. rat ;- Cat ;~ ~ a o o U. ;^ ~ ~ o ·e ·= ·= o ~ ~ ·-- ~ no to ~ ~ v c u,.= _ 11 5 ~ 5
DISSEMINATION AND IMPACT Adopter Characteristics Behavior and Structure 73 Physician characteristics and the context of their practice are criti- cal to any dissemination effort. Table 2 indicates that several coun- tries have conducted evaluations of their programs and that the level of awareness of both the conferences and their content vary by coun- try and within country by physician characteristics. For example, evaluation results suggest that there are significant differences in the level of awareness between U.S. and Swedish physicians. In Swe- den, a national sample of physicians revealed a significant level of awareness among targeted groups. Awareness of a single consensus conference was very high, ranging from 86 to 94 percent in all target groups, with approximately 7 to 10 percent of the respondents indi- cating that the consensus statement evoked some changes in clinical practice (Iohnsson, 19881. In a national sample of U.S. physicians, 41 percent reported general awareness of the program but only IS percent said they were somewhat or very familiar with it. Awareness varied by specialty, with oncologists indicating the greatest familiar- ity and family practitioners the least (Kanouse et al., 19871. Awareness also appears to be a function of physicians' sociode- mographic characteristics. Analysis of the U.S. data (Kanouse et al., 1987) revealed that physicians who had heard of the program were somewhat older, had practiced medicine about two years longer, were less likely to work in private group practice and more likely to work in a hospital, clinic, or other institutional setting; or were more likely to be the members of a medical school's teaching staff and to report that they had responsibility for training students, residents, and interns. Moreover, their information habits and preferences were quite different. Physicians who were aware of the program reported spending more time reading journals such as the Journal of the American Medical Association (where many NIH consensus state- ments are published) and tended to talk informally with their col- leagues about medical topics. They also reported spending 30 per- cent more time attending continuing medical education courses and tended to receive patient referrals from a larger number of physi- cians than did those who were less aware of program recommenda- tions. What is the relationship of conference recommendations to actual clinical practice? Three countries report evaluation efforts to link conference recommendations with actual clinical practice patterns. Of these three studies' only the Norwegian results suggest that the
74 CONSENSUS DEVEl~OPMENT consensus conference recommendations affected practice patterns. However, as the evaluation suggests, and as discussed under the section dealing with environmental constraints, this may be more a function of environmental factors than of a consensus statement. The researchers used a preconference, postconference survey design to gather data from the country-wide registration of ultrasound pro- cedures. The analysis revealed a reduction in the average number of diagnostic ultrasound examinations per woman following the pro- mulgation of the consensus conference recommendations on the use of ultrasound in pregnancy. Data from two remaining studies indicate the contrary. A recent Canadian analysis of practice guidelines for use of cesarean sections on the attitudes and behavior of physicians reveals that there was substantial awareness and agreement on the proposed guidelines as well- as a self-reported change in practice (Lomas et al., 1989~. However, the actual analysis of hospital discharge data suggested that the consensus guidelines produced little or no change in actual practice. A similar analysis comparing self-reported practice with actual practice was conducted as part of a U.S. evaluation (Kanouse et al., 1987~. Here, changes were measured in several hospital- based procedures that were subject to consensus conference recom- mendations, providing a comparison between physician self-reported and actual practice vis-a-vis exposure to particular consensus recom- mendations (Kosecoff et al., 1987~. Results showed that the confer- ences largely failed to stimulate change in physicians' practices, despite moderate success in reaching appropriate target audiences. While this analysis involved physicians and hospitals in only one geographic region of the United States, and thus is subject to severe limits of generalization, the data did reveal that physicians' pre- ferred practice patterns bear a strong relationship to what they actu- ally do. The link, however, between consensus development confer- ences and actual practices was quite disappointing. For example, in the analysis of a breast cancer conference, relationships between physician awareness of the conference and their compliance with the recommendations reflected preexisting differences, not a program effect. Environmental Constraints and Incentives The context in which dissemination occurs can facilitate or inhibit the process. The literature on diffusion points to a series of organi-
DISSEMINATION AND IMPACT 75 zations at the federal, state, and local levels that affect this process (Fineberg, 1985~. For example, in Norway, the endorsement by the Health Directorate of the consensus conference statement on ultra- sound screening of pregnant women as the national guideline af- fected the use of ultrasound screening in clinical practice (Backe and Nafstad, 19891. Equally important, but often overlooked, may be subseries of"supply-side factors" (Robertson and Gatignon, 1987~. Sources of innovative technology other than consensus statements may influence physician behavior and are important in determining the amount of persuasive information being transmitted to potential adopters. For example, the aggressive marketing strategies of pA- vate corporations that produce medical technologies and the clinical policies of hospitals and clinics that provide care influence physi- cian behavior. The extent to which environmental constraints and incentives af- fect dissemination or are considered in formulating a dissemination plan varies according to the structure and function of the program. The very composition of a consensus development panel is often intended to address some of these constraints by including relevant administrators and policymakers in the consensus development pro- cess. For example, the Swedish program considers social, organiza- tional, and economic aspects of technologies. Thus, the conferences involve experts in medicine, health economics, epidemiology, and health policy, as well as administrators and concerned patient groups. The resulting statements go beyond addressing the safety and effi- cacy of the technology. The statements are directed toward a much broader audience, with dissemination targeted to concerned physi- cians, politicians, and administrators. Evaluation studies have focused both on the awareness of consen- sus development conferences by health administrators and politi- cians and their awareness of consensus results. Among one sample of Swedish health administrators and politicians, awareness of con- sensus conferences was high (Calitorp, 1988~. Eighty-nine percent indicated that they knew about the conference. Ninety-nine percent of the administrators were aware of the consensus conference, and 85 percent of the politicians knew about the consensus conference. When queried about their awareness of the outcomes (i.e., the con- tent of the consensus statements), results were equally impressive. Eighty-three percent of the respondents were aware of the outcome of one or more conferences. Administrators ranked highest (96 per- cent of the administrators were aware of one or more statements).
76 CONSENSUS DEVELOPMIINT Eighty-four percent of the politicians (full- or part-time) were aware of one or more of the statements. The latter ranged from a low of 49 percent who were aware of diagnostic imaging of liver tumors and 67 percent who indicated awareness of the recommendations on sight- improving surgery. Communication Channels The channels used to learn about new medical technologies have been a frequent subject of investigation (Eisenberg, 1986; Fineberg, 19XS). Consensus development conferences have used a variety of channels, giving primary importance to professional journals, both general and specialty, and direct mailings of consensus statements. Table 3 lists the various channels used by countries' programs as part of their dissemination efforts. Few programs use direct mail or continuing education. The majority of programs disseminate vari- ous forms of publications to health care providers, managers, and policymakers, whereas fewer programs target the general public. The evaluation of specific channels has been limited. Two exceptions include the RAND Corporation study (Kanouse et al., 1987) of the U.S. consensus development program and the Canadian Research Group proposal. The latter is a potential series of quasi experiments to evaluate alternative dissemination strategies (Lomas, 1989~. Spe- cifically, the Canadian Research Group is conducting a large ran- domized controlled trial to evaluate two dissemination strategies; one strategy uses local '`educational influentials" and the other strat- egy involves the use of chart audit and feedback. The U.S. evaluation suggests that consensus recommendations are more likely to reach specialists than generalist physicians (Kanouse et al., 19X7~. The study was not able to establish a direct link between awareness and publication in a generalist type channel, such as the Journal of the American Medical Association (lAMA), which publishes most of the NIH consensus statements. The analysis re- vealed that reading lAMA had no predictive value after other factors such as specialty and type of practice were taken into account. Reading the New England Journal of Medicine and various specialty journals, however, and being part of a well-defined network of clini- cians was associated with greater awareness of the consensus devel- opment program. Similar patterns existed with respect to actual knowledge of relevant conferences, with higher levels of awareness being recorded by physicians reading specialty journals.
~ - w v' To to . · 3 V, w o U. 5w - Cal o 4 - :: o V D Cal Cal V o V 5w - .5 W ~ 5w ~ U _ w ~ ~ 5: W ~ w5 U w V ~ ~ ~ X X AX ~ ~ ~ ~ ~ X X~ rat TIC ~ ~ ~ ~ O _ ~ . ~ oo .O ~ ~ ~ o.b o ED ~ ~ ~ C' ~ U 77 o w .~ C) .= 0 ._ P4 ~ 0 W ~ W 11 ~ := 11 ~ lo:
78 CONSENSUS DEVELOPMENT The U.S. evaluation also assessed continuing medical education (CME) programs and direct mailings as alternative communications channels (Kanouse et al., 1987~. Dissemination through CME pro- grams was considered an important source in that more than two- thirds of respondents in the physician survey regarded conferences, meetings, and CME programs as "very important" information sources both for first hearing about new medical procedures and for deciding whether to use them. Nearly half of the physicians surveyed had learned of the Consensus Development Program through direct mail- ing of different reports by the National Institutes of Health. A sub- study to examine direct mailings to five relevant specialties in the metropolitan St. Louis, Missouri, area revealed that significantly more physicians who received such mailings were aware of the confer- ence and its recommendations than was a comparative sample of physicians who did not receive such direct mailings (Iacoby and Clarke, 1986). AREAS OF FUTURE RESEARCH The development of consensus development conferences, many with different functions and structures operating within a variety of different health care systems, provides an opportunity for future col- laborative research and evaluation. Variations in function, structure, and setting provide a unique opportunity to set up a series of natural experiments that can increase the overall understanding of the basic diffusion process as well as the effectiveness of alternative dissemi- nation strategies. This understanding is critical for (~) enhancing the effectiveness of consensus development conferences vis-a-vis their stated goals and objectives and (2) providing a data base from which to evaluate the consensus conference approach to dissemina- tion compared with other methodologies and programmatic initia- fives that can influence physician practice patterns (e.g., standards and protocol participation). Listed below are several research areas that capitalize on a systematic and comparative assessment of con- sensus development conferences. · Evaluation of specific dissemination methods. The effective- ness of specific dissemination strategies is an empirical question and is contingent on several covariants. One approach would be to set up a series of experiments or quasi experiments to evaluate alterna-
DISSEMINATION AND IMPACT 79 live strategies beyond those traditionally used such as professional journals and newsletters. For example, the Canadian Research Group randomized trial evaluates two implementation strategies: use of local influentials versus use of chart audit and feedback Comas, 19X9~. Obviously, this requires replication in other settings along with the opportunity to evaluate other dissemination strategies such as the use of"clinical alerts" by relevant governmental agencies, computerized physician query systems, and the use of follow-up meetings for selected clinicians that are related to a particular con- sensus statement. · Role of organizational intermediaries. An underlying assump- tion of most dissemination efforts is that the individual physician is the unit of analysis and should be the target of the dissemination effort. In reality, however, health care organizations are involved in the delivery and financing of the technology and represent important constituent groups. Thus, they are critical actors in the adoption process. This is clearly recognized by several of the European pro- grams, in that their panels include administrators and policymakers, yet even within this context, the explicit target is the physician. The explicit targeting of organizations and their decision-making processes requires an understanding of organizational behavior, which is quite different from individual adoption processes. Consideration needs to be given to a range of factors including structural charac- teristics, the role of coalitions within organizations, and the idea of secondary choices (i.e., the implementation decision by the organi- zation and the subsequent adoption decision by the physician). · Interaction of attributes, adopters, and environmental charac- teristics. A central issue is the kind of diffusion and adoption pro- cess that may occur with different types of technology, different types of consensus statements, and different environmental condi- tions. The opportunity to explore each, as wed as their interactions, is present in a cross-cultural assessment of dissemination practice. There is a unique opportunity to employ a natural experiment using a series of tracer technologies for consensus development programs and dissemination efforts in a variety of countries. This approach would clarify the nature of the interaction between the attributes of the technology being diffused, the characteristics of the adopting unit and the political and policy context of various countries. For example, one would expect different diffusion patterns in competi- tive and regulatory environments. The ability to monitor these dif-
80 CONSENSUS DEVELOPMENT fusion patterns over time and to compare different delivery systems permits insight as to how larger cultural factors and values influence practice patterns (Waitzkin, 1983~. This type of evaluation would help us move away from a "unitary approach to diffusion" and to- ward building and testing theory that would have greater relevance to the actual design of dissemination strategies. · Dissemination and changes in actual clinical practice. While evaluations of dissemination strategies have focused primarily on awareness of Me consensus development program and/or specific program recommendations, the real issue is whether dissemination and the resulting awareness actually change physician practice pat- terns. Few evaluations have attempted to assess such changes; and evaluations that examine the relationships between dissemination efforts, levels of awareness, and attitudes toward specific confer- ences with physician behavior are needed. This information would provide a critical link to targeting areas of practice requiring con- sensus recommendations since it is precisely these areas of practice that determine physician readiness to comply with recommended changes. · Revising assumptions about the role of dissemination. A good share of our diffusion models, or at least the assumptions underlying them, are borrowed from a simpler time and a simpler problem. Clearly, the dissemination of information to change physician prac- tices is far more complex than efforts to change the buying habits of the general public. A close examination of clinical practice patterns and decision-making processes reveals a level of intractability that is not easily influenced by fairly simple dissemination practices. For example, a qualitative analysis comparing decision-making prac- rices of physicians in the United Kingdom and the United States in terms of adoption of formed versus dynamic (unformed) technolo- gies revealed that physician decision-making processes are greatly shaped by the perspectives of local clinical practices and are not easily influenced by more formal dissemination channels. Ann Greer suggests: "there are no magic signatories or formats which will cause knowledge to jump off the page and into practice" (Greer, 1988~. The complexity of the decision-making process clearly suggests that consideration needs to be given to other strategies beyond simple dissemination and that in the future the relative cost-effectiveness of these strategies must be given greater attention. This is not to sug-
DISSEMINATION AND IMPACT 81 gest that consensus development conferences and concomitant dis- semination strategies be abandoned, but they need to be supple- mented with private and other public sector initiatives consistent with the underlying process of clinical practice. One approach may be to couple the involvement of clinicians in ongoing research (e.g., clinical trials) with the systematic selection of topics for consensus development and subsequent diffusion to the practicing community. For example, within the United States, the National Cancer Institute has instituted a Community Clinical Oncology Program that allows local practitioners to participate in clinical trials research. Combin- ing this type of involvement with the diffusion of consensus state- ments may be worthy of consideration. Another possibility is to capitalize on larger ongoing initiatives to target dissemination ef- forts. For example, in the United States the Joint Commission on Accreditation of Healthcare Organizations has launched a program to monitor selected organizational and clinical outcome indicators as part of the overall hospital accreditation process. The availability of this information may help to target dissemination efforts to those clinical areas amenable to and/or requiring change. Finally, the chang- ing of practice patterns requires a new recognition of the role of patients and administrators in the clinical decision-making process. As reflected in many of the European consensus development con- ferences, consumers and/or their representatives, along with admin- istrators and various policymakers, are important contributors in the evaluation of given technologies. Consumer involvement and influ- ence in changing physician practice patterns is not well understood and is worthy of investigation and evaluation. SUMMARY The consensus development programs and the resulting consensus statements represent one approach to influencing clinical practice. The development of this approach and its adaptation in both struc- ture and function under different countries' initiatives provide an important opportunity to assess its utility under a variety of condi- tions. Evaluations of programs' effectiveness in changing clinical practice patterns must take into account attributes of the technolo- gies considered, as well as the consensus statements themselves and a variety of organizational and environmental factors that influence
82 CONSEiNSUS DElfELOPMENT the dissemination process. The consensus statement should be rec- ognized as only one influence on clinical practice and should be considered an integral part of a broader strategy of dissemination and technology transfer. REFERENCES Backe, B., and P. Nafstad. 1989. Development in practice of diagnostic ultrasound in Norway after the consensus conference recommending routine screening. Paper presented at the Fifth Annual Meeting of the International Society of Technology Assessment in Healthcare, London. Calltorp, J. 1988. Consensus development conferences in Sweden: Effects on health policy and administration. International Journal of Technology Assessment in Health Care 4:75-88. Downs, G., and L. Mohr. 1976. Conceptual issues in the study of innovation. Administrative Sciences Quarterly 21~4~(December):700-714. Eisenberg, J.M. 1986. Doctors' Decisions and the Cost of Medical Care: The Rea- sons for Doctors' Practice Patterns and Ways to Change Them. Ann Arbor, Mich.: Health Administration Press. Fennell, M. 1984. Synergy, influence and information in the adoption of adminis- trative innovations. Academy of Management Journal 27:113-129. Fennell, M., and R. Warnecke. 1988. The Diffusion of Medical Innovations. New York: Plenum Press. Fineberg, H.V. 1985. Effects of clinical evaluation on the diffusion of medical technology. Pp. 176-207 in Assessing Medical Technologies, Institute of Medi- cine, National Academy of Sciences. Washington, D.C.: National Academy Press. Greer, A. 1988. The state of the art versus the state of the science. International Journal of Technology Assessment in Health Care 4:5-26. Jacoby, I., and S.M. Clarke. 1986. Direct mailing as a means of disseminating NIH consensus statements. Journal of the American Medical Association 255~10~:1328- 1330. Johnsson, M. 1988. Evaluation of the consensus conference program in Sweden: Its impact on physicians. International Journal of Technology Assessment in Health Care 4:89-94. Kanouse, D.E., R.H. Brook, J.D. Winkler, J. Kosecoff, S.H. Berry, G.M. Carter, J. P. Kahan, L. McKloskey, W.H. Rogers, C.M. Winslow, G.M. Anderson, L. Brodsley, A. Fink, and L. Meredith. 1987. Changing medical practice through technology assessment: an evaluation of the NIH consensus development pro- gram. Santa Monica, Calif.: RAND Corporation. Kosecoff, J., D.E. Kanouse, W.H. Rogers, L. McCloskey, C.M. Winslow, and R.H. Brook. 1987. Effects of the National Institutes of Health consensus develop- ment program on physician practice. Journal of the American Medical Associa- tion 258~19~:2708-2713. Lomas, J. 1989. Profile for the consensus development program in Canada. Paper prepared for the Institute of Medicine Workshop on Consensus Development for Medical Technology Assessment, London.
DISSEMINATION AND IMPACT 83 Lomas, J., M.W. Enkin, G.M. Anderson, K. Domnick-Pierre, E. Vayda, and W. Hannah. 1989. The role of consensus statements in changing physician's aware- ness and attitudes, knowledge and behavior. Paper presented at the Fifth Annual Meeting of the International Society of Technology Assessment and Healthcare, London. Robertson, T., and H. Gatignon. 1987. The diffusion of high technology innova- tions: A marketing perspective. In New Technology as Organizational ~nova- tion: The Development and Diffusion of Microelectronics, J. Pennings arid A. Buitendam, eds. New York: Ballinger. Scott, W.R. In press. Innovations in medical care organizations. Medical Care Review, Summer. Waitzkin, H. 1983. The Second Sickness. New York: Free Press.
Program Profiles
