Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
12 Determining Outcome We can no longer afford to provide health care without Mowing more about its successes aM failures. The Era of Assessment and Accountability is dawning at last; it Is the . . . Iatest~ut probate) not the last~hase of our efforts to achieve an equitable health care system, of satisfactory quality, at a price we can afford Arnold Relman The Rationale for Outcome Determination This report focuses primarily on ways to improve the outcome of treatment for alcohol problems. In previous chapters it was suggested that the clarification of basic concepts, comprehensive pretreatment assessment, more precise characterizations of the treatments provided, and careful matching of individuals to treatment by means of explicitly stated and modifiable guidelines would enhance the proportion of positive outcomes. In this chapter the committee examines issues involved in understanding the target of all of this activity, the outcome of treatment. For purposes of orientation, a distinction may be drawn between the short-term goals of treatment and its outcome. The goals of treatment include detoxification (where required); the reduction or elimination of alcohol use; the concomitant reduction of the signs and symptoms and of the consequences of alcohol use; the resolution of intercurrent medical, psychiatric, and social problems; and a modification in attitude toward drinking behavior leading to a commitment to its amelioration in future. The attainment of these goals is a major therapeutic achievement, a fact that should not be obscured by what follows. Goals will not be extensively discussed in this chapter, although it should be emphasized that, as with outcomes, there is a great need for their more frequent and detailed documentation. The outcome of treatment has to do with the maintenance of these goals over the longer term; that is, with whether the commitment to the amelioration of drinking behavior has been realized. Although achievement of the short-term goals of treatment is laudable, it does not ensure a favorable outcome. And as the epigraph suggests, outcome is the bottom line. A health economist put it with characteristic directness in discussing the issue of quality of care: Quality means: did the patients get better?" (McClure, 1985a:43~. Organizations that accredit treatment facilities have increasingly stressed the evaluation of treatment outcome as a prerequisite for accreditation (Schroeder, 1987~. One such organization recently advised as follows: The Commission now has an entire subsection of standards specifically focused on program evaluation and the importance of program results. This reflects a blending of technical requirements and the Commission's long-term emphasis on utilization of outcome measurements . . . organizations should keep in mind that an important focus of accreditation site surveys will be on the extent that evaluation reports are actually assisting them to accomplish their goals. (Commission on Accreditation of Rehabilitation Facilities, 1988:10-11) 3~13
314 BROADENING IlIE BASE OF TREATMENT FOR ALCOHOL PROBLEMS Current interest in treatment outcome is fueled in large measure by concern over the growth of health care expenditures. There is an understandable desire to be certain that a return on such expenditures is being realized. concerned about treatment outcomes-and for other reasons. Yet clinicians have long been The provision of treatment in the absence of knowledge of results has been likened to playing golf in the fog (Ziskin, 1970~. One can stand at the tee, drive balls into the distance with impeccable form, fantasize about what good drives they were, and congratulate oneself on being a surpassingly good golfer. Yet what does this avail if one does not know where the balls are landing? Golfing under these circumstances becomes an exercise in unreality; no reasonable feedback can be provided that will be useful in improving one's game. In addition to its potentially improving effect on clinical practice, there are ethical and legal reasons for promoting a systematic knowledge of treatment outcome. Because an effective treatment may nevertheless prove harmful to some individuals (see Chapter 6), an ethical obligation exists to monitor outcome so that deleterious effects can be detected and countered. Such an obligation is consistent with the principle of primum non nocere the first duty of the treater is to do no harm. Should the detection of adverse effects of treatment be overlooked or ignored by treaters, remedies may be sought at law. Wishing to practice in the most effective and ethical manner possible, clinicians commonly take steps to assure positive outcomes. In the regular follow-up of individuals after treatment, they observe them closely and often modify their therapeutic approaches accordingly. They are also at increased pains to utilize those treatments that sophisticated research studies have indicated are efficacious. Unfortunately, accumulated experience now suggests that such strategies, while laudable, are not sufficient. ~ ~ _ . Following patients clinically is an appropriate and useful practice, but one that is directed primarily at a determination of whether the goals of treatment were achieved rather than at treatment outcome. It is likely to provide an incomplete picture of outcome results: attrition is considerable, and those who fail to improve are less likely to return for follow-up appointments. If a follow-up visit is not carefully structured, it may not systematically and quantitatively explore the multiple aspects of outcome that are now felt to be important. If follow-up is carried out by those who administered treatment, there is a tendency to perceive more favorable outcomes than may actually exist, and treatment recipients will be reluctant to bring forth evidence that the treatment they have been given has not been effective. Being guided by research studies in the provision of treatment may enhance the probability of positive results, but it does not guarantee them. The results of a treatment outcome study are considered positive if, in the aggregate, the outcomes are significantly better than those following either no treatment or a comparison treatment. But it is rare that all treated subjects have positive outcomes, and the relevant clinical question is whether a particular individual had a positive outcome. Moreover, there are factors that may constrain the more general applicability of research findings to clinical practice. The subjects of research studies frequently differ in important ways from persons seen in clinical settings (Seiden, 1961; Hlatky et al., 1984; Longabaugh and Lewis, 1988~. Although a treatment may be effective for research subjects, it is not necessarily effective for a very different clinical population. Apart from subject differences, there may also be differences in the treatment as delivered. It used to be common for discussions of treatment outcome to be graced by a modest qualifying phrase: "Treatment X had the following results in our hands." Although now used only infrequently, the phrase is still highly meaningful, especially in the case of complex nonbiological interventions, such as those that are often utilized in the treatment of alcohol problems. Even with adequate quality assurance mechanisms in place (see Chapter 5) there are likely to be differences between a treatment as studied in a research
DETERMINING OUTCOME 315 setting and the same treatment as delivered in a treatment setting. This state of affairs is not necessarily unfortunate-variations on a basic treatment theme may be advantageous in dealing with an extremely variable target population- but it does mean that results obtained in a research setting cannot automatically be extrapolated beyond it. Thus, although the carefully controlled research study may be an important method for exploring treatment efficacy, its clinical applicability is limited by numerous factors. From the standpoint of an individual seeking treatment, controlled studies have commonly been carried out in other programs, at other times, by different staff. What the prospective consumer of treatment services wants to know, however, is whether, given his problem, his characteristics, and his circumstances, he can reasonably expect to achieve a positive result from this program, at this time, and with these staff (cf. Paul, 1967; Pattison et al., 1977~. The desire for such knowledge is shared by prospective third-party reimbursers. Knowledge of outcome results is a cornerstone of the "buy right" strategy advocated for all purchasers of health care (McClure, 1985a,b). According to this strategy, third party reimbursers can improve the quality of treatment by systematically shifting their economic support to providers who produce the best results at the most reasonable cost. In the past the limited number of treatment providers constrained such a strategy, but the growth of the treatment enterprise has now made it feasible. What may be concluded from the foregoing considerations of clinical practice, research design, accreditation, ethics, legal considerations, marketing, and financing is relatively straightforward. Knowledge of the achievement of treatment goals is important, and very much to be encouraged, but it is not sufficient. Individual treatment programs must develop systematic and detailed knowledge of the outcomes experienced by those to whom they deliver services. Because of constant flux in critical dimensions of the treatment scene over time (changes in the patient population, in the treatment staff, in the available alternatives, in funding policy, etc.), such knowledge cannot be occasional but must be ongoing. Although information regarding the achievement of short-term treatment goals is sometimes sought, the development of a comprehensive understanding of outcome is not common in the treatment of alcohol problems. Why this should be the case when multiple considerations favor knowledge of outcome is uncertain, and (as will be seen in the next section) there are some important exceptions. The reasons for the relative neglect of outcome determination may not be specific to the treatment of alcohol problems, since it is also a feature of the treatment of medical problems (Schroeder, 1987; Bunker, 1988; Relman, 1988; Lohr et al., 1988; Wennberg, 1988~. Some Examples of Systematic Outcome Determination In some instances, concerted attempts have been made to determine the outcome of treatment for alcohol problems. The state of Oklahoma operates a mental health information system that generates data on the performance of all psychosocial treatment programs. An addition to this system has been made by the state agency responsible for the treatment of alcohol problems: [W]e require the alcoholism programs to submit follow-up data on a random sample of all the clients taken in. This information is collected in a standardized form, and the service is reimbursed [for the collection of this data] in the amount designated by the schedule of payments. The random sample is generated every month by the computer from the pool of patients intaken jsic] by the program 6 months earlier. The level of completion of the follow-up quota is an important criterion in the decision
316 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS of our department to renew or deny renewal funding by each program. (Paredes et al., 1981:384~. More recently (as of January 1, 1988), the state of Minnesota has instituted a requirement that all alcohol treatment programs receiving public monies must participate in a similar sort of management information system that includes regular follow-up of treated individuals. The development of systems of outcome monitoring is not a feature of the public sector of service delivery exclusively, however. With the rise of utilization management in the treatment of alcohol problems (Korcok, 1988; Lewis, 1988) a concern with outcome has surfaced in the private sector. Given increasing competition among pro- viders of treatment for alcohol problems, documentation of positive outcomes may become a critical element in marketing. Commercial organizations that retail such documentation in the alcohol field, for example, the Chemical Abuse/Addiction Treatment Outcome Registry (CATOR), already include this inducement in their promotional materials. Some details of the CATOR operation may be cited to illustrate how such an organization operates. A private corporation located in Saint Paul, Minnesota, CATOR contracts with individual treatment programs on a per capita basis (currently $98 per client per year) to provide treatment outcome data. Program staff are trained by CATOR staff working onsite to administer client data forms and forms documenting the details of treatment. These forms are then submitted to CATOR, which conducts posttreatment follow-up of individual clients by telephone from its main office. The data that have been gathered are analyzed by CATOR staff, and feedback is provided to the individual programs that subscribe to the service, not only in terms of aggregate outcome for the program itself but in terms of how the program is doing in comparison with similar programs in the same registry. Such comparisons are possible because uniform data are obtained from all subscribing programs. The CATOR registry is quite large; its most recent version contains individual data on approximately 31,000 persons, with completed follow-up data on approximately 17,000 (N. G. Hoffmann, CATOR, personal communication, March 2, 1989~. Although it is used primarily to prepare reports for its subscribers, the registry can also be employed to explore general issues in the treatment of alcohol and drug problems. Thus reports have been prepared on 1,776 women in treatment (Harrison and Belille, 1987), on 1,824 adolescents in treatment (Harrison and Hoffmann, 1987), on 569 adults who completed outpatient treatment and were successfully followed for four consecutive six-month intervals after treatment (Harrison and Hoffmann, 1988), and on 2,303 adults discharged from inpatient treatment who were successfully followed in the same manner (Hoffmann and Harrison, 1988~. In summary, there are multiple reasons for favoring the systematic monitoring of the outcome of treatment, both in health care generally and in treatment for alcohol problems specifically. Many treatment programs attempt to develop information on the achievement of short-term treatment goals; although this is praiseworthy, it is not sufficient. The systematic determination of the status of individuals at the end of treatment is not common, and determination of outcome during the postdischarge period is even less common. Several examples can be cited in which monitoring of the outcome of treatment for alcohol problems has been and is currently being carried out. In view of the fundamental importance of knowledge of outcome, however, much more needs to be done.
DETERMINING OUTCOME 2 ~ ~ A Major Caveat For all of the reasons discussed above, the monitoring of outcome results is of central importance to the treatment of persons with alcohol problems. In addition to its many advantages, however, outcome monitoring has an important limitation. It does not prove that the outcomes observed following treatment are the result of the treatment provided. From the manner in which outcome information is often used in marketing treatment programs, this limitation seems not to be well understood. Imputations of treatment efficacy based on outcome monitoring are a regular feature of mass media advertising. The committee believes that outcome monitoring should be used only with an understanding of its limitations; it also considers an examination of its limitations to be a necessary counterbalance to the foregoing discussion. If a given result occurs following a given event, there is an understandable tendency to believe that the event caused the result. The belief is particularly compelling if the event was intended to cause the result. Treatment is intended to produce positive results; when a positive outcome is observed after treatment, the tendency is to conclude that it is due to the treatment provided. Certainly, that is one possibility. Yet a little reflection will indicate that there are alternative possibilities and that this sequence-treatment followed by a positive result-does not necessarily mean that the positive result was caused by the treatment. As an example, let us take the familiar sequence of the alarm clock going off, followed by the rising of the sun. That this sequence occurs is readily and repeatedly verifiable. But the alarm clock does not cause the sun to rise, as may also be readily verified by not setting the clock. The two events happen to occur in a time sequence identical to the one that would be observed if the second event in the sequence were the result of the first, that is, if there were a causal relationship between them. But there is not. The temporal sequence is consistent with causality but does not demonstrate causality. So frequent and ancient is the logical fallacy of the assumption that a temporal sequence demonstrates causality that there is a Latin tag for it: post hoc ergo propter hoc, or (roughly) After this, therefore because of this. It follows that improvement subsequent to treatment is not necessarily the result of the treatment provided. Of what might it be the result, then, if not of the treatment itself? There are many possible alternative explanations. Alcohol problems, like other problems, have a natural course~r, more accurately, several natural courses (see Chapter 2~. They may come and go. Treatment may have happened to occur in the middle of this process but may not have affected it (much as the alarm clock happened to ring while the sun was in the process of rising but did not cause it to rise). Life goes on while people are in treatment, and such life events as the threatened loss of a job or an important relationship may be the crucial determinants of outcome rather than treatment. That alcohol problems can improve in the absence of any formal treatment is well known (see Chapter 6~. None of these considerations negates the value or importance of treatment. There is ample evidence (see Chapter 5) that treatment can be crucial to positive outcome in many persons. Moreover, treatment can be carried out with certainty; nontreatment events or processes that might favor positive outcomes may, indeed, exist, but their occurrence is far from certain. Some alcohol problems come and go, but others do not. Even if improvement were eventually to occur without formal treatment, formal treatment may still be indicated because it might accelerate the process. Thus, treatment is very important; but the point here is that, because positive outcomes can occur that are related to factors other than treatment, proof that treatment has produced a positive outcome is not a simple matter.
318 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS To establish with certainty in a given instance that there was a causal relationship between treatment and improvement requires a more elaborate procedure. Basically, the results obtained in a group of persons who receive treatment must be compared with the results obtained in an essentially similar group who do not receive treatment. If there are differences in outcome between these two groups beyond what can be expected on the basis of chance, the results can be attributed to the treatment with confidence. The randomized controlled trial ARCH is this kind of procedure (see the discussion in Chapter 5~. Randomization is the method commonly used to ensure the essential comparability of the treated and untreated groups on factors affecting outcome. The term controlled, used in this context, means that the presence of treatment has been controlled for by having a comparison group that was not treated (or, in a common variant, received a different kind of treatment). Outcome studies characteristically look only at people who have been treated; they do not look at randomly selected, untreated controls drawn from the same admission population. Hence the RCI, aided by a suitably selected comparison group, is better able to test the effect of treatment. Yet the RCI has its disadvantages (see Chapter 5), and outcome monitoring has corresponding (and in many respects complementary) advantages. In a recent editorial in the New England Journal of Medicine, the disadvantages of the RCI in medicine generally were summarized as being of a practical nature: "We cannot afford to conduct randomized controlled trials for every test, procedure, or medication in use. To do so would require far too many research resources and would not produce results soon enough" (Greenffeld, 1989:1142~. In contrast, The scope of observational studies can be expanded much more easily than that of randomized controlled trials to include large numbers of patients and providers, maximizing the opportunity to gauge the effectiveness of routine medical care practices in various clinical settings, by various clinicians, and for various patient groups. Answers to questions about the effectiveness of care for important subgroups of patients (including those with specific coexisting morbid conditions), which would not be supplied by randomized controlled trials, can therefore be provided .... Longitudinal observational studies permit both the examination of a complex set of decisions, including the decision not to perform a procedure, and the assessment of the supportive care that follows. (Greenfield, 1989:1142) The editorial goes on to caution that "care must be taken in interpreting the results of longitudinal observational studies, even though they are intuitively appealing and offer a quick solution to the information needs of policy makers." (p. 1143) It speaks of the capability of such studies to "reduce our dependence on randomized controlled trials" (p. 1143) rather than eliminating the need for such trials. A similar note is struck in a recent Institute of Medicine background paper on the assessment of technological innovation in medical practice (Gelijns, 1989~. The virtues of observational methods, which are felt to have been enhanced by recent methodological developments, are enumerated (pp. iii, 51-52), but the recommendation is for a mixture of methods: "Evaluation of the risks and benefits of new technologies during their development will have to rely not only on experimental methods (including randomized controlled clinical trials) but also on improved observational methods of clinical conditions" (p. 51~. As these sources suggest, there is an important and highly complementary relationship between the RCE and the treatment outcome study. The RCI furnishes evidence that improvements in outcome are due to the treatment provided but only under the conditions of the experiment; it does not demonstrate, as indicated above, that the
DETERMINING OUTCOME 319 same results will consistently be produced in The real world. On the other hand, the treatment outcome study does not prove that positive outcomes are due to the treatment provided. But it does demonstrate that improvements have occurred sin the real worlds following the delivery of the treatment. Let us suppose that there has been (a) ample demonstration from controlled trials that a particular treatment can be efficacious and (b) ample demonstration from treatment outcome monitoring that the same treatment, concretely embodied in a specific treatment program, is consistently associated with positive results. If these two circumstances coexist, one can entertain a reasonable certainty that effective treatment is being provided. If either element (a) or (b) is absent, one cannot be as certain. Thus, an important goal for the immediate future is the increased implementation of both randomized controlled trials and routine monitoring of treatment outcome. Implementing Outcome Monitoring Setting the Stage for Outcome Monitoring Imagine a situation in which the following statement is made: "Sixty percent of the people who pass through our treatment program achieve a positive outcome. Let us assume that the statement is valid. How meaningful is it? HA cautious response to the presentation of any treatment outcome rate, it has been noted, Requires that we ask several questions" (Emrick and Hansen, 1983:1086~. One question that immediately comes to mind is "What sorts of alcohol problems do the people who come to your program exhibit?" A second question might be "What are the characteristics of the people that are treated in your program?" A third question might be "What sort of treatment is provided by your program?" In the absence of answers to these and other questions, a simple statement about the proportion of positive outcomes is not highly meaningful. Specifying the Problem arid the Individual A 60 percent positive outcome rate in a program seeing persons with mild alcohol problems of brief duration has a different meaning than the same rate in a program seeing persons with severe problems of long duration. A 60 percent positive outcome rate in a program seeing socially stable individuals has a different meaning than the same rate in a program seeing socially unstable individuals. The specification of problems and of individuals being considered for treatment has been discussed earlier in Chapter 10. In a very real sense the determination of outcome is a process of reassessment, in which the individual's status following treatment is compared with his status prior to treatment. As has been noted, "pretreatment functioning needs to be assessed for comparison with posttreatment adjustment, using parallel pre-post data-gathering procedures" (Emrick and Hansen, 1983:1082~. Assessment thus helps to set the stage for outcome determination. Specifying the Treatment A 60 percent positive outcome rate in an elaborate, time-consuming, and expensive form of treatment has a different meaning than the same rate in a simple, brief, and inexpensive form of treatment. The relevant question here is Outcome of what?" The specification of treatment is also important for the classification of treatment programs and for matching to particular treatments (see Chapters 3 and 11~. Briefly, there is a need to specify the treatments that are provided along multiple dimensions including treatment orientation or philosophy, the stage of the problem at which the treatment is directed, the setting of the program, the treatment modality utilized, its goals or objectives, the criteria for selecting individuals for treatment, its length, its intensity, its content, and other factors.
320 BROADENING lilE BASE OF TREATMENT FOR ALCOHOL PROBLEMS For purposes of outcome determination it is not enough to know what treatment was provided; one should also know what treatment the individual received. Many treatment programs offer several treatment options simultaneously (e.g. group therapy, individual counselling, and educational lectures), and it is assumed that all persons in the program receive all of them in similar proportions; in fact, this may not be the case (W. J. Filstead, Parkside Medical Services, personal communication, March, 1988~. If a medication such as disulfiram (Antabuse) is dispensed, it is important to know whether it is being taken. Experience suggests that compliance with a prescribed regime of medication does not always occur, and a monitoring mechanism may be advisable to ensure that the medication is actually being used (cf. Peachey and Annis, 1984; Peachey and Kapur, 1986~. Perhaps it is only stating the obvious to emphasize that, in order to understand treatment outcome data, one must be clear about the nature of the problems being treated, the nature of the individuals undergoing treatment, and the nature of the treatment. Unfortunately, clarity about these matters is too frequently lacking at the present time. It has long been a goal of research on the treatment of alcohol problems that all of these critical dimensions should be explicitly accounted for (but the goal is not always realized; see Sobell et al., 1987~. What is being urged, in a sense, is that the distinction between treatment and research practice be reduced to the point that they approximate each other. The Content of Outcome Monitoring Knowledge of an individual's status after completing treatment is much more meaningful if it can be compared closely with his or her status prior to treatment. Accordingly, there should be a parallelism between the coment of assessment and the content of outcome determination. A comparison of the "core indices [of outcome] to be used for all treatment-evaluation studies" noted in a recent review (Emrick and Hansen, 1983:1084-1085) and the content of assessment suggested in Table 10-2 of this report demonstrates such parallelism. Among the shared content domains are physical health, including morbidity and mortality; drinking behavior; other substance use; legal problems; vocational functioning; family and social functioning; emotional functioning; cognitive functioning; and the operation of situational variables and life events. Some aspects of a comprehensive pretreatment assessment might not be directly relevant posttreatment for example, a specification of the individual's treatment goals. On the other hand, it would be of great importance to determine at the follow-up point or points whether the goals initially viewed as appropriate had been achieved, or whether there had been a shift of goals during the course of treatment. A thorough posttreatment assessment of customer satisfaction might be valuable in several ways. Guidance about modifications in program content and process could be obtained, but a more positive and participatory feeling toward the program and toward treatment generally might also be engendered by the fact of soliciting such feedback. In addition, regular reports of customer satisfaction may serve as an incentive and reward to the staff. Prior treatment history would not be reassessed in outcome monitoring, but it would be essential to document any further treatment received in the interim between treatment and follow-up. Certain individual characteristics that might remain relatively unchanged (e.g., intelligence, personality, and family history) might not be reassessed, although specific reasons for doing so in individual cases come to mind. Because full testing of cognitive functioning is an expensive and time-consuming procedure, it might be repeated only if initial impairment had been detected. However, it is a common experience that marked improvement in cognitive functioning within the normal range can be characteristic of a positive treatment outcome, and the documentation of this improvement may have a salutary effect on the individual's outlook.
DETERMINING OUTCOME 321 There has been a tendency in the field to rely primarily on changes in the level of use of alcohol to determine the outcome of treatment. Certainly, level of use is a critical focus for both treatment and outcome, and its status will bear a strong relationship to other outcome parameters (Babor et al., 1988~. But the relationship is not invariable and may be quite discordant in some individuals, particularly those with lower levels of alcohol use (cf. Edwards, 1974~. A case can therefore be made, as with assessment, for the use of multiple dimensions in the determination of outcome. "Because of the non- orthogonal relationships among treatment outcome domains," one authoritative paper states succinctly, Use of multiple outcome measures is essential" (Emrick and Hansen, 1983:1084~. Brief mention should be made here of a problem in selecting individuals for treatment that was touched upon earlier in this chapter. It has long been clear that certain population characteristics are associated in general with favorable outcomes. Those with less severe problems accompanied by fewer symptoms and consequences and of shorter duration, as well as such personal and situational characteristics as affluence, high intelligence, high educational levels, social stability, a high level of verbal skills and personal attractiveness are likely to do well in treatment and are more attractive to work with (cf. the TRAVIS syndromes of Schofield [19643~. A program could substantially improve its proportion of positive outcomes, and the ease of its staff, by choosing to deal only with problems and individuals of this kind. Although it is true that some such individuals do not do well and that persons in this group may have as profound a need for treatment as any others, a high proportion of positive outcome results in a population of this kind carries a rather different meaning than would be the case in other populations (e.g., a population of skid row inhabitants). Less substantial gains in a more impaired and less advantaged group may represent an equal or superior therapeutic achievement. Thus, in the understanding of outcome results, relative rather than absolute standards are best employed. At present, there is great variability in the measures that are used when outcomes are examined (Emrick and Hansen, 1983; Sobell et al., 1987~. The use of a uniform set of treatment outcome criteria would be an enormous advantage because it would permit the aggregation of data and the comparison of treatment outcomes across programs. For example, with uniform outcome criteria it would be possible to compare the outcomes of individuals treated with pharmacotherapy and those treated with counseling. At present, data cannot be aggregated in this way because different and often incompatible outcome measures are used. The argument that particular programs need to present their data in particular ways to satisfy various data and funding requirements does not preclude the use of uniform criteria as well. If unique information is required, it can be collected in addi- tion to shared uniform measures. As with the development of a uniform assessment process, agreement on uniform outcome criteria needs to be forged through a series of consensus exercises involving all segments of the treatment community. Indeed, if the determination of outcome is conceptualized in large measure as reassessment, consensus on the content of both as- sessment and outcome determination could be sought simultaneously. The increment in useful information that would result from uniformity in assessment and outcome data would be considerable and would contribute importantly to the further enhancement of . . positive treatment outcomes. The Process of Outcome Monitoring Training A quite practical aspect of viewing outcome monitoring as reassessment is that only one set of staff, rather than two, is required. The measures being parallel, training to use them for assessment is also training to use them for follow-up. In many
322 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS instances, staff will be reassessing individuals that they assessed initially; such familiarity may contribute to a greater likelihood that reassessment will be accepted and provides valuable feedback directly to relevant staff. Timing How long after treatment should outcome be determined? In practice the range has been very wide indeed, ranging from a few weeks to as many as five years (Ogborne, 1984). What period might be optimal is a knotty question. Outcome results tend to be unstable-that is, individuals who are evaluated at time A tend to have differ- ent outcomes than when evaluated at time B (Annie and Ogborne, 1980)-and it is questionable whether time A or time B represents the Cereals outcome. There is also the issue that "although a lengthy period of follow-up may cast important light on the course of patients' drinking and related problems posttreatment, as the period of evaluation is lengthened less of what is observed can be attributed directly to treatment" (Emrick and Hansen, 1983:1082). What one may be observing over lengthy periods of time is the effect of nontreatment factors (e.g., loss of jobs, promotions, marriages, divorces, etc.) rather than the effect of treatment (Cronkite and Moos, 1980). Accordingly, there may be a progressive decline in treatment effect from the point of treatment termination. Such a progressive decay curve is often seen in medical treatments, in which the active ingredient in therapy is what the treater does. It has also been seen in treatment for alcohol problems, as well as for similar problems (Hunt et al., 1971~. Individuals pursuing such a posttreatment course have been termed "faders" (Moos et al., 1982). But there are other patterns of response over time. For example, in some treatments for alcohol problems, individuals are taught skills that they then apply to their life situations. With increased practice, they may apply the skills more effectively. In these circumstances the treatment effect may increase rather than decrease from the point of formal treatment termination. Such a pattern has been observed (Annie and Ogborne, 1980); in this case the individuals have been called ~sleepers" (Moos et al., 1982). If different forms of treatment have variable effectiveness curves over time after treatment, how can one specific point in time be chosen as the "gold standard" for determination of outcome? The answer probably is that it cannot be. A compromise would be to assess outcome at regular intervals following treatment and to continue to do so for a reasonable period of time. A schedule of reassessment, then, at 6 months, 12 months, and 18 months seems defensible, especially if the reassessment serves clinical as well as outcome determination purposes. For clinical reasons alone one would wish to evaluate individuals posttreatment on a schedule of at least this stringency. It would be possible, with such a pattern, to identify both Faders" and nsleepers.n Sampling Outcome monitoring can be carried out for all persons who enter treatment. As an alternative the determination can be made on selected samples (e.g., every third admission) as in the Oklahoma system. Using a sample is an effective procedure if a generalized knowledge of outcome is what is required. However, if the purposes of outcome monitoring are also in part clinical, it is more reasonable to opt for the monitoring of outcome for all persons who have entered treatment. For clinical purposes, one must know the outcome of treatment in every case, not excluding those who have failed to complete the prescribed course of treatment. The effectiveness of sampling in meeting the ethical obligations of prisms non nocere and in preventing the potential legal consequences of failure to identify harmful effects of treatment upon particular individuals is unclear. Setting In research studies, follow-up interviews are commonly carried out by means of a face-to-face interview, often in the patient's home. As an alternative, patients may be asked to return to the scene of their treatment for follow-up interviews. Because either of these options involves direct personal contact, additional direct observations can
DETERMINING OUTCOME 323 be made, and such procedures as breathalyzer testing or the use of biological markers can be deployed. CATOR, as noted earlier, has conducted its follow-up interviews by telephone; others have used mailed questionnaires. These techniques, although less costly and time-consuming, do not permit the observations and methods available with direct contact and are thought to be less likely to produce valid responses, particularly from those who have not been satisfied with the treatment they have received. There is little in the way of empirical data to substantiate this impression. It may be that, to achieve a satisfactory response level-the higher, the better-a combination of methods will need to be used. Corroboration The issue of the reliability of self-reports was discussed in Chapter 10. Self-reports seem inherently to be neither reliable nor unreliable; rather, they are markedly affected by the circumstances in which they are elicited. Under favorable circumstances, self-reports achieve a satisfactory level of reliability. On the other hand, it is undoubtedly useful to obtain corroborating data from wari^~c f`~rtPrn~1 cn'1r~Pc. Anti at least in research studies. there has been an increasing tendency to do so (Sobell et al., 1987~. The usual sources include family and other informants, public records, and the use of various biochemical tests to measure alcohol consumption either directly (e.g., breathalyzers, urinalyses) or indirectly (e.g., such markers as gamma glutamyl transpeptidase and high-density lipoprotein). These measures increase the validity of outcome information but at a cost of time, money, and effort. Utility Outcome information may be principally useful in persuading third-party payers to purchase services from a particular treatment program, in persuading prospective clients to enter the program, and in qualifying for accreditation. In a larger sense, however, outcome data provide an ethical justification for purveying treatment and a means of improving its effectiveness. From an ethical standpoint the provision of treatment in the absence of knowledge of results is a questionable procedure. (Of course, if the results are not favorable, the ethical problems in continuing to provide treatment may be insu- perable.) In most instances, outcome information is likely to indicate that admission to a given treatment program is followed by positive outcomes in a significant proportion of the people who seek its services; at the same time, however, it is followed by no significant change or by a worsening of problems in another but also significant proportion. A number of responses are possible in the face of such information: (a) improvements to the treatment being currently provided may be introduced, after which outcome may be examined again; (b) additional treatments that seem likely to help those who are not being helped may be introduced, and their outcomes may be examined; and (c) those who are not being helped by the treatment provided may be preferentially referred to programs whose outcome information indicates a higher probability of a positive outcome for individuals with their characteristics and types of problems. These alternatives are more extensively discussed in Chapters 11 and 13. All of them may substantially improve treatment outcomes, both for the individual programs and for the persons seeking treatment. , in, V %*" arm._, arm. ~ _. __ , _ _, _ The Locus of Responsibility for Outcome Monitoring The monitoring of treatment outcomes can be carried out under differing auspices. Responsibility for determining outcome can be taken (a) by individuals external to the program, (b) by the program itself, or (c) by some combination of the two. Theoretical and practical advantages may be advanced for all three approaches. The principal advantage of external evaluation is greater objectivity. Programs themselves may be presumed to have an interest in demonstrating that their treatment is associated with a high level of positive outcomes; external evaluators presumably would not share such an interest. The principal disadvantages of external evaluation include lack
324 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS of understanding of the program being evaluated, lack of availability, and cost. In Chapter 10, some of the advantages of an assessment process that was functionally independent of treatment were discussed. Similar advantages would accrue to functionally independent determination of treatment outcome; and, because outcome determination and reassessment are substantially the same process, both could be performed by the same functional unit. Because this option has already been discussed, the focus here will be on options (b) and (c). Programs themselves may conduct treatment outcome monitoring. Indeed, when clinical as opposed to research treatment programs are being evaluated, this method is probably the most common. Program personnel are at least geographically available and presumably sympathetic to and knowledgeable about their own programs. In this case, information on treatment outcomes would be directly useful to the very personnel who carried out the evaluation and hence would tend to ensure its pertinence and perhaps also narrow the notorious treatment-research gap (cf. Garfield, 1978; Miller, 1987~. Mounting research projects within treatment programs may also foster a more observing and objective attitude among staff toward their program than would otherwise be the case. As an example of a within-program evaluation, half of a group of 100 patients who were seen for initial assessment of alcohol problems were randomly sent "a personal letter expressing concern for the patient's well-being and repeating our invitation for further assistance." Fifty percent of those sent the letter returned for additional contact, as compared with 31 percent of those not sent the letter; 76 percent returned the same day the letter was received, as compared with 12.5 percent; and 80 percent returned sober, as compared with 31 percent. All results were significantly beyond what would have been expected on the basis of chance (Koumans and Muller, 1965~. A second and similar study demonstrated the effectiveness of a telephone call versus no call (Koumans et al., 1967~. A large residential treatment program visited by staff and committee members of this study conducted an evaluation of the effects of their program on persons who had alcohol or drug problems (or both) and eating disorders. The evaluation was carried out by an internal research group, which designed the study in close collaboration with clinical staff. Records were reviewed to establish the prevalence of eating disorders, and special questionnaires to be used both at admission and at follow-up were devised. The key finding was that individuals with eating disorders were found to have outcome results with respect to alcohol or drug problems that were similar to those of individuals without eating disorders. After much discussion, the program decided to continue to admit individuals with both alcohol or drug problems and eating disorders (J. Spicer, Hazelden Foundation, personal communication, 1988~. If programs conduct their own evaluations, problems arise in terms of validity, allocation of resources, and the uniformity of the data gathered. How objective program personnel would be in evaluating the results of their own work is an important issue. If the research were in the hands of program but nonclinical personnel, as in the example immediately above, the effects of bias might be somewhat attenuated. Even if the results were in fact not biased, their persuasiveness to others might be less than with an external evaluation. Finally, when individual programs gather data on their own clients they tend to gather them in an idiosyncratic manner. This lack of uniformity makes comparisons across programs, even on such seemingly straightforward items as demographic variables, difficult or impossible. Consensus on outcome measures, as recommended above, could be helpful in reducing this problem. At present, most programs do not possess staff devoted to outcome monitoring. One approach would be to reallocate staff assignments so that some proportion of clinical time (and thus some proportion of the program budget, depending upon the intensity of follow-up to be done and the methods to be employed) was used for this purpose.
DETERMINING OUTCOME 325 Alternatively, individual programs could be enriched by providing them with additional personnel to design and carry out outcome monitoring. Programs may collaborate with external organizations to monitor their outcomes. CATOR is an example of this approach. It relies upon program personnel (after training) to produce descriptive data on patients and treatments but does its own follow-up and data analysis. In its promotional material, CATOR stresses the value of its externality: "Perhaps most importantly, potential patients and referral sources know you care about treatment outcomes by your willingness to have them audited by an objective, respected outside source" (Belille, n.d. [ca. 1987~:4~. The state systems in Oklahoma and Minnesota that were described earlier are similar in their approach, in that they involve collaborations between programs and external agencies. However, they differ in that treatment program personnel, rather than personnel of the collaborating agency, are responsible for conducting the outcome determinations. This practice also changes the financing pattern; in Oklahoma, for example, programs are paid for conducting outcome determinations on a prearranged per capita basis. Also, of course, the basis of participation differs; participation in CATOR is optional, but partic- ipation in the state systems is obligatory for programs receiving public funds. It would appear from this brief consideration of the locus of responsibility for outcome monitoring that all of the options described have both advantages and disadvantages. If validity is the bottom line with respect to outcome determination (and it is for most of the purposes that outcome information is asked to serve) then the maximum feasible degree of externality in outcome determination is desirable. If outcome determination for practical reasons cannot be wholly external, some combination of internal and external loci of responsibility would be the next most desirable option. Examples would be a commercial organization such as CATOR in the private sector or the state systems in Oklahoma and Minnesota in the public sector. Providing that steps were taken to ensure functional independence, the use of an assessment or research group within a program might be yet another option. Wholly internalized outcome monitoring may be useful for various local purposes but will tend not to be persuasive beyond the confines of the program. Nevertheless, as indicated above, such internal program research can be of great value in making sound decisions about the future course of the program. In addition, a program that has become accustomed to such internal research may well develop a capacity for objective self-scrutiny that will eventually result in an openness to external evaluation. Although not a fully satisfactory method of determining treatment outcome, internal monitoring is much to be preferred to an absence of effort toward this end. The Funding of Outcome Determination It is generally assumed that the introduction of widespread, comprehensive, and ongoing outcome monitoring, whether in the alcohol treatment field or in the medical treatment field generally, will raise the costs of providing treatment. At the least it is assumed that outcome monitoring will require an initial investment until a payoff of improved treatment efficiency is realized. Thus a medical periodical has editorialized that To achieve these objectives will require much new financial support and unprecedented cooperation among physicians, government, private insurers, and employers" (Relman, 1988:1222). Some are uncertain whether a financial payoff will in fact be realized. In calling for a national system of "outcomes management," Ellwood (1988) says that such a program Will not automatically favor a decrease or increase in health care expenditures (p. 1556~.
326 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS This is especially the case if any increase in the efficiency of treatment makes it more attractive so that demand for treatment is also increased. As will be discussed in Chapter 21, the cost of treatment for alcohol problems is great, but is dwarfed by the cost of the consequences of alcohol problems. Systematic knowledge of outcome and its application to treatment may increase effectiveness and efficiency by improving treatment matching. The committee considers it likely that the overall costs of treatment may rise as a result of its recommendations because savings owing to improved effectiveness and efficiency will be more than offset by additional costs arising from the treatment of greater numbers of persons who are newly attracted to an expanded and improved system. Even though the costs increase, however, they will continue to represent only a fraction of the cost of the consequences of alcohol problems. The effort involved in determining treatment outcome and utilizing that knowledge to improve treatment is thus likely to prove worthwhile. More people will be treated, and they will be treated more appropriately; these constitute important benefits. Conclusions and Recommendations There is much to be said for determining whether the short-term goals of treatment for alcohol problems, such as reduction in the level of use of alcohol, have been achieved. More needs to be done in this regard. But a more pressing (although related) need is the determination of the longer-term outcome of treatment. Not a substitute for more rigorous controlled techniques that can demonstrate treatment efficacy, outcome monitoring nevertheless offers benefits in that it addresses many important issues, is more readily implemented on a broad basis, and complements significantly what can be learned in other ways. Ideally, the outcome of treatment for all individuals entering specialized treatment programs for alcohol problems should be determined. There are a multiplicity of reasons for such a course, including ethical reasons, but the principal purpose would be to improve the ability to provide the most effective treatment to each individual by serving as a guide to matching. To reach this goal, consensus must be achieved or' the need for outcome determi~ifor', on. the parameters to be used in determining outcome, and on the optimal waylays) to go about making outcome determinations. Many variant approaches to all of these matters are possible. The committee believes, however, that a quantum increment ire alteration to outcome determir~wr' Is crucu~l to the future of the effort to treat alcohol problems. REFERENCES Annis, H. M., and A. C. Ogborne. 1980. The temporal stability of alcoholism treatment outcome results. Addiction Research Foundation, Toronto. Babor, T. F., Z. Dolinsky, B. Rounsaville, and J. H. Jaffe. 1988. Unitary versus multidimensional models of alcoholism treatment outcome: An empirical study. Journal of Studies on Alcohol 49:167-177. Belille, C. n.d. (ca. 1987~. Bringing definition to a soft science. Saint Paul, Minn.: Chemical Abuse/Addiction Treatment Outcome Registry (CATOR), Ramsey Clinic. Bunker, J. P. 1988. Is efficacy the gold standard for quality assessment? Inquiry 25:51-58. Commission on Accreditation of Rehabilitation Facilities (CARFV). 198$. Program Evaluation in Alcoholism and Drug Abuse Treatment Programs. Tucson, Arizona: Commission on Accreditation of Rehabilitation Facilities. Cronkite, R. C., and R. H. Moos. 1980. Determinants of the posttreatment functioning of alcoholic patients: A conceptual framework. Journal of Consulting and Clinical Psychology 48:305-316.
DETERMINING OUTCOME 327 Edwards, G. 1974. Drugs, drug dependence, and the concept of plasticity. Quarterly Journal of Studies on Alcohol 35:176-195. Ellwood, P. M. 1988. Outcomes management: A technology of patient experience. New England Journal of Medicine 318:1549-1556. Emrick, C. D., and J. Hansen. 1983. Assertions regarding effectiveness of treatment for alcoholism: Fact or fantasy? American Psychologist 38:1078-1088. Garfield, S. L. 1978. Research on client variables in psychotherapy. Pp. 191-232 in Handbook of Psychotherapy and Behavior Change: An Empirical Analysis, 2nd ea., S. L. Garfield and A. E. Bergin, eds. New York: John Wiley and Sons. Gelijns, A. C. 1989. Technological Innovation: Comparing Development of Drugs, Devices, and Procedures in Medicine. Washington, D.C.: National Academy Press. Greenfield, S. 1989. The state of outcome research: Are we on target? New England Journal of Medicine 320:1142-1143. Harrison, P. A., and C. A. Belille. 1987. Women in treatment: Beyond the stereotype. Journal of Studies on Alcohol 48:574-78. Harrison, P. A., and N. G. Hoffmann. 1987. CATOR 1987 Report: Adolescent Residential Treatment: Intake and Follow-Up Findings. Saint Paul, Minn.: Chemical Abuse/Addiction Treatment Outcome Registry (CATOR), Ramsey Clinic. Harrison, P. A., and N. G. Hoffmann. 1988. Adult Outpatient Treatment: Perspectives on Admission and Outcome. Saint Paul, Minn.: Chemical Abuse/Addiction Treatment Outcome Registry (CATOR), Ramsey Clinic. Hlatky, M. A., K L. Lee, F. E. Harrell, Jr., R. M. Califf, D. B. Ptyor, D. B. Mark, and R. A. Rosati. 1984. Tying clinical research to patient care by use of an observational data base. Statistics in Medicine 3:375-384. Hoffmann, N. G., and P. A. Harrison. 1988. Treatment Outcome: Adult Inpatients Two Years Later. Saint Paul, Minn.: Chemical Abuse/Addiction Treatment Outcome Registry (CATOR), Ramsey Clinic. Hunt, W. A., L. W. Barnett, and L. G. Branch. 1971. Relapse rates in addiction programs. Journal of Clinical Psychology 27:455-456. Korcok, M. 1988. Managed Care and Chemical Dependency: A Troubled Relationship. Providence, R.I.: Manisses Communications Group, Inc. Koumans, Al J. R., and J. J. Muller. 1965. Use of letters to increase motivation for treatment in alcoholics. Psychological Reports 16:1152. Koumans, A. J. R., J. J. Muller, and C. F. Miller. 1967. Use of telephone calls to increase motivation for treatment in alcoholics. Psychological Reports 21:327-328. Lewis, J. S. 1988. Growth in managed care forcing providers to adjust. Alcoholism Report 16~24~:1. Lohr, K N., K D. Yordy, and S. O. Thier. 1988. Current issues in quality of care. Health Affairs 7:5-18. Longabaugh, R., and D. C. Lewis. 1988. Key issues in treatment outcome studies. Alcohol Health and Research World 12:168-175. McClure, W. 1985a. Buying right: The consequences of glut. Business and Health 2(9):43-46. McClure, W. 1985b. Buying right: How to do it. Business and Health 2(10):41-44. Miller, W. R. 1987. Behavioral alcohol treatment research advances: Barriers to utilization. Advances in Behavior Research and Therapy 9:145-164. Moos, R. H., R. C. Cronkite, and J. W. Finney. 1982. A conceptual framework for alcoholism treatment evaluation. Pp. 1120-1139 in Encyclopedic Handbook of Alcoholism, E. M. Pattison and E. Kaufman, eds. New York: Gardner Press.
328 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS Ogborne, ~ G 1984. Issues in follow-up. Pp. 173-215 in A System of Health Care Delivery, vol. 3, F. B. Glaser, H. M. Annis, H. A. Skinner, S. Pearlman, R. L. Segal, B. Sisson, A. C. Ogborne, E. Bohnen, P. Gazda, and T. Zimmerman. Toronto: Addiction Research Foundation. Paredes, A, D. Gregory, and O. H. Rundell. 1981. Empirical analysis of the alcoholism services delivery system. Pp. 371-404 in Research Advances in Alcohol and Drug Problems, vol. 6, Y. Israel, F. B. Glaser, H. Kalant, R. E. Popham, W. Schmidt, and R. G. Smart, eds. New York: Plenum Press. Pattison, E. M., M. B. Sobell, and L. C. Sobell. 1977. Emerging Concepts of Alcohol Dependence. New York: Springer Publishing Company. Paul, G. L. 1967. Strategy of outcome research in psychotherapy. Journal of Consulting Psychology 31:109-118. Peachey, J. E., and H. M. Annis. 1984. Pharmacologic treatment of chronic alcoholism. Psychiatric Clinics of North America 7:745-756. Peachey, J. E., and B. M. Kapur. 1986. Monitoring drinking behavior with the alcohol dipstick during treatment. Alcoholism: Clinical and Experimental Research 10:663 666. Relman, ~ S. 1988. Assessment and accountability: The third revolution in medical care. New England Journal of Medicine 319:1220-1222. Schofield, W. 1964. Psychotherapy: the Purchase of Friendship. Englewood Cliffs, NJ.: Prentice-Hall. Schroeder, S. A. 1987. Outcome assessment 70 years later. Are we ready? New England Journal of Medicine 316:160-162. Seiden, R. H. 1961. The use of Alcoholics Anonymous members in research on alcoholism. Quarterly Journal of Studies on Alcohol 21:506-509. Sobell, M. B., S. Brochu, L. C. Sobell, J. Roy, and J. A. Stevens. 1987. Alcohol treatment outcome evaluation methodology: State of the art 1980 84. Addictive Behaviors 12:113-128. Wennberg, J. E. 1988. Improving the medical decision-making process. Health Affairs 7:99-106. Ziskin, J. 1970. Coping with Psychiatric and Psychological Testimony. Beverly Hills, Calif.: The Law and Psychiatry Press.
