DEFINITIONS OF SUCCESS
Evidence about treatment outcomes has become increasingly important in the health field in general and has become especially important in the behavioral health care field. A wide range of effective treatments are available for individuals with mental health and substance abuse problems, including psychosocial, pharmacological, and educational interventions. By now a large body of research demonstrates that treatment for these problems can be effective (Hubbard et al., 1989; McLellan et al., 1996; Polich et al., 1981; Simpson and Sells, 1990; Tims et al., 1991).
Defining outcomes for behavioral health problems is difficult. First, there are important differences among mental health and alcohol or other drug problems. For example, the same outcomes should not necessarily be expected for each drug of abuse, for example, heroin versus alcohol, or for every person with a substance abuse problem, for example, a pregnant woman, an adolescent, and a person with a dual diagnosis of mental health and substance abuse problems. The same is the case for mental health problems; outcomes differ by diagnoses (e.g., mild or major depression, chronic schizophrenia vs. adjustment disorder) and depend on individual characteristics (e.g., children vs. adults). It is appropriate to expect different outcomes at different stages of treatment, as the review of McLellan et al. in Appendix B points out. Differences in types of problems, the severities of problems, stages of illness and recovery, and other patient characteristics require a range of services and a multidimensional approach to measuring outcomes (Burnam, in press; Miller et al., 1995).
A second issue affecting the selection of outcomes measures has to do with
the actual goals of treatment and the lack of consensus about what is considered successful treatment. For mental health problems, the goals of treatment differ according to the diagnosis, the severity of the illness, and past responses to treatment. Treatment generally begins with a comprehensive assessment that results in an individualized treatment plan. In many cases, full recovery from a mental health problem can be expected, with an individual returning to his or her former preillness level of functioning. Treatment also may have the goal of controlling symptoms or preventing relapses among those with recurring episodes of a disorder, such as depression. Improvements in functioning, rather than a complete return to preillness functioning, may be the goal for those with severe and chronic disorders such as schizophrenia. Improving quality of life by facilitating access to social services (e.g., through case management) may also be a goal, particularly for patients with severe debilitating and chronic mental disorders. Finally, the avoidance of violence may be a goal, such as when persons with suicidal or homicidal tendencies are restrained in a heavily supervised setting such as a hospital or locked nursing facility. Additional goals could include an improved ability to live independently in the community, or an ability to maintain employment in a supervised setting.
In the substance abuse field, a fundamental and sometimes controversial issue is whether treatment goals should be directed toward abstinence versus improvement. On the whole, alcohol treatment programs are oriented to an abstinence model rather than one of controlled drinking or improvement. Political considerations related to the management of illicit substances have played a large part in shaping treatment goals for chronic substance abusers (IOM, 1990a; Marlatt, 1983; Roizen, 1987). As a consequence of these strong ideological stances, some of the treatment interventions with proven efficacy (e.g., methadone maintenance with heroin addicts and controlled drinking strategies with carefully selected populations) are not often found in managed care treatment systems and therefore cannot be included in current managed care outcomes studies.
An additional issue affecting the definitions of outcomes for mental health and substance abuse treatment has to do with the public and policy expectations of treatment. Although some mental health and substance problems are chronic relapsing conditions like other medical problems, they also raise issues concerning individual and community responsibility that are not usually associated with other medical problems. Concerned stakeholders include individual clients, treatment programs, reimbursers, regulatory agencies or monitors, family members, agencies that have legal (the criminal justice system) or contractual relationships (employers), and the public. The goals of these groups are often conflicting, and the group whose interests are primary changes over time. These goals result in broad expectations of treatment, including reduced crime, improved health status, prevention of human immunodeficiency virus infection, reduction of unsafe sexual practices, improved employment, and improved family functioning. Reduced levels of alcohol or drug use or improved mental health symptoms are not
necessarily the predominant expectations by these groups (McLellan and Weisner, 1996).
Thus, measurement of treatment outcomes for behavioral health treatment is complex. In its 1990 study of alcohol treatment, the Institute of Medicine (IOM) approached treatment effectiveness with the following question: “Which kinds of individuals, with what kinds of alcohol problems, are likely to respond to what kinds of treatments by achieving what kinds of goals when delivered by which kinds of practitioners? ” (IOM, 1990b, p. 7). This perspective is appropriate for addressing treatment outcomes for drug and mental health problems as well.
We believe that the full evaluation of an organization has to combine knowledge of current outcomes with the standards-based information that lets you know and predict future outcomes.
Joint Commission on Accreditation of Healthcare Organizations
Public Workshop, April 18, 1996, Washington, DC
In approaching these multidimensional questions, the committee takes an evidence-based approach to defining outcomes in populations with substance abuse and mental health problems. This approach recognizes some fundamental commonalities across conditions and levels of severity. In the next section, the committee considers the general outcomes indicators of improvements in health and social functioning, such as improved quality of life and fewer family, employment, legal, and medical problems, that are applicable for each of these conditions. However, the committee also recognizes the value of condition-specific indicators, such as reduced alcohol and drug use and reductions in the symptoms of mental health problems (such as sleep and appetite disturbances as depressive symptoms, or hyperventilation for anxiety).
GENERAL AND SPECIFIC MEASURES OF OUTCOMES
It is important to identify the outcomes that treatment can be expected to bring about (see Table 7.1), including drawing a distinction between short-term and long-term goals. For example, the short-term goals of substance abuse treatment include detoxification, which is the reduction or elimination of alcohol use; the concomitant reduction of the signs, symptoms, and consequences of alcohol use; and a modification in attitude toward drinking behavior leading to a commit-
TABLE 7.1 Substance Abuse-Specific Outcomes Objectives by Level of Treatment
Level of Treatment
Reduction in physiological and emotional instability; lack of serious medical or psychiatric complications; integration and engagement in rehabilitation phase of treatment
Maintain changes initiated during detoxification; prevent relapse; develop and practice changes in alcohol and drug use behaviors showing improvements in personal health (medical and psychiatric); improved social functioning (family, friends, employment, and legal areas); engagement in ongoing aftercare
Aftercare and recovery
Maintain changes developed in rehabilitation; continued abstinence and/or levels of use that are nonproblematic; continued improvement in personal health and social functioning
ment to resolution of the problem in the future. The attainment of these goals is a major therapeutic achievement (see Appendix B).
The longer-term outcomes of treatment have to do with the maintenance of these goals over time, for example, whether improvements in drinking or drug use behavior, as well as social functioning, have been accomplished. It is also important to distinguish between outcomes measures that are broadly relevant to mental health-, alcohol-, and drug-related conditions and those that are specific to the different disorders and conditions, characteristics of patients with the disease, and particular levels of treatment. This chapter provides a general framework for the development of performance measures in the context of outcomes research. Appendix B and Appendix C provide more detailed discussions on substance abuse outcomes and mental health outcomes.
Many outcomes studies examining different facets of treatment have been conducted over the past 20 years. However, only recently have managed care organizations been the subject of such studies. The review by McLellan et al. (see Appendix B) indicates that those listed in Table 7.1 are the best supported.
New Approaches in Outcomes Measurement
Outcomes measurement can improve accountability and lead to an improved understanding of which practices are most effective in producing positive outcomes with different kinds of patients (Kane et al., 1995; Appendix C). Basic forms of outcomes assessment include patient satisfaction surveys, ratings by clinicians, and reviews of clinical charts. To standardize the assessment of mental
TABLE 7.2 General Outcome Measures for Substance Abuse and Mental Health Populations
health outcomes and improve the collection of outcome data, two new approaches show promise.
One approach is the development of clinical outcomes information systems, with most of the information coming directly from patients and clients through questionnaires or other forms of self-report (Ellwood, 1988; Kane et al., 1995). The information can easily be analyzed and assessed in standard formats. Another development in outcomes assessment is the design of modules or standardized sets of validated instruments that combine information from the patient and the clinician to assess the outcomes of treatment (Rost et al., 1992). The emphasis in the development of these modules has been the systematic approach to validating the instruments through field testing, primarily with routine clinical interventions (Smith et al., in press). (See Appendix C for a discussion of research priorities in the measurement of mental health outcomes.)
General Measures of Outcome in Behavioral Health
Many variables are common to substance abuse and mental health treatment (see Table 7.2). Some of the newly developed outcomes information management systems may make outcomes assessment more feasible to achieve (Kane et al., 1995), but the systems can be expensive to implement and to support and currently are more common in the private sector. Still, in behavioral health, it is important to identify and use measures that can be monitored fairly easily, such as functional outcomes of returning to work or school. Organizations that lack the capacity to implement a computerized outcomes information system can develop their own ways to build this basic information into their existing monitoring systems.
There is no evidence that a particular treatment setting (an inpatient, outpatient, or community-based setting, or a hospital) or even treatment modality (group vs. individual counseling) produces better outcomes across all patient groups (see Appendix B). The types of treatment settings (e.g., inpatient, outpatient, medical, and nonmedical settings) do not seem to be as important as the services received (DHHS and NIAAA, 1996; McLellan et al., 1996).
The committee notes that the evidence base across types of disorders and types of treatment approaches is very uneven. For example, for some treatment approaches, the research base is very strong (e.g., cognitive behavioral therapy for depression and methadone maintenance for heroin addiction), whereas for other treatments and types of mental and substance use disorders, research is lacking or equivocal (e.g., 28-day hospital programs for cocaine addiction and medication for childhood depression). Thus, Figure 7.1 provides an analytical framework for the development of an outcomes research base.
One application of this type of framework has been developed using the format of a psychiatric outcomes module (Rost et al., 1992). The module consists of
four different forms that are used to assess the types of care, outcomes of care, and patient characteristics associated with different outcomes and types of care. The forms include baseline assessments made by patient and clinician; a follow-up assessment; and a medical record review form. These tools can be used to inform administrative decisions about the quality of care and potentially can influence the decisions made by clinicians, patients, and payers of care (Smith et al., in press).
LINKS AMONG STRUCTURE, PROCESS, AND OUTCOMES
The committee uses a conceptual framework of structure, process, and outcome as characterized in the work of A. Donabedian (1992). The framework provides a multidimensional assessment of quality. From this perspective, outcome can be seen as emanating from important structural and process characteristics.
Within this framework, structure refers to the larger environment —the delivery system, the health care organization and system of delivery, interorganizational networks, communications, lines of authority, and the individual provider agency and staff. It is argued that organizational characteristics have differential effects on outcome (and access and utilization as well) and should be taken into consideration in outcomes research. These variables have not often been examined, especially in the context of managed care organizations. Studies by D'Aunno and Vaughn (1995) and others address the importance of examining the organizational characteristics within managed care organizations as part of outcomes monitoring.
Measuring process is an important part of assessing outcomes. It is necessary to determine which treatment characteristics and modalities have been applied and how they interact with patient characteristics to affect the outcomes. Once the indicators of the process of treatment are associated with outcomes, performance indicators can be more easily monitored by the field.
Within this framework an outcome is defined as the expected achievement. Donabedian (1992) describes outcomes measures as being integrative and reflecting the contributions of all those who provide care, including the contributions of patients to their own care. This includes examining the contributions of organizational and treatment characteristics. Currently, less is known about indicators of outcomes than about structural and process indicators.
Thus, examining outcomes is extremely important at this stage in quality improvement efforts since so little is known about the factors that influence treatment. The relative effectiveness of different approaches needs to be known, and routine monitoring of outcomes in a treatment setting or system of care can be a basis for continuous quality improvement within a treatment delivery organization (Berwick, 1989). A 1990 IOM study argued, “In a larger sense, outcome data provide an ethical justification for purveying treatment and a means of improving its effectiveness. From an ethical standpoint the provision of treatment in the absence of knowledge of results is a questionable procedure” (IOM, 1990b, p. 323).
TABLE 7.3 Performance Indicators Based on Outcomes Research
Performance indicators based on outcomes research require individual client-level data in addition to aggregated, agency-level data. Examples include the following:
PERFORMANCE INDICATORS AS OUTCOMES MEASURES
Table 7.3 lists potential performance indicators as outcomes measures that emerge from the outcomes literature.
In addition to recognizing the importance of these treatment characteristics, studies have demonstrated the efficacies of particular treatment methods, such as coping skills training, cognitive behavioral therapies, methadone treatments at particular dosage levels, naltrexone therapy (naltrexone is a narcotic antagonist), along with counseling or psychotherapy, disulfiram therapy (more commonly known as Antabuse), relapse prevention, community reinforcement approaches, motivational enhancement, and marital or family therapy (DHHS and NIAAA, 1996). Studies have also indicated that treatment intensity, length of stay, and therapist characteristics affect outcomes (Gotheil et al., in press) (see Appendix B).
This is a complex literature based on clinical trials rather than studies conducted in real-world settings. These studies are also often conducted on narrow sets of patient characteristics. Many strategies have not been evaluated across types and severities of substances and patient characteristics, particularly severity, comorbidity, age, ethnicity, and so forth. Clinical management information systems in managed care organizations can be designed to identify the practices that tend to be associated with better outcomes with particular types of patients (Kane et al., 1995).
Differences in outcomes have not been compared for carved-out programs versus treatment integrated with medical services, particularly among populations receiving care in the public and private sectors. Information on such differences is particularly crucial considering the trend toward the use of managed care organiza-
tions for public populations across states, as discussed in Chapter 2, Trends in Managed Care. The range of treatment strategies, such as the provision of brief interventions for their prevention role, as well as treatment, should be examined within managed care organizations.
It is better to have a 10-item scale that works reasonably well than a 450-item scale that works perfectly that nobody will use.
Center for Mental Health Services
Public Workshop, April 18, 1996, Washington, DC
EFFICACY AND EFFECTIVENESS
Donabedian (1990a, b) and others (DHHS and NIAAA, 1996; IOM, 1990c) distinguish between efficacy and effectiveness. Efficacy is the assessment of treatment characteristics under ideal clinical conditions. The committee is concerned here with effectiveness as measured in the real world across different types of organizations. Effectiveness is improvement in health that is achieved, or that can be expected to be achieved, under the ordinary circumstances of everyday practice. In defining and assessing quality, effectiveness can be more precisely specified as the degree to which the care whose quality is to be assessed attains the level of health improvement that studies of efficacy have established (Donabedian, 1990a, b; Lohr, 1988).
A goal of integration between efficacy and effectiveness research is to conduct basic clinical research on structure, process, and outcomes to provide information on the characteristics associated with particular outcomes for particular client groups. Then, ongoing performance outcomes linked research can assess how well programs are doing, given what should be expected in terms of treatment characteristics and case mix.
For a variety of practical and ethical reasons, however, it is not always possible to conduct highly controlled research to evaluate the impacts of various structural and process components of treatment on outcomes. For this reason, it is desirable for quality assessment systems to begin to incorporate direct measures of client outcomes, in addition to treatment characteristics and client case mix.
OUTCOMES AND QUALITY IMPROVEMENT
The treatment field has been internally and externally motivated to assess
performance. Data such as provider ratings and consumer satisfaction report cards are most often used to assess performance. However, some managed care organizations (such as Kaiser Permanente and Group Health Cooperative of Puget Sound) are initiating the use of routine outcomes monitoring systems; for example, they are establishing internal research divisions that perform research on their own internal standards and outcomes or contract with outside entities to conduct evaluations. These are important efforts; of concern is standardization and the comparability of measures and methodologies and the development of performance indicators related to the main predictors of good outcomes.
We encourage programs to use structured instruments. If they don't use structured instruments, we encourage them not to use dichotomous measures, like were you sober or were you not. If they have a measure like that, they don't have anything they can do in terms of program planning.
Rehabilitation Accreditation Commission
Public Workshop, May 17, 1996, Irvine, CA
The IOM and Agency for Health Care Policy Research approach to practice guidelines is one in which guidelines based on outcomes research provide the basis of performance measures and accreditation when possible (IOM, 1990c). In the behavioral health field, relatively few practice guidelines have been developed, and they tend not to have a direct relationship to the practice of accreditation. As accreditation organizations begin to focus more on outcomes research, the opportunities and need for further guideline development will increase.
The difficulties in conducting outcomes research within behavioral health care programs are well recognized. Such research is costly and imposes a burden on the programs; it also requires expertise not always available in smaller organizations. To make these efforts truly worthwhile, regardless of whether the research is done internally or externally, what is called for is the use of standardized instruments (with good reliability and validity) by all providers to assess outcomes, standardized instruments measuring content and process of treatment, instruments and methods that are not burdensome (require short response times) or costly, and instruments that are easily interpretable.
CRITERIA FOR EVALUATING OUTCOMES MEASURES
Many of the current indicators assess client satisfaction, which is a reflection of what is easy to measure given the constraints and costs of measuring outcomes.
Although this is an important area, additional emphasis should be placed on improvements in symptoms and functioning. To move the field forward and to begin to develop a process that injects data into a continually improving system, instruments and methods need to be standardized. Comparable, reliable, and valid instruments need to be used, and a system that does not put unreasonable burdens on programs needs to be developed. The existing indicators do not live up to these ideals.
Potential Data Sources for Outcomes Research
The only data source that can truly assess outcomes is client-level outcome information. This usually involves sample surveys of clients, often combined with other indicators assessing the validity of self-reports, such as breath analysis and urinalysis in substance abuse studies. This is the highest standard; however, its high cost and the expertise that is required to carry out the work are limitations to its use (see Appendix B).
Claims data are useful for examining utilization of care, appropriateness of treatment, retreatment rates, costs of treatment, and the cost offsets of care (e.g., money spent on mental health or substance abuse treatment that reduces the costs of treatment in primary care or other specialty care). The strength of claims data is their availability and low cost. Their limitations include a lack of direct measurement of outcomes. That is, claims data include diagnoses, clinical procedures, and numbers of visits, but do not include clinical information such as symptom reductions or behavioral changes experienced by the client or patient.
We need to provide outcome information to purchasers, especially medical cost offset findings, that clarifies it is in the best financial interests of their companies to care about quality because in the long run it will lower costs.
Institute for Behavioral Healthcare
Public Workshop, May 17, 1996, Irvine, CA
Admission or discharge forms (and medical records) can be used to assess the severity of a behavioral problem at intake and the provider 's assessment of the patient's status at discharge. They also should be available for all programs. Their limitations include a lack of standardization across agencies and among staff within agencies with regard to the data collected and the time frames and definitions that are used. They also do not provide data following treatment discharge; thus, information on the posttreatment retention of the improvement is not available.
Retreatment rates can often be determined through analysis of claims data or
admission data collected by managed care organizations. These rates do not reflect outcomes well because they are confounded, for example, by issues related to access and by the dropping of clients from insurance plans.
Accountability for Outcomes
The outcomes field would be improved significantly by developing standardized data that are publicly available. To accomplish this, it is necessary to move away from the focus on client satisfaction-based indicators of outcome to clinically-based indicators of outcome. This is not an easy transition to make. Client satisfaction indicators are easier to develop and to collect than are clinical indicators of outcomes, for reasons that are discussed throughout this chapter. Ideally, the steps should include clinical efficacy trials and then health services effectiveness trials, followed by comparisons of different practice settings, case mix adjustments, and so forth, for the assessment of outcomes in managed care settings. The assessments of outcomes could then become public information, through report cards or through other means.
As the treatment field moves in the direction of accountability for outcomes within programs, several difficult issues need to be addressed. The first has to do with external versus internal monitoring of treatment outcomes. External monitoring brings with it a higher potential for standardization across programs, the input of those with expertise in methodologies, and the credibility that comes from independent evaluations. Internal monitoring lacks those particular strengths, but offers the ability to collect the precise kinds of data most useful to managed care organizations in improving their particular programs.
All of our purchasers, including government and private purchasers, are looking at setting up outcomes and performance measures. Everybody has their own idea about what's a good outcome. Unless there can be some sort of consensus about what that means, I think increasingly we're going to see plans that are going to be ineffective at measuring outcomes.
William R. Mercer, Inc.
Public Workshop, May 17, 1996, Irvine, CA
A second issue has to do with finding valid measures for outcomes and requiring their collection by programs. Ongoing collaboration between outcomes researchers and clinicians can promote the further development of such measures
by combining what is learned from outcomes research and clinical consensus to select those particular indicators that predict good outcomes. Until valid and reliable measures are available, it may be appropriate to accept proxies—indirect measures—of outcomes.
The development of a system of programs that perform their own follow-ups with standardized instruments would complement and strengthen outcomes research. Programs could gauge themselves against the findings related to outcomes and could use the information to develop a means of continual improvement for their systems. Using the standardized information generated by these programs, researchers could isolate the treatment elements that are associated with good treatment outcomes and those elements that become the indicators for improvements in quality.
This can result in the development of a real partnership between research and practice, one that is geared to moving the outcomes measurement field forward and also to generating useful information for program improvement. Anonymity would be provided for programs and organizations in outcomes studies and would provide programs and consumers with applied evaluation information. This information also would be valuable for assessing the cost-effectiveness of various treatment strategies.
It is only by speaking with a single voice, and, more importantly, by talking with data rather than with promises, that the cause of quality will be advanced in the ongoing policy debate about behavioral health care and the role that it is going to play in the health care delivery system in this country.
American Managed Behavioral Healthcare Association
Public Workshop, April 18, 1996, Washington, DC
Outcomes research does not yet explicitly identify those performance indicators associated with good outcomes across patient characteristics and types of programs. Furthermore, few outcomes studies are under way that will provide this specific information, especially across types of managed care organizations and patient characteristics, in the near future. At the same time, existing performance indicators do not yet attempt to specify particular treatment characteristics (orga-
nizational and clinical) that can be argued to represent a consensus of clinical judgment with regard to their relation to outcomes.
In the committee's view, outcomes research is vitally important to improving the evidence base for treatment effectiveness. However, much needs to be done to standardize outcomes information and to link findings from outcomes research with the development of practice guidelines, performance measures, and accreditation approaches.
Berwick DM, 1989. Health services research and quality of care: Assignments for the 1990s. Medical Care 27(8):763-71.
Burnam MA. In press. Measuring routcomes of care for substance abuse and mental disorders . In: Steinwachs DM, Flynn L, eds. New Directions for Mental Health Services: Using Outcomes to Improve Care. San Francisco: Jossey-Bass.
D'Aunno T, Vaughn TE. 1995. An organizational analysis of service patterns in outpatient drug abuse treatment units. Journal of Substance Abuse 7(1):27-42.
DHHS (U.S. Department of Health and Human Services) and NIAAA (National Institute on Alcoholism and Alcohol Abuse). 1996. Final Report: Panel on Effectiveness and Outcome. Washington, DC: National Advisory Council on Alcohol Abuse and Alcoholism, August.
Donabedian A. 1990a. The seven pillars of quality. Archives of Pathology and Laboratory Medicine 114(11):1115-1118.
Donabedian A. 1990b. Quality and cost: Choices and responsibilities. Journal of Occupational Medicine 32(12):1167-1172.
Donabedian A. 1992. The role of outcomes in quality assessment and assurance. Quality Review Bulletin 18(11):356-360.
Ellwood PM. 1988. Outcomes management: A technology of patient experience. The New England Journal of Medicine 318(23):1549-1556.
Gotheil E, McLellan AT, Druley KA. In press. Length of stay, patient severity and treatment outcome: An example from the field of alcoholism. Journal of Studies on Alcohol.
Hubbard RL, Marsden ME, Rachal JV, Harwood HJ, Cavanaugh ER, Ginzburg HM. 1989. Drug Abuse Treatment: A National Study of Treatment Effectiveness. Chapel Hill, NC: University of North Carolina Press.
IOM (Institute of Medicine). 1990a. Treating Drug Problems. Vol. 1. Washington, DC: National Academy Press.
IOM. 1990b. Broadening the Base of Treatment for Alcohol Problems. Washington, DC: National Academy Press.
IOM. 1990c. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academy Press.
Kane RL, Bartlett J, Potthoff S. 1995. Building an empirically based outcomes information system for managed mental health care. Psychiatric Services 46(5):459-462.
Lohr KN. 1988. Outcome measurement: Concepts and questions. Inquiry 25:37-50.
Marlatt GA. 1983. The controlled-drinking controversy: A commentary. American Psychologist 38:1097-1110.
McGlynn EA, Norquist GS, Wells KB, Sullivan G, Liberman R. 1988. Quality of Care research in mental health: Responding to the challenge . Inquiry 25:157-170.
McLellan AT, Weisner C. 1996. Achieving the public health potential of substance abuse treatment: Implications for patient referral, treatment “matching” and outcome evaluation . In: Bickel W, DeGrandpre R, eds. Drug Policy and Human Nature. Philadelphia: Wilkins and Wilkins.
McLellan AT, Woody GE, Metzger DS, McKay J, Durell J, Alterman AI, O'Brien CP. 1996. Evaluating the effectiveness of addictions treatments: Reasonable expectations, appropriate comparisons. The Milbank Quarterly 74(1):51-85.
Miller WR, Westerberg VS, Waldron HB. 1995. Evaluating alcohol problems. In: Hester RK, Miller WR, eds. Handbood of Alcoholism Treatment Approaches: Effective Alternatives, Second Edition. New York, NY: Allyn and Bacon. Pp. 61-88.
Polich JM, Armor DJ, Braiker HB. 1981. The Course of Alcoholism: Four Years After Treatment. New York: John Wiley and Sons.
Roizen R. 1987. The great controlled drinking controversy. In: Galanter M, ed. Recent Developments in Alcoholism. New York: Plenum. Pp. 245-279.
Rost KM, Smith GR, Burnam MA, Burns BJ. 1992. Measuring the outcomes of care for mental health problems: The case of depressive disorders. Medical Care 31:189-200.
Simpson DD, Sells SB, eds. 1990. Opioid Addiction and Treatment: A 12-Year Follow-up. Malabar, FL: Robert E. Krieger Publishing Company.
Smith GR, Rost KM, Fischer EP, Burnam MA, Burns BJ. In press. Assessing the effectiveness of mental health care in routine clinical practice: Characteristics, development, and uses of patient outcomes modules. Evaluation and the Health Professions.
Tims FM, Fletcher BW, Hubbard RL. 1991. Treatment outcomes for drug abuse clients. In: Pickens RW, Leukefeld CG, Schuster R, eds. Improving Drug Abuse Treatment. Washington, DC: U.S. Government Printing Office. Pp. 93-113.