Cover Image


View/Hide Left Panel
Click for next page ( 153

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 152
18 Using Patient Reports of Outcomes to Assess Effectiveness of Medical Care Paul D. Cleary In this chapter, I provide a brief overview of some of our current and recent work on the development and evaluation of outcomes measures based on patient reports. First, I make some general comments about the way we think about quality and effectiveness and discuss the range of outcomes that we think are important to consider in these types of studies. I also discuss a study recently conducted in six hospitals in California and Boston, includ- ing some results from the use of both generic and disease-specific measures in that study. I illustrate several of my points using data from patients with total hip replacements and conclude with some observations that bear on the strengths and weaknesses of these measures for assessing effectiveness and outcomes in health care. LINKING PROCESS AND OUTCOME Donabedian has described three approaches to the assessment of the quality of medical care: observation of structure, of process, and of outcomes (1~. Most programs for evaluating quality focus primarily on process, and we tend to use terms such as "consensus," "norms," "standards," "criteria," and "appropriateness" when we describe the ways in which we evaluate process. These are all part of our lexicon and are an integral part of the way we think about quality. Donabedian pointed out that we typically evaluate quality on the basis of observations of process, but he also asserted that judgments of quality tend to rest on what is known about the relationships between process and outcome. I would emphasize that point and argue that evaluations of process and outcomes are inextricably linked. Unless we know what the outcome of a 152

OCR for page 152
DEVELOPMENT AND USE OF OUTCOMES MEASURES 153 particular process is, we cannot determine whether it represents quality care. The purpose of measuring outcomes is to help establish the relationships between process and outcomes. Little is learned by studying variations in outcomes by themselves, and little is gained by developing better measure- ment tools in isolation. These types of research activities are undertaken to help develop a practical, valid model of the linkages between process and outcomes so that we can improve quality of care. EARLY STUDIES OF OUTCOME Outcomes are so widely considered to be the ultimate indicator of quality that it is surprising how infrequently we analyze them carefully. In the 1830s, a physician named Pierre-Charles-Alexandre Louis started a group in Paris that discussed the use of statistics to examine patterns of medical care. In 1838, a physician from that group named George Norris returned to the United States and looked at 55 cases in which an amputation had been performed. Norris found that 21 of the patients who had had an amputation had died. This was an important finding, and it challenged many people's assumptions about the dangers associated with amputation. In subsequent work, Norris compared how surgery outcomes at the Pennsylvania Hospital compared with those of hospitals in other cities and counties. It is interesting that although we think of the publication of mortality statistics as a very recent phenomenon, they have been published on a hospital-specific basis for 150 years! It is disappointing that we have not improved our methods for assessing the relationships among case mix, process of care, and outcomes, given the long history of work in this area and the importance of the issues. THE NEED FOR DISEASE-SPECIFIC MEASURES One of the central issues in outcomes assessment concerns the range of outcomes that should be assessed and whether it is better to use disease- specific or general measures or both. I think that we should definitely include both measures in outcome batteries. A number of researchers have argued that assessing disease-specific outcomes is not necessary, that if one measures generic outcomes, specific measures will not help explain addi- tional variance. I disagree strongly with that position. If we want to be able to detect differences in outcomes that are related to the process of care, it is usually necessary to include measures of outcomes specific to the condition studied and, in some cases, specific to the process of care being examined.

OCR for page 152
54 EFFECTIVENESS AND OUTCOMES IN HEALTlI CARE The domain of general outcomes that we think are important to study includes general health perceptions, disability, activities of daily living, role performance, well-being, fatigue, cognitive functioning, and satisfaction with care. It is not always necessary to measure all of these outcomes. Depending on the application, one may want to measure only one or two dimensions. It is often useful to have measures of all of the areas mentioned above, but to develop a practical system that can be used for policy-related relevant research and for quality assessment and assurance programs, it may be necessary to be more selective with respect to the measures used. SCOPE OF THE STUDY The specific study I describe here is an investigation of variations in case mix, patterns of care, and outcomes at six hospitals. The investigators besides myself were Barbara McNeil, Sheldon Greenfield, Albert Mulley, Steven Pauker, Steven Schroeder, and Lewis Wexler. The hospitals were not-for-profit, university-affiliated teaching hospitals. Three of the hospitals are in California and three in Boston. The sample was about 3,000 patients receiving treatment for one of six medical or surgical conditions: acute myocardial infarction (AMI); rule-out AMIs; total hip replacements; cholecystectomies; coronary artery bypass graft surgery (CABG); and transurethral prostatectomy (TURP). During a one-year enrollment period, all eligible patients were sent a letter after discharge explaining the purposes of the research and encourag- ing them to take part in the study. For each patient who agreed to participate, we obtained from the medical records data on disease severity, comorbid conditions, and the process of care during the index hospitalization. Information about sociodemographic characteristics, health-related quality of life before and after hospitalization, perceived improvement in health status, health care utilization, and satisfaction with care were collected using a self-ad- ministered questionnaire mailed after discharge (2~. The timing of the fol- low-up questionnaire was determined by a panel of experts and varied, depending on the condition, from 3 to 12 months. Patients who did not return the original questionnaire within two whelps were sent a second ques- tionnaire, which was followed up with a telephone call reminding them to return the questionnaire. Patients who still did not return a completed questionnaire were interviewed over the telephone, when possible. A medical record reviewer abstracted information about sociodemographic characteristics, indicators of severity, comorbid conditions, surgical procedure, occurrence of in-hospital complications, and use of services in the hospital (for example, laboratory tests, days in the intensive-care unit, and so on). To record information on comorbid conditions, we used the approach developed

OCR for page 152
DEVELOPMENT AND USE OF OUTCOMES MEASURES 155 by Greenfield and colleagues (3,4~. Another measure used was the physical status classification of the American Society of Anesthesiologists (5~. Us- ing the medical record, we coded disease-specific indicators of severity and generic, as well as disease-specific, complications. To synthesize the infor- mation on complications, a panel of experts selected a subset of complications that they considered to be "serious." For these complications, we created an index representing a count of the number of these complications experi- enced by the patient. PATIENT QUESTIONNAIRE The outcome questionnaire asked about perceived general health, number of days disabled, use of health services, symptoms related to the hip replacement, current social activities, activities of daily living, well-being, satisfaction with medical care and health, whether patients thought the operation made them feel better, whether their health was better or worse than expected, whether they felt "back to normal," employment, and role functioning, as well as indicators of socioeconomic status such as education and income. The questionnaires also had questions about condition-specific outcomes. For example, the questionnaire sent to total hip replacement patients contained questions about the amount of pain experienced doing a range of activities, degree of limping, and use of walking supports. Finally, the questionnaire also asked about daily activities, limping, use of walking supports, well- being, employment, and role functioning in the month preceding surgery. The measures of social activities, functioning, and well-being were adapted from the Functional Status Questionnaire (6~. The pain scale was also derived from a measure used by Jette and colleagues. The questions about use of walking supports and limping were developed for this study. The questionnaires were designed to be easy to read and answer, and took approxi- mately 30 minutes to complete. The psychometric properties of the generic components of this scale, when used with different groups of surgical patients, have been described elsewhere (2~. We collected billing data primarily to look at process. We extracted comparable computerized information on 101 resource elements or process of care variables at each of the hospitals. I discuss primarily data from the patient questionnaire, but I want to emphasize again that those data are part of a system of quality assessment. Although many studies have used patient questionnaires to assess out- comes, this study was unusual in a couple of important ways. The first is that these patient questionnaires were administered a substantial period after the hospitalization. The second is that we asked about these variables before as well as after hospitalization. What we wanted to know about was not just outcomes or simply variations in outcomes, but how postdischarge

OCR for page 152
156 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE health status differed from preadmission status and how those changes re- lated to differences in the process of care. I would like to emphasize that it is possible, in some cases, to get very important information with a few simple questions. Although a single ques- tion about general health status may appear to have limited validity to some clinicians, empirical studies have shown that asking such a question can elicit information related to clinical measures of health status. Other types of issues are also easy to assess. For example, one question on the health interview survey was, "During the past month, how many days did illness or injury keep you in bed all or most of the day?" National data on how people respond to this question are available, as are data from a variety of studies in hospitals, clinics, and communities. A simple question of this kind can be extremely informative. When we talked to orthopedic surgeons and internists, they invariably said that an important outcome for hip replacement patients is pain. They operate for pain, they try to relieve pain, they try to get people back to functioning without pain-and so we included a pain scale on our questionnaire. It is important to note that the scales used in this study are in most cases closely related to existing scales: it is not the content of the scales, but rather the way they are applied, that is different. Orthopedic surgeons also usually say that one of their primary goals is to enable people to walk without support again. To assess this outcome, we included simple questions: "What type of walking supports do you use now? What kind of limp do you have now?" Again, this is not a long battery, it is not complicated, it is very easy to answer, and it is very easy to administer. Surgeons often do not know what proportion of their patients are still limping a year after surgery or what proportion of their patients are using walking supports, so they often find the responses to such questions . ~ . very Informative. To assess psychological well-being, we used five items from the Functional Status Questionnaire (6) that are the same as those used in the Medical Outcomes Study. They include questions such as "Have you been a very nervous person?" and "Have you felt calm and peaceful?" To measure role functioning, we ask a series of questions about how the patient is doing at work or at home. In the six hospitals study, we asked about patient satisfaction using tradi- tional questions such as "How satisfied were you with your hospital stay in general?" I am now conducting a different study, the Picker/ Commonwealth Study of Patient-Centered Care, in collaboration with Tom Delbanco at Beth Israel Hospital in Boston, Tom Moloney at the Commonwealth Fund, and a number of other colleagues at Harvard and the Commonwealth Fund. We are collecting information from a national probability sample of about 6,000 patients nationally and 2,000 of their caregivers and asking them very

OCR for page 152
DEVELOPMENT AND USE OF OUTCOMES MEASURES 157 specific questions about the process of care that we think one would want to know about when evaluating quality and effectiveness. We do not usually think of patient satisfaction as a measure of outcome, but I think after hospitalization we would like one of the outcomes to be an informed, involved, cooperative patient. Thus, in the Picker/Commonwealth Study, we ask a series of questions such as: "Were you involved in the decisions about your care as much as you wanted?" "Were the important side effects of the medicines that you were getting explained to you in a way you could understand?" and so on. We have a sample of 62 hospitals nationally, and I think we will be able to make some very interesting observations about the differences among hospitals. That study is almost completed and the results should be published this fall. FINDINGS One of the first things we wanted to know about our outcomes study was whether it is feasible to distribute a questionnaire like ours to patients from multiple institutions. We found that it was a very practical way of collect- ing information. The questionnaire we used was 30 minutes long; we probably could make it much shorter. Patient acceptance was high. In most surveys there tends to be a reluctance to participate, but in this study there was a great deal of interest in the study. Rather than feeling burdened by the questionnaire, many patients reported that they were pleased that the hospital was checking on how they were doing. We got a response rate of approxi- mately 80 percent. About 10 percent of patients said they do not want to participate in a research study, and about 90 percent of the remaining patients returned a usable questionnaire. It is important that measures be reliable and valid. Our scales were very reliable, with coefficients ranging from 0.64 to 0.92. For the more established scales, the reliabilities were quite high. Data on the correlations among measures and the correlations with other health measures indicate that ours are valid measures of health status. One common concern is that these scales may reflect, to a great extent, differences in general psychological well-being: that is, if patients are depressed, they will say they are doing poorly; if they are feeling good, they will say they are doing well. We did not find that to be the case in our study, probably because we focus on questions that are as concrete as possible. Questions about specific activities, such as limping and the use of walking supports, are less likely to be confounded. Another important feature of our health status measures is their responsiveness to changes in health status. For most of the conditions we studied, there is a ceiling effect, that is, everyone is doing well and the observer cannot see any difference. For total hip replacement patients, however, there was a

OCR for page 152
158 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE dramatic improvement in functioning. One could see that the changes are similar to what a clinician would predict after hospitalization, patients' basic activities are largely back to normal. An important question is whether it is necessary to measure different dimensions separately or whether it is possible to use a combined measure. The data demonstrate why I think it is better to measure components separately. If one measures intermediate activities separately from basic activities, one can see a very different pattern emerge. Patients who have had a total hip replacement still show quite dramatic improvements, but there is a slightly different picture for patients who have had CABG surgery. These patients show a very strong and statistically significant improvement in functioning on the intermediate level that we would not have picked up with a basic . . . activities scale. The data on work performance also show a different pattern. With total hip replacement patients there is a dramatic improvement in performance. That contrasts with the perplexing but fairly consistent clinical finding that CABG patients do have impaired work performance and do not return to work as much as one would expect them to. Among the AMI and rule-out AMI patients, postdischarge functioning is worse than predischarge. Again, a separate scale picks up an important phenomenon that I think would have been obscured in a combined measure. The data on psychological well-being indicate that most patients are doing pretty well, and everyone shows slight improvement. It is difficult to describe the relative improvement across these scales. I have taken each scale and calculated an improvement score, which is basically how they are doing before hospitalization minus how they are doing later, divided by the standard deviation of the change. Using this statistic as a gauge of responsiveness, we find that the question about limping gives us the best sense of how people are doing. Use of walking supports is not quite as good. As we would expect, there are big improvements in intermediate and basic activities: among hip replacement patients the basic and intermediate scores show a .78 correction. This provides more evidence that in the future we might be able to shorten our questionnaire. The mental health scores did not show much change, and I frequently hear an argument that one should not include social or psychological components that are not directly related to the condition being studied. I would make a plea for not discounting such measures so quickly. First of all, mental health is a very, very important component of case mix. Another reason is that it may be very important for interpreting good and bad outcomes. For example, we are now engaged in analyzing older and younger patients, and we have found that mental health status is related to perceived health status in both groups and that it may be critical in determining differences.

OCR for page 152
DEVELOPMENT AND USE OF OUTCOMES MEASURES CONCLUSIONS 159 The first conclusion from these data is that we have adequate measures for most constructs. We have available a series of comprehensive batteries or instruments. They are brief and can be made briefer. They are acceptable to patients. They meet or exceed our normal standards for reliability. They are very valid, and they are responsive to changes in health status. Outcomes assessment should be an integral part of quality assurance activities because it is not possible to assess fully the quality of processes of care without data on associated outcomes. There has been a substantial amount of research on how to assess case mix, and there are many systems for monitoring the process of care. A fair amount is now known about variations in certain outcomes, such as mortality, and I think the main factor that limits us at this point is a lack of understanding about the linkages among case mix, process of care, and outcomes. I would argue that if we understood these linkages better, cost containment and regulation would again become an administrative inconvenience rather than a threat to the practice of medicine as we know it today, a frequently expressed concern. We have the tools; we have the creativity; and we have the will to address these issues. It is up to us to seize the day. REFERENCES 1. Donabedian, A. Explorations in Quality Assessment and Monitoring, Volume I. The Definition of Quality and Approaches to Its Assessment. Ann Arbor, MI: Health Administration Press, 1980. 2. Cleary, P.D., Greenfield, S., and McNeil, B.J. After Surgery. Controlled Clinical Trials, in press. Assessing Quality of Life 3. Greenfield, S., Blanco, D.M., Elashoff, R.M., et al. Patterns of Care Related to Age of Breast Cancer Patients. Journal of the American Medical Association 257:2766-2770, 1987. 4. Greenfield, S., Aronow, H.U., Elashoff, R.M., et al. Flaws In Mortality Data: The Hazards of Ignoring Comorbid Disease. Journal of the American Medical As- sociation 260:2253-2255, 1988. 5. Owens, W.D., Felts, J.A., and Spitznagel, E.L., Jr. ASA Physical Status Classifications: A Study of Consistency of Ratings. Anesthesiology 49:239-243, 1978. 6. Jette, A.M., Davies, A.R., Cleary, P.D., et al. The Functional Status Questionnaire: Reliability and Validity When Used in Primary Care. Journal of General Internal Medicine 1:143-149, 1986.