|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 179
~ 1
Applying Effectiveness and Outcomes
Research to Clinical Practice
Albert G. Mulley
Wide variations in medical practices for seemingly similar patients have
called into question the adequacy of the knowledge base that supports clinical
decision making (1~. Such variations have also fueled concerns about both
the cost and the quality of medical care. The research community has
responded with proposals for a new focus in clinical research on outcomes
of patient care, and the National Center for Health Services Research has
recently announced a new program to sponsor such research (2~. To meet
its responsibility for ensuring the quality of care provided to Medicare ben-
eficiaries, the Health Care Financing Administration launched a program
within the Department of Health and Human Services. This Effectiveness
Initiative will systematically gather information to improve our understanding
of the relative effectiveness of alternative therapeutic approaches to conditions
that commonly afflict Medicare beneficiaries (3~.
My purpose is to examine the methodological issues associated with the
application of effectiveness and outcomes research to clinical practice. What
are these issues? The answer depends on how one defines effectiveness and
outcomes research and how one distinguishes them from the clinical research
that has informed, however well or poorly, clinical practice until now.
WHAT IS DIFFERENT ABOUT EFFECTIVENESS AND
OUTCOMES RESEARCH?
First, the Effectiveness Initiative and outcomes research are motivated by
concerns about the quality and cost of medical care. New knowledge alone
may not be enough to declare success. The current wave of enthusiasm and
support will be sustained only if the initiative has a demonstrable impact on
quality, cost, or both.
179
OCR for page 180
180
EFFECTIVENESS AND OUTCOMES IN HEALTH CARE
Second, the initiative insists on measurement of outcomes that are im-
portant to patients. Five-year survival is too coarse a measure. Physiologi-
cal measures are fine, but they are often irrelevant. However, detailed
measures of health states and of subjective responses to those states will be
just as irrelevant unless they can be communicated to the persons responsible
for clinical and policy decisions.
Third, effectiveness and outcomes research must recognize clinical practice
as a source of information in the production of new knowledge rather than
as a passive and not always attentive consumer of knowledge produced by a
separate enterprise called clinical research. This distinction between more
traditional research and the new initiative is evident in the use of claims
data and other administrative data bases to capture more of the collective
experience of clinical practice (3-5~.
Each of these premises about how outcomes and effectiveness research
are different from traditional research raises a different set of methodological
issues. The first set of issues is related to the need for dissemination of
information. How do we get new information about effectiveness and out-
comes to the point where it can have a positive impact on quality or a
restraining effect on inflation? It is no accident that each of the bills
supporting outcomes research also has provisions for practice guidelines.
The second premise, that the new research is different because of a focus
on patient-oriented outcomes, complicates matters. Results of this new
research must include the subjective responses of patients that determine
their quality of life, as well as the trade-offs between quality and quantity
that are acceptable to them. Communication of such subjective value judgments
involves a set of methodological issues that must be addressed if we are to
preserve the responsiveness of health care to the wants and needs of individual
patients.
The third premise raises pragmatic issues about the possible integration
of research and practice, not to mention epistemological questions. Which
elements of clinical investigation are essential for valid inferences to be
drawn about effectiveness? When can collected experience in clinical practice
be an acceptable, or even preferable, source of information? Methodologists
will recognize this as a variant of the tension between the "internal validity"
of a clinical study and the limits that requirements for internal validity
place on the "external validity," or generalizability, of a study finding (6~.
A CLINICAL EXAMPLE: OUTCOME
PROBABILITIES IN DECISION MAKING
These methodological issues will be less abstract in the context of a
clinical example. A 72-year-old man, married and sexually active, has
increasing symptoms of benign prostatic hyperplasia. He gets up twice
OCR for page 181
APPLICATION TO CLINICAL PRACTICE
181
each night to void, and during the day he voids frequently, with a sensation
of urgency. The patient experiences a clinical process, and the result is a
health outcome that we can define simply or in great detail, if we include
the physical, psychological, and social dimensions.
Any simple model that goes from patient, through process, to outcome is
too deterministic. There is no single, discrete clinical process that is uniquely
suited to a particular patient. The path followed through clinical practice is
the result of a decision or series of decisions (in this case, whether to
proceed with prostatectomy), with the outcomes contingent on each deci-
sion being more or less uncertain (see Figure 1~. This may seem too obvi-
ous to belabor so, but the critical link between "outcomes research" and
effectiveness is the ability to make valid comparisons between outcomes
produced by the alternative pathways. This may be forgotten by the outcomes
researcher assembling a cohort from claims data procedure codes or the
guideline developer whose frame of reference is a particular procedure rather
than a particular condition. The irreducible uncertainty, or stochastic element
of medicine in any individual case, may be forgotten by the quality assurance
reviewer who equates a bad outcome with bad care.
It is worth taking a closer look at the decision-making process (Figure 2~.
The patient faces, with the help of his or her physician, a choice between
two alternative treatment strategies. The first alternative is a bit risky
because the eventual outcome is uncertain. Although there is a chance that
it will produce the most valued outcome (in this case, relief from symptoms),
there is also a chance that it will produce the outcome that is least valued
(operative death). An intermediate outcome is also possible (for example,
impotence). The second alternative is less risky: the only possible outcomes
are the most valued and an intermediate health state that happens to be the
Q
r 1 ~ Clinical
~L~ Decision
Clinical
A,/..///,'/
Outcome
Practice
FIGURE 1 A health outcome can be viewed as the product of a clinical process that
begins with a decision or series of decisions about the interventionist most likely to
meet a particular patient's health care needs and wants.
OCR for page 182
182
EFFECTIVENESS AND OUTCOMES IN HEALTH CARE
patient's current health state. A more elaborate model of the prostatectomy
decision has been developed, but this simpler model suffices to illustrate
the process (7~.
What is needed to make this choice? First, the patient and physician
need to know how likely each of the outcomes is. These probabilities can
be depicted as pie diagrams, as seen in Figure 3, where our hypothetical 72-
year-old is referred to as Patient A. Alternative 1 has a 90 percent chance
of producing the most valued outcome, a 1 percent chance of catastrophe,
such as operative death, and a 9 percent chance of a bad but not fatal result,
such as incontinence or impotence. Alternative 2, which looked so good
without these numbers, now looks less promising: there is only a 10 percent
Q
l
,~
1 ~
A
~1
Alternative ~ <-
Alternative 2 :
FIGURE 2 An Abstract Representation of a Simple Clinical Decision. The cross-
hatched triangle superimposed on the patient represents the health state that has
prompted medical care and the current decision. The square node represents a
choice between Alternatives 1 and 2. Alternative 1 offers a chance, indicated by the
round node, of dramatic improvement (represented by the white triangle) but with a
risk of death (the black triangle) or a serious complication (the diagonally hatched
triangle). Alternative 2 offers a chance of improvement with the only other outcome
the baseline symptom state.
OCR for page 183
APPLICATION TO CLINICAL PRACTICE
5% 10%
Alternative 1 ( ~( (
Alternative 2
183
Or
r
Patient A Patient B Patient C
)1oo%
FIGURE 3 Outcome Probabilities. Probabilities of each of the outcomes in Figure
2 for both alternatives for three hypothetical patients (including the 72-year-old man
cited In the text, here designated patient A).
chance of improvement the odds are 9 to 1 that the health state that was
bad enough to bring the patient to the doctor will persist. It can be said that
knowledge is power because it confers the capacity to predict. Accurate
estimation of outcome probabilities, as represented in these simple pie dia-
grams, captures the essence of professional knowledge related to the practice
of medicine.
SOURCES OF PROBABILITIES AND
A ROLE FOR OUTCOMES RESEARCH
Where does this knowledge come from? The most obvious source o
probabilities is the experience of previous patients. This constitutes the
"clinical experience" of the provider that is so important to "clinical judgment."
There are problems, however, with this source of information. First, there
are problems with the way clinicians characterize individual patients. Sec-
ond, clinical practice is not standardized. Interventions are not carefully
defined and uniformly applied. Third, there is no routine mechanism to
define outcomes with the appropriate level of detail or to aggregate and
organize the information that could be derived from collective clinical experience.
Without such systematic aggregation and analysis, the cognitive heuristics
that we all use routinely may mislead the clinician's unaided, intuitive probability
estimate (8~.
Recognizing these problems, the profession relies heavily on published
f
OCR for page 184
184
EFFECTIVENESS AND OUTCOMES IN HEALTH CARE
clinical research when it is available. The randomized trial is the standard
against which other clinical studies are measured. Information about patients
entering the trial is systematically collected. The group is made homogeneous
by applying exclusion and inclusion criteria. The alternative interventions
are carefully defined and their elements carefully segregated. Outcomes, at
least one or two of the more objective outcomes, are carefully catalogued.
The scientific requirements of research designed to determine the effectiveness
of one intervention relative to another, which is nothing more than the
relative outcome probabilities, include: similarity of the initial states; the
integrity of the interventions; and similarity of detection or measurement of
outcomes.
Unfortunately, clinical research that meets these requirements is the exception
rather than the rule. In the case of benign prostatic hypertrophy there are
no randomized trials. Studies published in English describe outcome prob-
abilities for very few men with symptoms who elected not to have surgery,
and there are many methodological problems that call the accuracy of these
few data into question (9~.
Even when well-conducted randomized trials are available, problems arise
in using the results to estimate outcome probabilities. Clinicians may forget
about differences between the circumstances of the clinical trial and the
circumstances of clinical practice. They may also forget about the patients
excluded from the clinical trial. These exclusions are not trivial; they com-
monly represent more than 90 percent of the patients for whom the intervention
would be used in practice.
The exclusions are also important because different patients face different
outcome probabilities, even when the care rendered is identical. Figure 3
represents different outcome probabilities for three hypothetical patients.
Clearly, a choice made by or for one of these patients should be based on
probabilities derived from the experience of similar patients. Any inference
about the effectiveness of a particular intervention must adjust for different
mixes of patients with different outcome probabilities.
Outcomes research is an opportunity to integrate clinical research and
clinical practice (Figure 4~. Obviously, there will still be a place for rigor-
ously controlled trials. What outcomes research gives up in terms of inter-
nal validity it more than makes up for in enhanced external validity and
relevance to clinical practice. We need also to characterize patients, that is,
determine disease severity, comorbidity, and other variables that affect prognosis.
We need to characterize the processes of care. We must in addition describe
and sort outcomes by the alternative care processes used and by patient
type. Each of these tasks presents a challenging set of methodological
issues that we must deal with if we are to realize the potential of outcomes
research.
OCR for page 185
185
V REV
~7 ~ ~7
· W ~ W
.
. . . .
VV,
VVV Vie
~VV ~V~
· ~ ~ a ~ ~ ~
Imp <:mo
A
Is ~
· -
c can
~ ,
E Id ^
O ~ ,0
<
ct O
.o can
.
a ~
,
~ c,
~ ~c~ (_ ~ c~-~,
~ , ==~ ~ . .
( \< ~ (en ~ ~
q( i, . ~
... . - .. ~c
, in ~ , ~ ,~
... .
c .
\ ..m _ ____ __ :
(age L-'~__ ~ ('it '2-:~~~''
( AW, . ;
c- c, T: (- ~
C--,m ~ ., (--~-
)
o
·
~ -
04
~- ~
(-- ~- ~ ( -~( - ~}
-'m
~, -
(--N'~ ~ '- ~'< ~O
au
t~o.
c~
~ -
-
~ -
~D
o
o
u,
u}
~ -
~ .
c~ ~ ~
~ - ~ ~
a' ~ ~
~ i, au
c) .
._ a,
~ ~ c)
cd ~
.
c) ~
. - ~
~- ~
c~s ·-~
.O ~ o
. ~ ~ ~
- ~ - ~
~ ~ o
.
3
o
C,) ·
oC ~ ~
~ ._
{V · ^
o
o
d_ ~
a'
_
~ 3 ~
a'
o ~ C~
3 ~ ~ ~
O =.D
· - ,=
.
e.=
O
;~ C~ ~
,4_ ~ ~
~ ~ Cq
ca ~ ~
a.) O -
(~ C) C~
ca O ~
~i
·= =^
C)
~ .=
q)
·_'
OCR for page 186
186
EFFECTIVENESS AND OUTCOMES IN HEALTH CARE
VALUE JUDGMENTS IN DECISION MAKING
Outcomes and effectiveness research has the potential to improve dra-
matically the clinician's ability to estimate clinically relevant outcome
probabilities. Probabilities alone, however, are insufficient for informed
decision making. Whether the pie diagrams in Figure 3 represent probabili-
ties of outcomes for a health care decision or a simple game of roulette,
information about the likelihood of the outcomes must be accompanied by
information about their relative values in order to be helpful to a decision
maker.
The top bar in Figure 5 represents a scale on which we can register the
value judgments of the hypothetical patient with prostate disease. It is
anchored by the least and most desirable outcomes. The markings on the
scale indicate that he prefers his current state to one that would be imposed
by a complication of Alternative 1 (for example, impotence). This patient
might, therefore, opt for the less risky Alternative 2. The bottom two scales
display different value judgments of different hypothetical patients; these
patients are similar enough to face the same outcome probabilities, but with
different preferences. For the second patient, the same health state diminishes
life's quality more; for him, Alternative 1 may be preferable despite the
risks. For the third patient, Alternative 1 would almost certainly be the best
Patient ~ ~
.,
Least
Valued
Patient 2 ~
Least
Valued
Patient 3 ~
Least
Valued
Most
Valued
Most
Valued
Most
Valued
FIGURE 5 Value Judgments for Three Patients. Value judgments for three hypo-
thetical patients, all of whom face the same outcome probabilities. Patient 1 values
the baseline health state highly (the cross hatched triangle), relative to the state
associated with a complication of alternative 1 (the diagonally hatched triangle [e.g.,
impotence]~. Patient 3 prefers the latter to the former.
OCR for page 187
APPLICATION TO CLINICAL PRACTICE
187
choice. The current health state is perceived as a serious hardship, and the
state associated with a complication of Alternative 1 is not.
How confident can a patient be about these value judgments? He or she
may be more confident in making ~ determination about the goodness or
badness of a state that he or she has experienced than one that must be
imagined. Such imaginings may be helped by hearing about the experiences
of other patients. Physicians can provide such vicarious experience, but it
severely tests their communication skills. Furthermore, there is no system-
atically collected body of experience on which to draw.
VALUE JUDGMENTS AND THE ROLE OF
OUTCOMES RESEARCH
The assessment of values or preferences is extraordinarily difficult and
raises a new set of methodological issues (10~. As indicated in Figure 5,
preferences for the same health states vary widely among patients. This has
been demonstrated in a number of important studies that used hypothetical
case scenarios (11,12) and in a large patient interview study of men under-
going surgery for prostate disease (13~. Varying medical practice to reflect
accurately these differences is both appropriate and desirable. These value
judgments also change over time and are influenced by the context of the
decision or the scaling task used. Even when preferences are measured
accurately, there are difficulties in communicating them to other patients
who might benefit from them. At this interface between outcomes research
and clinical practice, the methodological issues relate more to the physician-
patient relationship and its effect on care and outcomes than to the scientific
basis of medicine as defined by the biomedical model.
New information about subjective responses to health states could also
be of value to policymakers. It could help bridge the gap between the
statistical person of cost-effectiveness analysis and the real patient when
making coverage decisions or choosing those conditions for which restrictive
boundary guidelines may be more or less appropriate.
CONCLUSIONS
The methodological issues in the application of effectiveness and out-
comes research to clinical practice depend on the form that the new research
takes in the coming years. Dissemination of results to decision makers will
be of critical importance. Clinicians must be provided with information
that will allow them to estimate accurately the outcome probabilities for
different patients. Clinicians and their patients must be provided with information
that will allow them to make informed value judgments about different
potential health outcomes.
OCR for page 188
188
EFFECTIVENESS AND OUTCOMES IN HEALTH CARE
A more ambitious view of outcomes research would see it take full ad-
vantage of clinical practice as a source of information to inform future
practice (Figure 6~. Aggregate outcomes of individual decisions would
inform professional knowledge regarding outcome probabilities and would
inform value judgments made by professionals and patients. In this case,
the list of issues expands to include accurate baseline description of patients,
measures to ensure the integrity of the therapeutic interventions used, and
the unbiased monitoring and measurement of outcomes, including patients'
subjective responses. Closing this loop of practice and research will require
unprecedented cooperation between clinicians and investigators, but both
have much to gain in the form of a more robust and relevant knowledge
base for the practice of medicine and the delivery of health services.
-
_{'Profession~
~ ,
Individual
Clinical
Decision
Patients
_, Values
\
-
/
Clinical
A///////,// :
Practice
~ V
Aggregate
\ Outcome
-
FIGURE 6 A Model of Clinical Practice and Outcomes Research Functioning as a
Feedback Loop. Aggregate outcomes of many individual clinical decisions serve as
an information base that informs professional knowledge with outcome probabilities
and simultaneously informs patients' and professionals' value judgments with previ-
ous patients' subjective responses to those outcomes.
OCR for page 189
APPLICATION TO CLINICAL PRACTICE
189
REFERENCES
1. McPherson, K., Wennberg, J.E., Hovind, O.B., et al. Small-Area Variation in
the Use of Common Surgical Procedures: An International Comparison of New
England, England, and Norway. New England Journal of Medicine 307:1310-1314,
1982.
2. Patient Outcomes Assessment Research Program: Extramural Assessment Teams.
NCHSR Program Note. Rockville, MD: U.S. Department of Health and Human
Services. November, 1988.i
3. Roper, W.L., Winkenwerder, W.L., Hackbarth, G.M., et al. Effectiveness in
Health Care: An Initiative to Evaluate and Improve Medical Practice. New England
Journal of Medicine 319:1197-1202, 1988.
4. Wennberg, J.E., Roos, N.P., Sola, L., et al. Use of Claims Data Systems to
Evaluate Health Care Outcomes. Mortality and Reoperation Following Prostatectomy.
Journal of the American Medical Association 257:933-936, 1987.
5. Fisher, E.S. and Wennberg, J.E. Administrative Data in Effectiveness Stud-
ies: The Prostatectomy Assessment. Pp. 80-89 in Effectiveness and Outcomes in Health
Care. Heithoff, K.A. and Lohr, K.N., eds. Washington, D.C.: National Academy
Press, 1990.
6. Rothman, K. Modern Epidemiology. Boston: Little, Brown, 1986.
7. Barry, M.J., Mulley, A.G., Fowler, F.J., et al. Watchful Waiting vs. Immediate
Transurethral Resection for Symptomatic Prostatism: The Importance of Patients'
Preferences. Journal of the American Medical Association 259:3010-3017, 1988.
8. Tversky, A. and Kahneman, D. The Framing of Decisions and the Psychology
of Choice. Science 211:453 -458, 1981.
9. Barry, M.J. Medical Outcomes Research and Benign Prostatic Hyperplasia.
In: The Prostate, in press.
10. Mulley, A.G. Assessing Patients' Utilities: Can the Ends Justify the Means?
Medical Care 27~3) Supplement:S269-S281, 1989.
11. Sackett, D.L. and Torrance, G.W. The Utility of Different Health States as
Perceived by the General Public. Journal of Chronic Diseases 31:697, 1978.
12. McNeil, B.J., Weichselbaum, R. and Pauker, S.G. Fallacy of the Five-Year
Survival in Lung Cancer. New England Journal of Medicine 299:1397, 1978.
13. Fowler, F.J., Wennberg, J.E., Timothy, R.P., et al. Symptom Status and
Quality of Life Following Prostatectomy. Journal of the American Medical Asso-
ciation 259:3018-3022, 1988.
iEditors' Note: Now the Patient Outcomes Research Teams of the Agency for
Health Care Policy and Research.
Representative terms from entire chapter:
outcomes research
