Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 8
Promise and Limitations of
Effectiveness and Outcomes Research
Summary Statement of the TOM Core Committee
The committee first acknowledges the major contribution of the Health
Care Financing Administration (HCFA) in advancing the conceptual and
practical ideas behind effectiveness research. Studies of the actual delivery
of health care and "what works in the practice of medicine" are a legitimate
and important priority for health scientists. We therefore applaud the imagination
and efforts of William Roper, former Administrator of HCFA, Acting Ad-
ministrator Louis Hays, and the HCFA staff for their leadership and energy
in stimulating interest and focusing attention upon effectiveness research.
IMPORTANCE OF EFFECTIVENESS RESEARCH
The emerging emphasis on effectiveness is welcome, if not overdue, as a
complement to the National Institutes of Health's (NIH) emphasis on effi-
cacy. The distinction between effectiveness and efficacy is an important
one. Efficacy is typically defined as the outcome of an intervention when it
is applied in "ideal," well-controlled circumstances, such as those inherent
in prospective randomized controlled trials (RCTs). By contrast, effective-
ness means the outcome of that intervention when it is applied in everyday
or average circumstances (such as the daily practice of medicine); the latter
may include patient groups that differ marginally or considerably from those
studied in RCTs.
The desirability of high-quality effectiveness research was clearly dem-
onstrated in the various committee deliberations. The committees identified
numerous areas in which the efficacy of a particular therapy has been documented
through RCTs but in which the effectiveness of that therapy is not necessar-
ily predicted by the results of the efficacy studies.
Two examples make this point. First, almost all RCTs in the treatment
8
OCR for page 9
INTRODUCTION
9
of acute myocardial infarction (AMI) exclude individuals over age 65. As a
result, it is impossible to extrapolate the results of such studies directly to
the Medicare population. Second, although excellent clinical trials have
revealed the most efficacious treatment for breast cancer, other research
makes it clear that physicians offer different options to their older and their
younger patients.
MEDICARE DATA BASES
We also believe that the existing Medicare administrative data bases
(known as the Medicare-Medicaid Decision Support System) contain much
potentially useful data. Over 31 million elderly individuals are currently in
the Medicare program, and they are covered for virtually all inpatient hospi-
tal care and a considerable portion of their outpatient care. Beginning in
1990, outpatient prescription drugs and various screening tests will also be
covered.) Thus, the size and scope of the HCFA data files offer remarkable
opportunities for effectiveness studies based on monitoring and surveillance
of large populations.
This data base allows us, with considerable accuracy and for specific
diagnoses, to (a) track the use of services, the patterns of care, and the costs
of those services and care over time; (b) monitor trends in care received and
to measure the variations (i.e., patterns of care) by geographical region,
institutional providers, type of practitioners, and patient demography; and
(c) track what happens to Medicare beneficiaries over time (for example, to
learn certain rates of death and utilization-related events such as rehospitalization
or use of home health services following a hospital admission).
KEY ASPECTS OF EFFECTIVENESS RESEARCH
RELIABILITY AND VALIDITY OF DATA
All the IOM workshops were concerned with the reliability and validity
of data. These problems center on the adequacy of information in the
Medicare files about diagnosis, procedures, coding (in general as well as for
new technologies), and timing of patient management events.
Data on AMIs, for example, raise the difficulty of separating the hospi-
talization for the infarction from a hospitalization for cardiac catheterization
two weeks later, thus calling into question up to 20 percent of these diag-
noses. As another example: when the count of surgical procedures for hip
~Editors' Note: Although true at the time this statement was drawn up, these
benefits ultimately were not covered because of the repeal of the Medicare Cata-
strophic Coverage Act late in 1989.
OCR for page 10
10
EFFECTIVENESS AND OUTCOMES IN HEALTH CARE
fracture is compared to the overall annual incidence of hip fracture, nearly
20 percent of patients cannot be matched with a procedure.
Moreover, the long lag between the introduction of a discrete new tech-
nology and its designation with a unique code means that an intervention
such as tissue plasminogen activator cannot yet be identified in the Medi-
care data bases. Many questions regarding the role and the effectiveness of
mammography, biopsy, and surgical therapy for breast cancer require that
the temporal relationships of those interventions to outcomes be known.
Efforts by HCFA to improve coding and dating of data and to find meth-
ods for validation of data are impressive. The current Medicare files are
superior to any similar set of insurance claims files that might be tapped
today for national effectiveness research. Nevertheless, this area will continue
to present major problems. For one thing, data generated for reimbursement
may be inadequate for research purposes. In addition, as long as coding is
driven mainly by the need to develop charges for care, biases derived from
efforts to maximize reimbursement may be introduced. Assessment of pro-
cedures may be limited, for example, because the Medicare files from hos-
pital discharge abstracts code for only three procedures. Moreover, in the
case of procedures, the major Medicare data bases (Part A and Part B) are
not consistent in the coding systems used. Clinical vagaries also present
problems with regard to initial diagnoses and treatments; accurate description
of the type of hip fracture is one example.
In short, effectiveness research requires, as does any area of scientific
inquiry, confirmation, reconciliation, and validation of data on a continuing
basis. Efforts to validate Medicare, Medicaid, and other important data
banks must be maintained or even expanded. In some cases, critical data
will be made available through the Medicare Peer Review Organizations
(PROs), but periodic independent validation of PRO data will also be essen-
tial. We conclude that continuing validation of the Medicare data base is
essential to the success of an effectiveness research program that relies
heavily on those files. Effectiveness research should ensure such validation.
LONGITUDINAL STUDIES
The three IOM clinical conditions committees noted the need to follow
patients over time and across settings of care. The limitations of unsupplemented
hospital outcomes data were particularly striking. The core committee rec-
ognized the clear need for an episode-of-care approach to analyzing the
outcomes of care, which among other things calls for appreciable efforts to
collect ambulatory and other out-of-hospital information, including
posthospitalization outcomes data.
Again, some examples may be useful. Because of varying lengths of
stay after inpatient hospital treatment for AMI, mortality rates from the
OCR for page 11
INTRODUCTION
11
acute event may need to be calculated at some specified, minimum period
of time following the hospital episodes (for example, 30 days after admis-
sion or after discharge) rather than from in-hospital deaths. In-hospital
mortality associated with initial treatment of both breast cancer and hip
fracture is very low; only longitudinal studies can identify the effectiveness
of a particular treatment combination. Finally, appropriate care for hip
fracture requires that the earliest stages of rehabilitation occur during acute
hospitalization but that rehabilitation be fully pursued in whatever settings
are appropriate and available a rehabilitation hospital, skilled nursing facility,
home health care, or family care.
In short, before the effectiveness of a particular therapeutic modality can
truly be determined, information about care in a variety of sites and from a
variety of practitioners will be needed. The diversity of sites of care,
coupled with the requirement for longitudinal studies, presents an excep-
tional challenge to effectiveness research.
TRACKING THE PATIENT
Effectiveness studies must follow the patient across all levels and sites of
care. Efforts by HCFA to link their inpatient (Part A) and outpatient (Part
B) files will help. Mechanisms for obtaining information about ambulatory
care, including care given in offices and clinics, will also be essential.
Adequate information about drug therapy is particularly important. This
concern arose for all three conditions addressed in this project. The assess-
ment of treatment for breast cancer depends on knowledge of the chemo-
therapeutic agents administered, including their timing and schedule. The
opportunity to obtain these data for Medicare recipients should be strongly
supported, and every attempt should be made to enter such information (for
example, at the point of sale) in a manner that facilitates matching drugs
with patients and diagnoses. In addition to the challenges offered by outpatient
use of medications, data on hospital drugs, including timing of administration
and coding of new agents, continue to be needed.
Several instruments intended to capture health status and clinical data are
being developed for use at the time of hospital admission and discharge or
in conjunction with nursing home or home health care. Such instruments
should be simple and comprehensive, yet sensitive and economical in terms
of time and money. These attributes would be enhanced by appending to
generic instruments some disease-specific risk stratification questions. This
requires coordination of instrument development and application. Assess-
ing changes in the health of patients tracked throughout the health care
system is easier if the content of the data sets derived from these instruments
is consistent.
OCR for page 12
12
HEALTH STATUS ASSESSMENT
EFFECTIVENESS AND OUTCOMES IN HEALTH CARE
Because health care aims at more than simply extending life, effective-
ness research must consider the range of outcomes that is, the diversity of
health states appropriate to the patient with the condition under study.
Desired outcome may differ with stage of patient management (from screening
and prevention through therapy and rehabilitation, to control of symptoms
and palliation in terminally ill patients). Each workshop committee recom-
mended that attention and care be given to the definition, design, and development
of appropriate measures of outcome as part of the effectiveness research
program, and each recognized the need to go beyond administrative files
and medical records data to obtain outcome information directly from pa-
tients or their families.
The use of a sensitive, yet simple, patient health status instrument (or set
of instruments) in longitudinal studies is highly desirable. This might be
based on a questionnaire completed by the Medicare patient and/or the
patient's family, or both. When that is not feasible, or when multiple views
of the patient's health status are required, the relevant information might be
provided by the health care practitioner.
Such an instrument should provide information about activities of daily
living as part of a functional assessment. Further, it should yield informa-
tion about emotional aspects of health status and, if possible, some insight
into cognitive function.
The core committee endorsed the proposal that such an instrument be
applied to 5 percent of all Medicare patients upon their enrollment into the
program. Even better, it could also be applied to, say, 5 percent of those
enrollees every five years thereafter, to provide a rich longitudinal data
base. For those beneficiaries who seek care for an acute illness, this health
status document could be updated at the onset of care for the acute episode
and at periodic intervals thereafter; a different schedule might be devised
for beneficiaries who seek care for chronic illness. In this way, the data
from a generic health status instrument would contribute to risk stratifica-
tion for this 5 percent cohort and would provide insight into the outcomes
of efforts at prevention as well as management of acute illness. The instru-
ment might be applied more frequently to the 5 percent cohort as it grows
older and the incidence of illness becomes more frequent.
A health status instrument could be used for all Medicare patients as
part of any long-term effectiveness studies of treatment for an acute illness.
For example, it could be filled out at the time of hospitalization after hip
fracture, at discharge, and periodically thereafter to ascertain functional
recovery after the episode of hip fracture. The committee believes that such
a health status instrument could be used effectively in risk adjustment, as an
outcome measurement, and to assess prevention efforts. We recognize that
for stratification of patients in mortality analysis, clinical information on
OCR for page 13
INTRODUCTION
13
the acute crisis will be more important than overall health status. The main
use of health status information is in charting the patient's progress in terms
of functioning and long-term recovery, not in acute episodes.
Many reliable and valid generic instruments for measuring health status
are available and could be synthesized effectively for this purpose. The
committee believes that the Karnofsky index (a physician-completed, can-
cer-specific index to measure physical status and activities) will not be
satisfactory for this purpose and recommends that attention be given to
developing a unique health status instrument. We also acknowledge that
more specialized instruments for evaluating health status may be required
for specific studies and note that many disease-specific instruments are
already available. Examples include measures of pain and of psychological
factors in illness. The committee believes, however, that a simple, generic
document for use at the time of acute illness as well as for follow-up of a 5
percent cohort is critical to the success of effectiveness studies.
RISK STRATIFICATION
Little can be concluded about effectiveness when variations in patients'
clinical status are unknown and uncontrolled. Risk stratification adjusts for
differences among patients, making use of such concepts as case mix and
disease stage. By whatever term it is described, the adjustment issue was
central for the committees that examined breast cancer, acute myocardial
infarction, and hip fracture. The acceptability of effectiveness research
results to the clinical community will depend in large part upon convincing
evidence that risk stratification and stage of disease have been accounted
for.
In breast cancer, for instance, accurately determining the stage of disease
at the time of initial diagnosis often determines what therapeutic options are
considered or offered to the patient (and thus are at issue in any effective-
ness investigation). Disease stage also has a profound impact on outcomes,
regardless of treatment. Thus, any comparison that neglects risk adjustment
is unavoidably biased. Likewise, understanding the effectiveness of treat-
ing hip fracture requires the careful identification and assessment of comorbidity,
cognitive function, and previous functional capacity, which are strong pre-
dictors of outcomes. HCFA has recognized the importance of this concept;
in fact, risk stratification provided crucial insights into HCFA's studies of
coronary angioplasty versus coronary bypass surgery as a treatment for
AMI.
Developing a good health status instrument should greatly improve risk
stratification and determination of disease stage. Efforts to identify mor-
bidity, comorbidity, and acute severity should be encouraged. Identifying
the stage of concurrent disease is necessary but may not be sufficient to
determine health status, and additional clinical data may be required; these
OCR for page 14
14
EFFECTIVENESS AND OUTCOMES IN HEALTH CARE
data might be gathered through the PRO mechanism, demonstration projects,
or other kinds of trials.
Newer, innovative methods for risk stratification should also be explored.
Use of the Predictive Instrument for Acute Ischemic Heart Disease, devel-
oped by Michael Pozen and Harry Selker, to determine an emergency room
patient's true probability of having acute cardiac ischemia, including AMI
and unstable angina (the group most physicians would consider appropriate
for coronary care unit admission), is one example of stratification in studies
of AMI. Demographic data, including socioeconomic information and data
about prior hospitalization and other use of the acute care system, will also
contribute to appropriate risk stratification.
PREVENTION
The prevention of disease is a long-standing ideal of health care, but it
suffers from many theoretical and practical difficulties. The importance of
early diagnosis is exemplified by the decision to include screening mammography
as a benefit under Medicare. However, the need for research on interven-
tions that prevent illness in the Medicare population continues. Prevention
involves not only primary prevention (for example, of the original ailment
or catastrophic event) but also secondary prevention (of a second hip fracture,
another myocardial infarction, or a recurrence of malignancy). Early detec-
tion of malignancy using mammography; prevention of atherosclerosis by
adequate control of blood pressure, diet, and smoking cessation; and the
role of estrogen therapy or certain diuretics in prevention of osteoporosis in
hip fracture are all examples of primary or secondary prevention. We need
to develop data bases that can identify risk factors and preventive measures
that are provided outside the hospital setting, and perhaps without regard to
. A. . . .
the specific diagnosis at Issue.
Because prevention is a long-term intervention, it clearly must begin
before age 65 for the Medicare population. Thus, our society has a major
stake in preventing disease before persons become eligible for Medicare, if
only to limit the burden of disease during Medicare coverage. At the same
time, third-party payers for individuals under age 65 have an interest in
effective prevention in the younger population. As the powerful HCFA
data base grows and improves, the capacity to connect it to data bases for
populations under age 65 should be carefully explored and expanded. HCFA
has improved Medicaid data in some states, which enhances its ability to
develop information about a special segment of the population, namely, the
elderly poor. Expanding these connections to other states and exploiting
the opportunity to link them with information from private insurers, prepaid
group practice systems, and other organizations will be essential if the ef-
fectiveness of care before age 65 is to be related to events after age 65.
Despite substantial problems of public-private relationships, accessibility,
OCR for page 15
INTRODUCTION
15
confidentiality, economic competitiveness, and similar factors, private projects
of this kind should be investigated. Developing such connections at a
relatively early stage might facilitate more cost-effective results for the
private sector as well as enhance the validity of Medicare's results. The
sharing of well-developed approaches to effectiveness research between public
and private sectors will benefit all.
THE AGING PROCESS
Effectiveness research will provide valuable insights into the aging pro-
cess in our society, independent of its contributions to our understanding of
the effects of health care. Tracking the health status of a cohort of patients
and understanding the impact of health status on acute illnesses should aid
in decision making about prevention, screening, diagnosis, therapy, and
rehabilitation of a cohort of aged patients.
PRINCIPLES TO GUIDE FEDERAL EFFORTS
As federal efforts in effectiveness research evolve, certain principles de-
serve consideration. The challenges offered by the need for data validation,
longitudinal surveillance, risk stratification, health status assessment to-
gether with the need for pursuing a diverse but coordinated approach to
effectiveness research led us to the following observations.
RANGE OF STUDY DESIGNS
The core committee believes that a diversity of approaches is needed in
this program; proper effectiveness research and outcome studies will re-
quire research and demonstration projects, case-control studies, and the like
to be conducted by a wide variety of investigators. Such studies will be
important not only to test instruments and hypotheses, but also to validate
further data from HCFA or other sources. Given this need for diversity, and
the complexity of funding mechanisms and research methods required, the
committee strongly endorses a coordinated, comprehensive, and balanced
DHHS approach to effectiveness and outcome research.
FUNDING
The importance of research funding that emphasizes extramural investi-
gation and investigator-initiated projects without excluding-intramural work
or research contracts was reiterated by every study in this project. The
recommended use of many research methods including randomized trials,
various quasi-experimental efforts such as case-control studies and demon
.
OCR for page 16
16
EFFECTIVENESS AND OUTCOMES IN HEALTH CARE
strations projects, natural history studies, and other approaches requires a
commensurate level of resources for effectiveness projects.
BALANCE OF APPROACHES
Although Medicare (and possibly other) data bases provide powerful tools
for biostatistica1 and econometric analysis, the committee believes that clinical
input and participation are critical in effectiveness and outcomes studies. In
this regard we believe that HCFA's decision to involve the IOM in the early
stages of its activities has been important. Indeed, all groups working in
effectiveness and outcomes research must have a good balance of statistical,
economic, and clinical perspectives. Without such balance, the risks of
misinterpretation, underinterpretation, or overinterpretation of data are sig
~ Al
Scant.
COORDINATION
We endorse the idea of appointing a high-level advisory committee of
individuals with clinical, economic, statistical, and organizational expertise
for the effectiveness research program, as suggested by Assistant Secretary
for Health James Mason. We further urge that this body have an experi-
enced staff and adequate support so that it can function effectively.
The DHHS unit responsible for effectiveness research will also be re-
sponsible for coordinating the activities of such diverse agencies as the
National Center for Health Services Research,2 HCFA, and others. This
coordination will need to (a) support the development of new instruments,
data bases, and research methodology that can be shared by investigators,
(b) prevent redundancy in funding between agencies, and (c) foster appropriate
translation and dissemination of results obtained in effectiveness research to
help health care providers, policymakers, and the public.
Both investigators and providers would benefit from regular updates on
data bases. Dissemination of information will be a considerable responsi-
bility as the field of effectiveness research evolves. Several workshop
participants congratulated HCFA on its willingness to provide investigators
with access to HCFA data bases; such accessibility and cooperation are
strongly encouraged and supported.
TECHNOLOGY ASSESSMENT
The effectiveness research organization should also consider some prac-
tical aspects of technology assessment. NIH studies do not provide funds
2Now the Agency for Health Care Policy and Research.
OCR for page 17
INTRODUCTION
17
for the use of a procedure or technology that is under investigation. More-
over, Medicare and other third-party payers will not pay for such technolo-
gies. To obtain meaningful data on effectiveness, HCFA may have to pro-
vide financial support for as-yet-unapproved technologies used in an approved
study. This difficult issue needs examination.
INFLUENCING PROVIDER BEHAVIOR
The effectiveness and outcomes programs of DHHS also need to support
studies on how to translate research results into practice. The committee is
impressed with how little we know about the factors that influence the
behavior of health care providers. High-quality data published in peer-
reviewed medical literature and supplemented by direct feedback to physi-
cians (for example, through new computer systems) are important mechanisms.
Certainly economic incentives and constraints have a major impact. If
economic limitations are relied upon as a principal method to influence
behavior, however, they may affect some patients in an undesirable way.
We need to understand more about how payment mechanisms affect the
behavior of health care providers.
THE CONFLUENCE OF BIOMEDICAL, HEALTH SERVICES, AND
EFFECTIVENESS RESEARCH
Our society (indeed, the world) has benefited substantially from scien-
tific advances made by biomedical research. The scientific substrate of
clinical medicine is maintained and extended by such research. What works
in medical practice derives, in a fundamental way, from the success of
public and private investments in biomedical research. These efforts deserve
continued support.
Clinical epidemiological research, with its emphasis on the incidence and
prevalence of disease, provides a necessary reminder of the magnitude and
importance of clinical problems across the nation. It thus provides indis-
pensable guidance to policymakers about research priorities. Epidemiologi-
cal research using claims data has been the principal means of demonstrat-
ing the variations in use and outcome of medical interventions.
Health services research has added important new dimensions to our
understanding of the mechanisms by which health care is organized, fi-
nanced, and delivered. It has laid the foundation for connecting clinical
data on use with expenditure data on resource consumption; it has nurtured
the development of assessments of functional and health status; it has promoted
concern for outcomes research; and it has been responsible for numerous
advances in measurement, methodology, and data base development.
Effectiveness research adds another dimension to these activities, one
that can be extremely valuable in guiding physicians, patients, the public,
OCR for page 18
18
EFFECTIVENESS AND OlITCOMES IN HEALTH CARE
and policymakers. It does not supplant existing efforts in biomedical, epi-
demiological, or health services research. On the contrary, it draws perti-
nent data from all of these sources and integrates them in an effort to
advance the assessment of clinical practice.
The IOM committees for the three workshops were repeatedly confronted
with evidence that short-term, quick answers to effectiveness will be rare
and of limited value. To describe hospital mortality after a particular surgi-
cal procedure for hip fracture provides little meaningful insight into the
first-year overall mortality rate of 25 percent among patients experiencing a
hip fracture; nor does it convey any important information about the level
of function of such a patient six months or a year later. Because level of
function has important health and economic consequences, the effectiveness
of treatment for this condition requires the longer view.
The core committee reflects all these perspectives in its emphasis on the
need for access to data about drugs and procedures (and outpatient care in
general), information on risk stratification, the development of appropriate
tools for measuring health status, and longitudinal studies of a cohort of
Medicare beneficiaries. Developing a consistent, comprehensive federal
approach that involves many agencies, adequate dissemination of informa-
tion, support of diverse analytical approaches, vigorous efforts at validation,
and development of effective tools for communicating results to providers
of care will do much to advance effectiveness studies.
Representative terms from entire chapter:
health status
