Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Promise and Limitations of Effectiveness and Outcomes Research Summary Statement of the TOM Core Committee The committee first acknowledges the major contribution of the Health Care Financing Administration (HCFA) in advancing the conceptual and practical ideas behind effectiveness research. Studies of the actual delivery of health care and "what works in the practice of medicine" are a legitimate and important priority for health scientists. We therefore applaud the imagination and efforts of William Roper, former Administrator of HCFA, Acting Ad- ministrator Louis Hays, and the HCFA staff for their leadership and energy in stimulating interest and focusing attention upon effectiveness research. IMPORTANCE OF EFFECTIVENESS RESEARCH The emerging emphasis on effectiveness is welcome, if not overdue, as a complement to the National Institutes of Health's (NIH) emphasis on effi- cacy. The distinction between effectiveness and efficacy is an important one. Efficacy is typically defined as the outcome of an intervention when it is applied in "ideal," well-controlled circumstances, such as those inherent in prospective randomized controlled trials (RCTs). By contrast, effective- ness means the outcome of that intervention when it is applied in everyday or average circumstances (such as the daily practice of medicine); the latter may include patient groups that differ marginally or considerably from those studied in RCTs. The desirability of high-quality effectiveness research was clearly dem- onstrated in the various committee deliberations. The committees identified numerous areas in which the efficacy of a particular therapy has been documented through RCTs but in which the effectiveness of that therapy is not necessar- ily predicted by the results of the efficacy studies. Two examples make this point. First, almost all RCTs in the treatment 8
INTRODUCTION 9 of acute myocardial infarction (AMI) exclude individuals over age 65. As a result, it is impossible to extrapolate the results of such studies directly to the Medicare population. Second, although excellent clinical trials have revealed the most efficacious treatment for breast cancer, other research makes it clear that physicians offer different options to their older and their younger patients. MEDICARE DATA BASES We also believe that the existing Medicare administrative data bases (known as the Medicare-Medicaid Decision Support System) contain much potentially useful data. Over 31 million elderly individuals are currently in the Medicare program, and they are covered for virtually all inpatient hospi- tal care and a considerable portion of their outpatient care. Beginning in 1990, outpatient prescription drugs and various screening tests will also be covered.) Thus, the size and scope of the HCFA data files offer remarkable opportunities for effectiveness studies based on monitoring and surveillance of large populations. This data base allows us, with considerable accuracy and for specific diagnoses, to (a) track the use of services, the patterns of care, and the costs of those services and care over time; (b) monitor trends in care received and to measure the variations (i.e., patterns of care) by geographical region, institutional providers, type of practitioners, and patient demography; and (c) track what happens to Medicare beneficiaries over time (for example, to learn certain rates of death and utilization-related events such as rehospitalization or use of home health services following a hospital admission). KEY ASPECTS OF EFFECTIVENESS RESEARCH RELIABILITY AND VALIDITY OF DATA All the IOM workshops were concerned with the reliability and validity of data. These problems center on the adequacy of information in the Medicare files about diagnosis, procedures, coding (in general as well as for new technologies), and timing of patient management events. Data on AMIs, for example, raise the difficulty of separating the hospi- talization for the infarction from a hospitalization for cardiac catheterization two weeks later, thus calling into question up to 20 percent of these diag- noses. As another example: when the count of surgical procedures for hip ~Editors' Note: Although true at the time this statement was drawn up, these benefits ultimately were not covered because of the repeal of the Medicare Cata- strophic Coverage Act late in 1989.
10 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE fracture is compared to the overall annual incidence of hip fracture, nearly 20 percent of patients cannot be matched with a procedure. Moreover, the long lag between the introduction of a discrete new tech- nology and its designation with a unique code means that an intervention such as tissue plasminogen activator cannot yet be identified in the Medi- care data bases. Many questions regarding the role and the effectiveness of mammography, biopsy, and surgical therapy for breast cancer require that the temporal relationships of those interventions to outcomes be known. Efforts by HCFA to improve coding and dating of data and to find meth- ods for validation of data are impressive. The current Medicare files are superior to any similar set of insurance claims files that might be tapped today for national effectiveness research. Nevertheless, this area will continue to present major problems. For one thing, data generated for reimbursement may be inadequate for research purposes. In addition, as long as coding is driven mainly by the need to develop charges for care, biases derived from efforts to maximize reimbursement may be introduced. Assessment of pro- cedures may be limited, for example, because the Medicare files from hos- pital discharge abstracts code for only three procedures. Moreover, in the case of procedures, the major Medicare data bases (Part A and Part B) are not consistent in the coding systems used. Clinical vagaries also present problems with regard to initial diagnoses and treatments; accurate description of the type of hip fracture is one example. In short, effectiveness research requires, as does any area of scientific inquiry, confirmation, reconciliation, and validation of data on a continuing basis. Efforts to validate Medicare, Medicaid, and other important data banks must be maintained or even expanded. In some cases, critical data will be made available through the Medicare Peer Review Organizations (PROs), but periodic independent validation of PRO data will also be essen- tial. We conclude that continuing validation of the Medicare data base is essential to the success of an effectiveness research program that relies heavily on those files. Effectiveness research should ensure such validation. LONGITUDINAL STUDIES The three IOM clinical conditions committees noted the need to follow patients over time and across settings of care. The limitations of unsupplemented hospital outcomes data were particularly striking. The core committee rec- ognized the clear need for an episode-of-care approach to analyzing the outcomes of care, which among other things calls for appreciable efforts to collect ambulatory and other out-of-hospital information, including posthospitalization outcomes data. Again, some examples may be useful. Because of varying lengths of stay after inpatient hospital treatment for AMI, mortality rates from the
INTRODUCTION 11 acute event may need to be calculated at some specified, minimum period of time following the hospital episodes (for example, 30 days after admis- sion or after discharge) rather than from in-hospital deaths. In-hospital mortality associated with initial treatment of both breast cancer and hip fracture is very low; only longitudinal studies can identify the effectiveness of a particular treatment combination. Finally, appropriate care for hip fracture requires that the earliest stages of rehabilitation occur during acute hospitalization but that rehabilitation be fully pursued in whatever settings are appropriate and available a rehabilitation hospital, skilled nursing facility, home health care, or family care. In short, before the effectiveness of a particular therapeutic modality can truly be determined, information about care in a variety of sites and from a variety of practitioners will be needed. The diversity of sites of care, coupled with the requirement for longitudinal studies, presents an excep- tional challenge to effectiveness research. TRACKING THE PATIENT Effectiveness studies must follow the patient across all levels and sites of care. Efforts by HCFA to link their inpatient (Part A) and outpatient (Part B) files will help. Mechanisms for obtaining information about ambulatory care, including care given in offices and clinics, will also be essential. Adequate information about drug therapy is particularly important. This concern arose for all three conditions addressed in this project. The assess- ment of treatment for breast cancer depends on knowledge of the chemo- therapeutic agents administered, including their timing and schedule. The opportunity to obtain these data for Medicare recipients should be strongly supported, and every attempt should be made to enter such information (for example, at the point of sale) in a manner that facilitates matching drugs with patients and diagnoses. In addition to the challenges offered by outpatient use of medications, data on hospital drugs, including timing of administration and coding of new agents, continue to be needed. Several instruments intended to capture health status and clinical data are being developed for use at the time of hospital admission and discharge or in conjunction with nursing home or home health care. Such instruments should be simple and comprehensive, yet sensitive and economical in terms of time and money. These attributes would be enhanced by appending to generic instruments some disease-specific risk stratification questions. This requires coordination of instrument development and application. Assess- ing changes in the health of patients tracked throughout the health care system is easier if the content of the data sets derived from these instruments is consistent.
12 HEALTH STATUS ASSESSMENT EFFECTIVENESS AND OUTCOMES IN HEALTH CARE Because health care aims at more than simply extending life, effective- ness research must consider the range of outcomes that is, the diversity of health states appropriate to the patient with the condition under study. Desired outcome may differ with stage of patient management (from screening and prevention through therapy and rehabilitation, to control of symptoms and palliation in terminally ill patients). Each workshop committee recom- mended that attention and care be given to the definition, design, and development of appropriate measures of outcome as part of the effectiveness research program, and each recognized the need to go beyond administrative files and medical records data to obtain outcome information directly from pa- tients or their families. The use of a sensitive, yet simple, patient health status instrument (or set of instruments) in longitudinal studies is highly desirable. This might be based on a questionnaire completed by the Medicare patient and/or the patient's family, or both. When that is not feasible, or when multiple views of the patient's health status are required, the relevant information might be provided by the health care practitioner. Such an instrument should provide information about activities of daily living as part of a functional assessment. Further, it should yield informa- tion about emotional aspects of health status and, if possible, some insight into cognitive function. The core committee endorsed the proposal that such an instrument be applied to 5 percent of all Medicare patients upon their enrollment into the program. Even better, it could also be applied to, say, 5 percent of those enrollees every five years thereafter, to provide a rich longitudinal data base. For those beneficiaries who seek care for an acute illness, this health status document could be updated at the onset of care for the acute episode and at periodic intervals thereafter; a different schedule might be devised for beneficiaries who seek care for chronic illness. In this way, the data from a generic health status instrument would contribute to risk stratifica- tion for this 5 percent cohort and would provide insight into the outcomes of efforts at prevention as well as management of acute illness. The instru- ment might be applied more frequently to the 5 percent cohort as it grows older and the incidence of illness becomes more frequent. A health status instrument could be used for all Medicare patients as part of any long-term effectiveness studies of treatment for an acute illness. For example, it could be filled out at the time of hospitalization after hip fracture, at discharge, and periodically thereafter to ascertain functional recovery after the episode of hip fracture. The committee believes that such a health status instrument could be used effectively in risk adjustment, as an outcome measurement, and to assess prevention efforts. We recognize that for stratification of patients in mortality analysis, clinical information on
INTRODUCTION 13 the acute crisis will be more important than overall health status. The main use of health status information is in charting the patient's progress in terms of functioning and long-term recovery, not in acute episodes. Many reliable and valid generic instruments for measuring health status are available and could be synthesized effectively for this purpose. The committee believes that the Karnofsky index (a physician-completed, can- cer-specific index to measure physical status and activities) will not be satisfactory for this purpose and recommends that attention be given to developing a unique health status instrument. We also acknowledge that more specialized instruments for evaluating health status may be required for specific studies and note that many disease-specific instruments are already available. Examples include measures of pain and of psychological factors in illness. The committee believes, however, that a simple, generic document for use at the time of acute illness as well as for follow-up of a 5 percent cohort is critical to the success of effectiveness studies. RISK STRATIFICATION Little can be concluded about effectiveness when variations in patients' clinical status are unknown and uncontrolled. Risk stratification adjusts for differences among patients, making use of such concepts as case mix and disease stage. By whatever term it is described, the adjustment issue was central for the committees that examined breast cancer, acute myocardial infarction, and hip fracture. The acceptability of effectiveness research results to the clinical community will depend in large part upon convincing evidence that risk stratification and stage of disease have been accounted for. In breast cancer, for instance, accurately determining the stage of disease at the time of initial diagnosis often determines what therapeutic options are considered or offered to the patient (and thus are at issue in any effective- ness investigation). Disease stage also has a profound impact on outcomes, regardless of treatment. Thus, any comparison that neglects risk adjustment is unavoidably biased. Likewise, understanding the effectiveness of treat- ing hip fracture requires the careful identification and assessment of comorbidity, cognitive function, and previous functional capacity, which are strong pre- dictors of outcomes. HCFA has recognized the importance of this concept; in fact, risk stratification provided crucial insights into HCFA's studies of coronary angioplasty versus coronary bypass surgery as a treatment for AMI. Developing a good health status instrument should greatly improve risk stratification and determination of disease stage. Efforts to identify mor- bidity, comorbidity, and acute severity should be encouraged. Identifying the stage of concurrent disease is necessary but may not be sufficient to determine health status, and additional clinical data may be required; these
14 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE data might be gathered through the PRO mechanism, demonstration projects, or other kinds of trials. Newer, innovative methods for risk stratification should also be explored. Use of the Predictive Instrument for Acute Ischemic Heart Disease, devel- oped by Michael Pozen and Harry Selker, to determine an emergency room patient's true probability of having acute cardiac ischemia, including AMI and unstable angina (the group most physicians would consider appropriate for coronary care unit admission), is one example of stratification in studies of AMI. Demographic data, including socioeconomic information and data about prior hospitalization and other use of the acute care system, will also contribute to appropriate risk stratification. PREVENTION The prevention of disease is a long-standing ideal of health care, but it suffers from many theoretical and practical difficulties. The importance of early diagnosis is exemplified by the decision to include screening mammography as a benefit under Medicare. However, the need for research on interven- tions that prevent illness in the Medicare population continues. Prevention involves not only primary prevention (for example, of the original ailment or catastrophic event) but also secondary prevention (of a second hip fracture, another myocardial infarction, or a recurrence of malignancy). Early detec- tion of malignancy using mammography; prevention of atherosclerosis by adequate control of blood pressure, diet, and smoking cessation; and the role of estrogen therapy or certain diuretics in prevention of osteoporosis in hip fracture are all examples of primary or secondary prevention. We need to develop data bases that can identify risk factors and preventive measures that are provided outside the hospital setting, and perhaps without regard to . A. . . . the specific diagnosis at Issue. Because prevention is a long-term intervention, it clearly must begin before age 65 for the Medicare population. Thus, our society has a major stake in preventing disease before persons become eligible for Medicare, if only to limit the burden of disease during Medicare coverage. At the same time, third-party payers for individuals under age 65 have an interest in effective prevention in the younger population. As the powerful HCFA data base grows and improves, the capacity to connect it to data bases for populations under age 65 should be carefully explored and expanded. HCFA has improved Medicaid data in some states, which enhances its ability to develop information about a special segment of the population, namely, the elderly poor. Expanding these connections to other states and exploiting the opportunity to link them with information from private insurers, prepaid group practice systems, and other organizations will be essential if the ef- fectiveness of care before age 65 is to be related to events after age 65. Despite substantial problems of public-private relationships, accessibility,
INTRODUCTION 15 confidentiality, economic competitiveness, and similar factors, private projects of this kind should be investigated. Developing such connections at a relatively early stage might facilitate more cost-effective results for the private sector as well as enhance the validity of Medicare's results. The sharing of well-developed approaches to effectiveness research between public and private sectors will benefit all. THE AGING PROCESS Effectiveness research will provide valuable insights into the aging pro- cess in our society, independent of its contributions to our understanding of the effects of health care. Tracking the health status of a cohort of patients and understanding the impact of health status on acute illnesses should aid in decision making about prevention, screening, diagnosis, therapy, and rehabilitation of a cohort of aged patients. PRINCIPLES TO GUIDE FEDERAL EFFORTS As federal efforts in effectiveness research evolve, certain principles de- serve consideration. The challenges offered by the need for data validation, longitudinal surveillance, risk stratification, health status assessment to- gether with the need for pursuing a diverse but coordinated approach to effectiveness research led us to the following observations. RANGE OF STUDY DESIGNS The core committee believes that a diversity of approaches is needed in this program; proper effectiveness research and outcome studies will re- quire research and demonstration projects, case-control studies, and the like to be conducted by a wide variety of investigators. Such studies will be important not only to test instruments and hypotheses, but also to validate further data from HCFA or other sources. Given this need for diversity, and the complexity of funding mechanisms and research methods required, the committee strongly endorses a coordinated, comprehensive, and balanced DHHS approach to effectiveness and outcome research. FUNDING The importance of research funding that emphasizes extramural investi- gation and investigator-initiated projects without excluding-intramural work or research contracts was reiterated by every study in this project. The recommended use of many research methods including randomized trials, various quasi-experimental efforts such as case-control studies and demon .
16 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE strations projects, natural history studies, and other approaches requires a commensurate level of resources for effectiveness projects. BALANCE OF APPROACHES Although Medicare (and possibly other) data bases provide powerful tools for biostatistica1 and econometric analysis, the committee believes that clinical input and participation are critical in effectiveness and outcomes studies. In this regard we believe that HCFA's decision to involve the IOM in the early stages of its activities has been important. Indeed, all groups working in effectiveness and outcomes research must have a good balance of statistical, economic, and clinical perspectives. Without such balance, the risks of misinterpretation, underinterpretation, or overinterpretation of data are sig ~ Al Scant. COORDINATION We endorse the idea of appointing a high-level advisory committee of individuals with clinical, economic, statistical, and organizational expertise for the effectiveness research program, as suggested by Assistant Secretary for Health James Mason. We further urge that this body have an experi- enced staff and adequate support so that it can function effectively. The DHHS unit responsible for effectiveness research will also be re- sponsible for coordinating the activities of such diverse agencies as the National Center for Health Services Research,2 HCFA, and others. This coordination will need to (a) support the development of new instruments, data bases, and research methodology that can be shared by investigators, (b) prevent redundancy in funding between agencies, and (c) foster appropriate translation and dissemination of results obtained in effectiveness research to help health care providers, policymakers, and the public. Both investigators and providers would benefit from regular updates on data bases. Dissemination of information will be a considerable responsi- bility as the field of effectiveness research evolves. Several workshop participants congratulated HCFA on its willingness to provide investigators with access to HCFA data bases; such accessibility and cooperation are strongly encouraged and supported. TECHNOLOGY ASSESSMENT The effectiveness research organization should also consider some prac- tical aspects of technology assessment. NIH studies do not provide funds 2Now the Agency for Health Care Policy and Research.
INTRODUCTION 17 for the use of a procedure or technology that is under investigation. More- over, Medicare and other third-party payers will not pay for such technolo- gies. To obtain meaningful data on effectiveness, HCFA may have to pro- vide financial support for as-yet-unapproved technologies used in an approved study. This difficult issue needs examination. INFLUENCING PROVIDER BEHAVIOR The effectiveness and outcomes programs of DHHS also need to support studies on how to translate research results into practice. The committee is impressed with how little we know about the factors that influence the behavior of health care providers. High-quality data published in peer- reviewed medical literature and supplemented by direct feedback to physi- cians (for example, through new computer systems) are important mechanisms. Certainly economic incentives and constraints have a major impact. If economic limitations are relied upon as a principal method to influence behavior, however, they may affect some patients in an undesirable way. We need to understand more about how payment mechanisms affect the behavior of health care providers. THE CONFLUENCE OF BIOMEDICAL, HEALTH SERVICES, AND EFFECTIVENESS RESEARCH Our society (indeed, the world) has benefited substantially from scien- tific advances made by biomedical research. The scientific substrate of clinical medicine is maintained and extended by such research. What works in medical practice derives, in a fundamental way, from the success of public and private investments in biomedical research. These efforts deserve continued support. Clinical epidemiological research, with its emphasis on the incidence and prevalence of disease, provides a necessary reminder of the magnitude and importance of clinical problems across the nation. It thus provides indis- pensable guidance to policymakers about research priorities. Epidemiologi- cal research using claims data has been the principal means of demonstrat- ing the variations in use and outcome of medical interventions. Health services research has added important new dimensions to our understanding of the mechanisms by which health care is organized, fi- nanced, and delivered. It has laid the foundation for connecting clinical data on use with expenditure data on resource consumption; it has nurtured the development of assessments of functional and health status; it has promoted concern for outcomes research; and it has been responsible for numerous advances in measurement, methodology, and data base development. Effectiveness research adds another dimension to these activities, one that can be extremely valuable in guiding physicians, patients, the public,
18 EFFECTIVENESS AND OlITCOMES IN HEALTH CARE and policymakers. It does not supplant existing efforts in biomedical, epi- demiological, or health services research. On the contrary, it draws perti- nent data from all of these sources and integrates them in an effort to advance the assessment of clinical practice. The IOM committees for the three workshops were repeatedly confronted with evidence that short-term, quick answers to effectiveness will be rare and of limited value. To describe hospital mortality after a particular surgi- cal procedure for hip fracture provides little meaningful insight into the first-year overall mortality rate of 25 percent among patients experiencing a hip fracture; nor does it convey any important information about the level of function of such a patient six months or a year later. Because level of function has important health and economic consequences, the effectiveness of treatment for this condition requires the longer view. The core committee reflects all these perspectives in its emphasis on the need for access to data about drugs and procedures (and outpatient care in general), information on risk stratification, the development of appropriate tools for measuring health status, and longitudinal studies of a cohort of Medicare beneficiaries. Developing a consistent, comprehensive federal approach that involves many agencies, adequate dissemination of informa- tion, support of diverse analytical approaches, vigorous efforts at validation, and development of effective tools for communicating results to providers of care will do much to advance effectiveness studies.
