Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 162
--> 7 Evaluating the Effects of Telemedicine on Quality, Access, and Cost Does telepyschiatry provide more timely access to appropriate behavioral health services than conventional arrangements for patients in a remote rural community? How does it affect patients' health and well-being compared to the alternatives? How do costs compare? Are patients and clinicians satisfied with the services? Would they want to use them in the future? Why or why not? These are the kinds of questions that clinicians, patients, managers, and policymakers want answered about telemedicine. This chapter focuses on questions about the quality, accessibility, cost, and acceptability of telemedicine services. Additional questions will, however, be relevant for some organizations, some communities, and some evaluations. For example, because many telemedicine programs also serve educational and administrative purposes, evaluations may reasonably seek to assess results in these areas. The committee's evaluation framework likewise provides for strategic objectives such as strengthening an organization's competitive positive. As described in Chapter 5, the evaluation domains proposed by the federal Joint Working Group on Telemedicine included the "health system interface." Differing in form but not significantly in substance, the committee's framework treats this domain as a set of intermediate technical, clinical, and administrative
OCR for page 163
--> factors that need to be tracked and understood as part of an evaluation of quality, access, cost, and acceptability outcomes. Broader community effects may also be considered in an evaluation. Policymakers may, for example, be interested in the effects of telemedicine on the survival of rural health care providers and the implications of such effects for the overall economic health of rural areas, including their ability to attract or maintain business, educational, and other resources (OTA, 1991; Council on Competitiveness, 1994; GAO, 1996). For any specific evaluation, the selection of measures and criteria will depend on the telemedicine application, the alternatives to which it is compared, the target clinical problems and populations, the setting, and similar factors. Evaluation Criteria And Questions As defined in Chapter 1, an evaluation criterion is a measure, indicator, standard, or similar basis for describing outcomes or making judgments. Because clinical telemedicine varies so much, the committee broadly interpreted its charge to propose a set of evaluation criteria related to its evaluation framework. Applications differ in the medical problems addressed, the evidence base for decisionmaking, and the diagnostic, therapeutic, and other strategies employed. It would have been far beyond the resources for this project to develop operational measures or standards of care specific to the array of teleradiology, teledermatology, telepsychiatry, home health, emergency care, and other applications described in this report. Rather, the committee started with the set of basic questions about quality, access, and cost that guide much health services research, particularly in the interrelated fields of clinical evaluation and technology assessment (IOM, 1993b, 1995a). Although patient satisfaction measures may be incorporated into assessments of quality of care, particularly in managed care plans (Cleary and McNeil, 1988; Gold and Wooldridge, 1995), more specific questions about patient and clinician satisfaction and other perceptions are presented separately in this chapter. Questions about health outcomes are largely subsumed in the discussion of quality but also enter into assessments of cost-effectiveness. Table 7.1 lists the broad categories of questions proposed by the committee. The importance of comparing telemedicine to an alternative
OCR for page 164
--> TABLE 7.1 Categories of Evaluation Questions for Comparing Telemedicine to Alternative Health Services 1. What were the effects of the application on the clinical process of care compared to the alternative(s)? 2. What were the effects of the application on patient status or health outcomes compared to the alternative(s)? 3. What were the effects of the application on access compared to the alternative(s)? 4. What were the costs of the application for patients, private or public payers, providers, and other affected parties compared to the alternative(s)? 5. How did patients, clinicians, and other relevant parties view the application, and were they satisfied with the application compared to the alternative(s)? NOTE: Each question assumes that an analysis of results will control for or take into account severity of illness, comorbidities, demographic characteristics, and other relevant factors. is highlighted in each question. The note for the table emphasizes that the research design and analytic strategy will need to take into account and control for such factors as the initial condition of patients. Thus, each question should be read with the phrase " other things being equal" as an implicit preface. The next sections of this chapter provide definitions, discuss key concepts, and present additional questions focusing on different aspects of quality, access, cost, and patient and clinician attitudes. These sections should be read in the context of the overall framework presented in Chapter 6. That is, relevant patient and organizational characteristics should be identified and considered as they might affect results. The level of an evaluation—whether it reflects a patient, corporate, or societal perspective—should also be identified. The fit between the project objectives and results and the evaluation sponsor's purposes or strategic plan also needs to be factored into the plan for analysis and the interpretation of results. The human and policy issues identified in Chapters 3 and 4 likewise warrant attention so that evaluation planning casts a wide net for possible benefits and costs of an application. Some telemedicine evaluations will focus less on individual patients than on populations, including but not limited to those enrolled in managed care plans. Analyses may consider outcomes for
OCR for page 165
--> an entire patient population or may concentrate on outcomes for the least healthy or most vulnerable groups in a population (e.g., elderly individuals, migrant workers). For example, a telemedicine application might target a high-risk group to test whether surveillance and early intervention could reduce hospitalization and net costs. Quality Of Care The ultimate purpose of any medical care is to maintain or improve health and well-being. Thus, how clinical applications of telemedicine affect the quality of care and its outcomes is a central evaluative question—as it is for any health service. Definitions and Concepts As defined in Chapter 1, quality of care is "the degree to which health care services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge" (IOM, 1990c).1 A few points about this definition are worth noting. First, the definition covers both individuals and populations and both current and potential users of health care. This is consistent with an increasing focus in health services research and health policy on how different clinical interventions, programs, and resources can be deployed to the greatest social advantage. Second, because the evidence base about what works in health care is still modest, the definition acknowledges the relevance of professional knowledge, which includes experience and judgment as well as the results of biomedical and clinical research. Third, as is traditional in the literature on quality of care, the definition encompasses the link between the processes and the outcomes of care (Donabedian, 1966, 1982, 1985), although the emphasis in recent years has been on the latter. Many studies of health care quality also search for structural aspects of quality, for example, characteristics of a health system's personnel or organization that are associated with better health outcomes and that can be incorporated into accreditation or credentialing programs. 1 The discussion in this section draws on the Institute of Medicine's work over the past decade on quality of care, effectiveness research, and related topics (in addition to IOM, 1990c, see IOM, 1985, 1990a, 1991, 1992a).
OCR for page 166
--> Finally, the definition deliberately omits resource constraints on the grounds that judgments of what constitutes excellent, acceptable, or unacceptable quality should be independent of constraints on resources. This does not, however, imply that decisionmakers can or should ignore resources in making decisions about what level of quality is desired and affordable. In recent years, traditional quality assessment and assurance concepts and strategies in health care have been powerfully reshaped by proponents of continuous quality improvement or total quality management models. These models stress internal responsibility for quality rather than external regulation. As noted in Chapter 6, they also posit planning, control, assessment, and improvement activities grounded in statistical and scientific precepts and driven by data. Conventionally, three broad types of quality problems have been differentiated. They are overuse of care (e.g., unnecessary telemedicine consultations); underuse of care (e.g., failure to refer a patient for a necessary consultation); and poor technical or interpersonal performance (e.g., incorrect interpretation of pathology specimen or inattention to patient concerns). In principle, no one of these three problems is more important than any other. Depending, however, on the setting, the clinical condition, the predominant financing mechanism, and other circumstances, one area may warrant more attention than another in a particular telemedicine evaluation. For instance, as discussed in Chapter 4, policymakers have been concerned that payment for telemedicine in a fee-for-service context might lead to excessive consultations that might, in turn, lead to overuse of diagnostic or therapeutic services for which the benefit would not be worth the risk. In capitated environments, the worry has been that financial incentives might lead to underuse of appropriate face-to-face consultations or other services and to poorer performance in the interpersonal aspects of patient care, including good communication between clinician and patient. For purposes of this discussion and consistent with past usage in IOM reports, appropriate care is defined as care for which "the expected health benefit [exceeds] the expected negative consequences by a sufficient margin" that the care is worth providing (Park et al., 1986, p. 6). At what point is the extra margin of expected benefit such that an intervention might be "worth" any additional risk, therefore making the intervention appropriate? Answering this question
OCR for page 167
--> necessarily involves subjective—and sometimes controversial—judgments as well as objective clinical information. Such judgments may be arrived at through expert consensus processes or by reference to other interventions that have been accepted as standard practice. The clinical effects of telemedicine applications can be measured and compared at several levels. One may, for example, look for effects on the process of care or for effects on the outcomes of care or both. In a discussion of the impact of diagnostic technologies, Fineberg and colleagues (1977) distinguished several process and outcome dimensions that might appropriately be assessed by evaluators. These dimensions include technical capacity—whether a technology is safe, accurate, and reliable (e.g., how do transmitted digital images compare to films?); diagnostic accuracy—whether a technology contributes to a correct diagnosis (e.g., was an initial dermatology diagnosis by a primary care clinician corrected after review by a dermatologist?); diagnostic impact—whether a technology provides diagnostic information that is useful in making a diagnosis (e.g., after the telemedicine consult, is a face-to-face consultation still necessary?); therapeutic impact—whether a technology influences patient management or therapy (e.g., do paramedics perform better when they have access to emergency cardiac telemetry?); and patient outcome—whether a technology improves patients' health and well-being (e.g., are postsurgical patients telemonitored in a nursing home more or less likely to develop wound infections than patients remaining in the hospital?). The first four dimensions involve processes of care. The last involves outcomes. Both categories figure in the question set presented below. In principle, several kinds of process and outcomes measures might be relevant for any specific telemedicine application. For example, in North Carolina, researchers studying an emergency medicine project involving rural emergency departments and four medical schools plan to collect process of care, utilization, and outcomes data on "patient flow, time to diagnosis, effectiveness of specialty
OCR for page 168
--> consultation, types of cases, appropriateness of intervention at local levels, and patient stabilization” (Evaluation Plan of the North Carolina Emergency Consult Network, p. 2). Questions about Quality of Care and Patient Outcomes As explained above, the committee concluded that it would identify basic questions about quality of care to guide evaluators in devising questions and criteria specific to their telemedicine project, its objectives, and its context. Table 7.2 lists these questions. Some measures such as survival appear to have limited relevance for most telemedicine uses, although mortality measures might be considered in evaluating certain applications in emergency care and home monitoring. Processes of Care The first set of measures in Table 7.2 relate to processes of care. Process of care measures are useful in their own right as they help evaluators to understand how care is provided, how an intervention changes other aspects of the care process, and how processes of care might be improved to achieve better outcomes or greater efficiency (Donabedian, 1966, 1982; IOM, 1990a; Wilson and Cleary, 1995; Wilson and Kaplan, 1995). It is important to note that the process measures discussed here do not cover a variety of important but often routine quality assurance procedures. For example, those involved with digital radiology and teleradiology have developed and are still improving quality assurance methods for testing, calibrating, and otherwise monitoring and maintaining equipment at central and remote sites (Forsberg, 1995). Sometimes, process measures are employed as proxies for health outcomes when data on the latter are limited or unavailable. For example, an early retrospective evaluation of Army telemedicine in Somalia and other sites was able to determine whether the diagnosis or patient care plan changed after the telemedicine consultation, but evaluators lacked data to judge whether the change made a difference in patient outcomes (Walters, forthcoming). Difference in diagnosis may be the most common outcomes-related measure found in tele-medicine evaluations to date. Ideally, previous research should
OCR for page 169
--> TABLE 7.2 Evaluating Quality of Care and Health Outcomes What were the effects of the telemedicine application on the clinical process of care compared to the alternative(s)? Was the application associated with differences in the use of health services (e.g., office visits, emergency transfers, diagnostic tests, length of hospital stay)? Was the application associated with differences in appropriateness of services (e.g., underuse of clearly beneficial care)? Was the application associated with differences in the quality, amount, or type of information available to clinicians or patients? Was the application associated with differences in patients' knowledge of their health status, their understanding of the care options, or their compliance with care regimens? Was the application associated with differences in diagnostic accuracy or timeliness, patient management decisions, or technical performance? Was the application associated with differences in the interpersonal aspects of care? What were the effects of the telemedicine application on immediate, intermediate, or long-term health outcomes compared to the alternative(s)? Was the application associated with differences in physical signs or symptoms? Was the application associated with differences in morbidity or mortality? Was the application associated with a difference in physical, mental, or social and role functioning? Was the application associated with differences in health-related behaviors (e.g., substance abuse)? Was the application associated with differences in patient satisfaction with their care or patient perceptions about the quality or acceptability of the care they received? NOTE: Each question assumes that analysis of results will control for or take into account severity of illness, comorbidities, demographic characteristics, and other relevant factors. have demonstrated a link between the proxy variable and the desired health outcome. Depending on the objective of an evaluation, the nature of the clinical problem and the intervention, and the resources and data available, the same variable (e.g., vaccination rates) may be treated as an outcome in some studies and as a process measure in others. Characteristics of a specific telemedicine project may affect the interpretation of utilization and other process information. For example,
OCR for page 170
--> given similar patient populations, one might expect an experienced primary care physician to refer fewer patients for specialty consultations than a nurse practitioner. One hypothesis for exploration is that the utility of telemedicine is greater when the (initial) difference between the skills and experience of consultant and the referring clinician is greater. Outcomes of Care The value of process measures notwithstanding, decisionmakers, clinicians, and patients have increasingly demanded information on outcomes and questioned the assumption that conformance to procedural standards equates to good health outcomes (Relman, 1988; IOM, 1990c; Lansky, 1993). As suggested in Table 7.2, measures of patient outcomes may focus on clinical status (physiological and cognitive); mental and emotional well-being; feelings of energy and vitality; or functional capacity (e.g., ability to perform various tasks related to personal life or employment). Patient outcomes are generally considered to include not just desired endpoints of health care (e.g., reduced mortality, improved functioning) but also a broad range of immediate and intermediate results (e.g., reduced blood pressure, higher vaccination rates, fewer hospital readmissions for surgical complications) (Brenner et al., 1995). Because patient outcomes data are often difficult to obtain for longer-term outcomes and outcomes that occur outside the hospital, immediate or intermediate clinical results (e.g., physiological signs such as blood pressure or postoperative complications) are frequently used in place of longer-term results. The advantage of such measures is that they may be more directly and strongly linked to elements of a clinical intervention. Their great disadvantage is that their relationship to outcomes of greater relevance to patients (e.g., function) may be theoretical rather than documented through prior research. The longer the interval that defines an episode of care or a long-term outcome and the more sources of care (and record systems) involved, the more difficult it is to obtain information. Eventually,
OCR for page 171
--> the integrated, longitudinal computer-based patient record should overcome some of the difficulties in securing satisfactory shorter-and longer-term outcomes data. A very large literature has accumulated on categories of health outcomes and the tools for measuring them (see, e.g., the quality primer in IOM, 1990c, Vol. II; Lohr, 1992; McDowell and Newall, 1993; CHPS, 1995; Fowler, 1995). Tools for assessing clinical performance and health outcomes have progressed considerably in recent years as methodologists and researchers have tested and improved the validity and reliability of measures and made them more relevant and usable in routine clinical practice. For example, health services researchers have developed shorter and more easily used instruments to measure health status. They also have devised both generic measures and more focused instruments for specific clinical conditions (e.g., diabetes) and settings (e.g., ambulatory care). Each telemedicine evaluation will have to select quality and outcomes measures that fit the patients, settings, services, desired outcomes, and other characteristics of its project. In some cases, well-established instruments (e.g., for measuring depression or determining patients' assessment of their quality of life) may be available and appropriate for measuring patient outcomes. In other cases, evaluators will have to create measures and data collection instruments, with less confidence in their validity and reliability (see the last section of this chapter). Adjustments for Patient Risk or Severity of Illness Proper interpretation of patient outcomes data requires good information on patient characteristics, in particular, their health status. Comparisons of clinical interventions or programs should be adjusted statistically to account for differences in patient risk factors. These adjustments are also essential for proper interpretation of comparisons involving the costs of patient care alternatives. Various schemes have been devised to measure and adjust for differences in the seriousness of patients' medical status (Thomas and Ashcraft, 1989, 1991; Iezzoni, 1992; Hopkins and Carroll, 1994). Some focus on care settings (e.g., intensive care units) whereas others are more general. Some are designed less for quality assessment purposes than for assuring that capitated, per case, or other payment mechanisms do not pay too much for healthier than average
OCR for page 172
--> patients and too little for sicker patients. Debate continues on the strengths and limitations of different strategies, but the committee stresses the importance of attempting to identify and adjust for differences in patient status. Other Quality of Care Issues As noted elsewhere in this report, primary care physicians or nurse practitioners who participate with patients in telemedicine consultations may learn more about clinical problems that they once referred to specialists and, thereby, become more proficient at identifying and managing repeat problems on their own. Telemedicine may, in this respect, be analogous to the informal "curbside" consultation about a specific patient, a process that clinicians may value more highly than consulting a journal or undertaking formal continuing medical education. The extent to which clinical applications of telemedicine have this kind of educational effect is not well documented. The committee believes this area warrants further study. Such study should consider not only changes in knowledge but also changes in practice and, preferably, in short- or long-term health outcomes. In addition, systems-oriented evaluations may be warranted to identify how telemedicine systems can support local quality improvement activities through (a) access to data resources, medical literature, and expert opinion, (b) focused educational interventions and mentoring initiatives; and (c) interorganizational collaborations. Another question related to the impact of telemedicine use on users' knowledge or skills is whether clinicians become more skilled in telemedicine (e.g., relating more effectively to patients during interactive video consultations, reading transmitted images more accurately) as they use a particular application more often. Does some kind of learning curve exist for certain applications? If so, would studies find that a higher volume of use was associated with better outcomes beyond the learning period?2 What might this imply for 2 Interest in the link between volume and quality of care has arisen primarily in the context of selected surgical and other procedures. Evidence suggests that surgeons who routinely perform a large number of certain relatively complex procedures tend to have better outcomes than those performing such procedures only occasionally (Flood et al., 1984; Hughes et al., 1987; Luft et al., 1987; Hannan et al., 1989; Woods et al., 1992; Hannan et al., 1992). Some
OCR for page 183
--> among services, particularly given the discounted, per case, or other payment arrangements that now apply for a substantial portion of health services. Payments, which are based on actual financial transactions, are usually preferable to charges, although in markets characterized by deep discounts to some payers, they too may be a poor proxy for direct measures of costs. Capitated payments or payments for packages of services, such as diagnosis-related groups (DRGs), however, may not vary with changes in resource use and cost. Documenting the actual use and per unit cost of resources to provide a service is clearly the preferable approach, though much more difficult to do (see, e.g., Williams, 1996). Conceptual Challenges Cost analyses of telemedicine face certain conceptual challenges that typify new device-based technologies with sizable fixed costs and multiple potential uses. Cost analyses can address these issues and clarify their implications but cannot definitively resolve them. One difficulty arises from the varied uses to which a telemedicine system may be put. Parts of the system might be used to support emergency medical services, radiology consults, interactive patient counseling sessions, and monitoring of patients in their homes. Although each application may have costs specific to its use, such as certain personnel and supplies, all the applications may share other costs related to certain equipment and perhaps certain personnel and supplies. In contrast to accounting conventions, which apply administrative rules to apportion such joint costs of production, economic principles call for allocating joint costs according to the demand that each service faces (OTA, 1980; Sisk et al., 1991). Another challenge arises because telemedicine, like other innovations, may lead to expanded indications for use. For example, a telemedicine system may be established to permit more timely diagnosis and treatment of trauma patients in rural areas. Once available and accepted, however, primary care physicians may use telemedicine for less urgent cases that they once handled on their own. Even if per unit costs of telemedicine decline with the greater volume, total use and total expenditures may increase. A third—and by now familiar—challenge is that technological change may render a static study of benefits, harms, and costs outdated, even before the analysis is completed. The diffusion and
OCR for page 184
--> evolution of technologies, such as those used in telemedicine, is a dynamic process that calls for ongoing evaluation. As adoption and use proceed, telemedicine users are likely to gain greater experience and proficiency that, in turn, may be reflected in lower costs and better outcomes. To better inform decisionmakers, the possibility of expanded indications or proficiency-related cost reductions may be modeled in a sensitivity analysis. As described in Chapter 6, if uncertainty surrounds the values of certain variables in the evaluation that are considered key, sensitivity analysis can vary the values over reasonable ranges. The findings will indicate how sensitive the results are to these uncertainties. Question about Costs and Cost-Effectiveness Table 7.4 summarizes the questions related to costs proposed by the committee. This summary does not distinguish between major categories of costs (e.g., fixed and marginal, capital and operating). Again, the selection of specific measures will depend on the type of application and the context in which it is employed. Some of the questions in Table 7.4 highlight an important but difficult problem for evaluations of telemedicine and, indeed, evaluations of any new technology. That is, what was the effect of the technology on costs over an episode of acute or chronic illness? An evaluation that cannot link services and costs to such episodes may fail to identify care that prevents the need for later, more expensive care or, alternatively, causes a cascade of additional services. For example, home monitoring via telemedicine might encourage quicker identification and response to problems that might be costly to treat if not caught early. Alternatively, such monitoring might identify more borderline problems and generate more home or office visits (see, e.g., Weinberger et al., 1996). As noted elsewhere in this report, the longer the interval that should be tracked in an evaluation, the more difficult become the problems in collecting and properly attributing relevant data. Decision Rules for Analyzing Cost-Effectiveness Results For some patterns of cost-effectiveness results, the findings strongly suggest certain decisions. For example,
OCR for page 185
--> TABLE 7.4 Evaluating Health Care Costs and Cost-Effectiveness What were the costs of the telemedicine application for participating health care providers or health plans compared to the alternative(s)? Was an application associated with differences in attending clinicians' costs for personnel, equipment, supplies, administrative services, travel, or other items? Was an application associated with differences in revenues or productivity? What was the net effect? Was an application associated with differences in consulting clinicians' or consulting organizations' costs for personnel, equipment, supplies, space, administrative services, travel, or other items? Was an application associated with differences in revenues or productivity? What was the net effect? Was an application associated with differences in the cost per service, per episode of illness, or per member (health plan enrollee, capitated lives) per month? What were the costs of the telemedicine application for patients and families compared to the alternative(s)? Was the application associated with differences in direct medical costs for patients or families? Was the application associated with differences for patients or families in other direct costs (e.g., travel, child care) or indirect costs (e.g., lost work days)? What were the costs for society overall compared to the alternative(s)? Was an application associated with differences in total health care costs, the cost per service, per episode of illness, or per capita? How did the costs of the application relate to the benefits of the telemedicine application compared to the alternative(s)? NOTE: Each question assumes that analysis of results will control for or take into account severity of illness, comorbidities, demographic characteristics, and other relevant factors. If an alternative is more costly and performs less well (e.g., produces fewer health benefits), it is undesirable. If an alternative is more costly and performs as well, it is undesirable. If an alternative is less costly and performs better, it should be used.
OCR for page 186
--> If an alternative is less costly and performs as well, it should be used. In other cases, cost-effectiveness results are more equivocal and judgments will be more subjective. For example, If an alternative is more costly and performs better, are the benefits gained worth the extra costs? If an alternative is less costly and performs less well, are the savings worth the health benefits foregone? Some analysts have suggested ranges of costs that are considered reasonable, for example, a year of healthy life gained for less than $100,000 (Laupacis et al., 1992). Technology assessments often compare the cost for the option being evaluated to the cost for a well-established technology. Thus, the cost-effectiveness of population-based screening for prostate cancer might be compared to the cost-effectiveness of screening for cervical cancer. In general, cost-effectiveness analysis can guide, but not dictate, judgments about the reasonableness of costs for the health benefits obtained from different health technologies. Decisionmakers must also consider budgetary limitations as well as cost-effectiveness. Indeed, it may well be that not all technologies considered to be cost-effective (e.g., that can gain a year of healthy life for less than $100,000) can be afforded, given the number of cases potentially involved and the total budgetary implications of different technologies. Patient And Clinician Perceptions The discussion of human factors in Chapter 3 stressed patient and clinician perceptions as they may affect the acceptance and adoption of telemedicine. This chapter has noted patient perceptions as a factor to be considered in evaluating quality, access, or cost-effectiveness. They are also important in their own right to the extent that successful telemedicine applications depend on patient and clinician acceptance. Attempts to assess patient satisfaction or perceptions of quality derive in part from the consumer movement and quality improvement philosophies that have promoted patient autonomy, informed
OCR for page 187
--> decisionmaking, and patient-centered care (see, e.g., President's Commission, 1983; Eddy, 1990; IOM, 1990c, 1992a; Kasper et al., 1992; and the sections on human factors and continuous quality improvement in Chapters 3 and 6, respectively). In recent years, increased competition in health care markets has also focused the attention of health plans, facilities, and clinicians on how patients or consumers view the quality, accessibility, or cost of the care they offer (Corrigan and Nielson, 1993; Gold and Wooldridge, 1995; Nelson et al., 1995). Employers and governments who purchase coverage for their employees or beneficiaries also have demanded such information. More generally, this is an era characterized by a steady stream of reports about reduced citizen trust in major social institutions and professions and increasing concern about the effect of managed care and selective contracting on physicians' allegiance to their patients. As a result, some effort may be warranted to assess patient trust in the clinicians and health care organizations involved in a telemedicine application. Clinician perceptions are less often evaluated than patient perceptions, but efforts to improve the effectiveness or efficiency of care may depend on how satisfied those who provide care are with the conditions of practice (e.g., how convenient a telemedicine consultation is). In the committee's view, those evaluating telemedicine have been fairly sensitive to the clinician perspective. They have recognized that the special demands created by the complex and sometimes unfriendly technical infrastructure of telemedicine may frustrate clinicians, slow the provision of care, and create concerns about professional image. The discussion of human factors in Chapter 3 underscores the importance of considering clinician perspectives and needs. In several telemedicine evaluations, patient satisfaction data appear to be the only patient-level data collected (ORHP, 1995). The committee considers this evaluative focus far too limiting, although it agrees that evaluators should consider patient—and clinician—views. The efforts by federal agencies to strengthen evaluations of federally funded telemedicine projects (as described in Chapter 5) reflects, in part, a recognition of the limitations of patient satisfaction data. Efforts to standardize questionnaires are also under way, as described in Chapter 5.
OCR for page 188
--> Methods and Focus Attempts to assess patient or clinician perspectives usually involve written questionnaires. Questionnaires are attractive tools because they are relatively inexpensive and convenient to administer and analyze, especially if they can be computer scored. They are also relatively flexible and can be administered on-site, by mail, or by telephone, although the validity and reliability of different forms of administration needs to be considered on a case-by-case basis. Some questionnaires focus on discrete encounters (e.g., an office visit) whereas others focus on institutions or organizations (e.g., hospitals or health plans). For the immediate future, telemedicine evaluations will most likely focus on encounters. The validity and reliability of various instruments for measuring patient satisfaction have been assessed, but more work remains to be done in general and with respect to specific populations, interventions, settings, and outcomes (Ware et al., 1988; Webster, 1989; Hall et al., 1990; IOM, 1990a; Rubin, 1990; Peterson and Wilson, 1992; Carey and Seibert, 1993; Rubin et al., 1993; Bayley et al., 1995; Gold and Wooldridge, 1995; Stump et al., 1995; Etter et al., 1996). Those who use surveys also have to be sensitive to the methodological problems frequently encountered in many kinds of survey research (e.g., nonresponse rates, accuracy of patient recall, positive response bias). Telemedicine applications potentially offer an unusual opportunity to explore patient satisfaction data in more depth. Because telemedicine encounters may involve video records, it may be possible to match individual encounters with questionnaires and to assess the encounters qualitatively in light of the survey responses. In addition to providing feedback to clinicians and program administrators, evaluators could explore how such qualitative assessments could provide additional guidance about improving practices that appear associated with negative responses. Video taping and critiquing has become relatively common as a teaching tool for medical students. As is true for feedback strategies in general, evaluators would need to provide for appropriate patient consent and be prepared for clinician reaction to negative evaluations.
OCR for page 189
--> Questions about Patient and Clinician Perceptions Tables 7.5 and 7.6 present general questions that may be asked about patient or clinician perceptions. The questions concerning patient satisfaction with telemedicine reflect the approach taken in the applicable Medical Outcomes Study (MOS) visit-specific questions. This approach has been extensively tested (Rubin et al., 1993; Bayley et al., 1995). Although the selection of specific questions will depend on the purposes of a particular evaluation, the design and administration of questionnaires should follow general principles of questionnaire construction (Rossi et al., 1983; Lessler, 1995). Depending on the objectives of an evaluation, relatively general questions may be adequate. If, however, the objective is to pinpoint problems, then questions may need to be not only more specific but also more quantitative. For example, rather than ask generally about whether clinicians found the application convenient, questions might be asked about how much time the consultation took or about whether the hardware or software was difficult to manipulate and TABLE 7.5 Evaluating Patient Perceptions Were patients satisfied with the telemedicine service compared to the alternative(s)? How did patients rate their physical and psychological comfort with the application? How did patients rate the convenience of the encounter, its duration, its timeliness, and its cost? How did patients (and family members) rate the skills and personal manner of the consultant and the attending personnel (e.g., primary care physician, nurse practitioner)? Was the lack of direct physical contact with the distant clinician acceptable? How did patients rate the explanations provided to them of what their problem was and what was being recommended? Did patients have concerns about whether the privacy of personal medical information was protected? Would patients be willing to use the telemedicine service again? Overall, how satisfied were patients with the telemedicine services they received? NOTE: Each question assumes that analysis of results will control for or take into account prior patient experiences with the health care system, severity of illness, comorbidities, demographic characteristics, and other relevant factors.
OCR for page 190
--> TABLE 7.6 Evaluating Clinician Perceptions Were attending/consulting clinicians satisfied with the telemedicine application compared to the alternative(s)? How did attending/consulting clinicians rate their comfort with telemedicine equipment and procedures? How did attending/consulting clinicians rate the convenience of telemedicine in terms of scheduling, physical arrangements, and location? How did attending/consulting clinicians rate the timeliness of consultation results? How did attending/consulting clinicians rate the technical quality of the service? How did attending/consulting clinicians rate the quality of communications with patients? Were attending/consulting clinicians concerned about maintaining the confidentiality of personal medical information and protecting patients' privacy? Did attending/consulting clinicians believe the application made a positive contribution to patient care? Would the clinicians be willing to use the telemedicine services again? Overall, how satisfied were the attending/consulting clinicians with the telemedicine service? NOTE: Each question assumes that analysis of results will control for or take into account severity of illness, comorbidities, demographic characteristics, and other relevant factors. how much time was lost to such problems. In addition, in depth interviews may be useful to develop a fuller understanding of how people perceive the advantages and disadvantages of telemedicine. The consistency and stability of patient perceptions may warrant particular attention. For example, one unpublished study of telecardiology patients found that patients did not find the experience unpleasant (93 percent), an invasion of privacy (95 percent), or unacceptable for lack of physical contact (88 percent). Nonetheless, only 67 percent said they would use the system for emergency or first visits and only 51 percent wanted to use it for follow-up visits (Mattioli, 1996). In an unpublished follow-up survey a year later (which had a 54 percent response rate), a third of the respondents said they would use the system only in an emergency and a third would go elsewhere if it were their only option.
OCR for page 191
--> Desirable Attributes Of Evaluation Criteria Drawing on the work of several groups considering practical but systematic means of improving clinical practice and health care delivery (IOM, 1990c, 1992a,b; Medical Outcomes Trust, 1995; CPRI, 1996), the study committee identified several desirable characteristics or attributes of evaluation criteria (Table 7.7). These attributes are generic, that is, in principle, they should apply to quality, access, and cost criteria alike and to qualitative as well as quantitative measures. They are also ideal attributes; actual criteria will almost certainly fall short on at least some aspects. For several of the attributes (including reliability and validity) and certain kinds of clinical measures, a controlled vocabulary (i.e., a precise, common clinical terminology) is important. The need for a controlled vocabulary arises from a common difficulty in clinical research, clinical practice guidelines, and medical informatics: the lack of unambiguous, uniform descriptors of patient problems (see IOM, 1990c, 1992a; Gibson and Middleton, 1994; Ozbolt et al., 1994). For example, terms like "moderate bleeding" or "persistent TABLE 7.7 Desirable Attributes of Evaluation Criteria Reliability/Reproducibility An evaluation instrument or criterion is reliable if repeated use under identical circumstances by the same or different users produces the same results. Validity An evaluation instrument or criterion is valid if it measures the properties, qualities, or characteristics it is intended to measure. Responsiveness An evaluation instrument or criterion is responsive if it can detect important differences in outcomes across evaluation groups or time periods. Interpretability An evaluation instrument or criterion is interpretable if users find the results of its application understandable. Feasibility An evaluation instrument or criterion is feasible if users can accomplish the required activities, collect the necessary information, and analyze the resulting data within available evaluation resources and without imposing excessive burdens on those whose cooperation is required for the evaluation. Flexibility An evaluation instrument or criterion is flexible if it is adaptable to a variety of evaluation problems or circumstances. Documentation An evaluation instrument or criterion is documented if the protocols for applying and interpreting it are specified and if evidence of its successful use is summarized or cited.
OCR for page 192
--> bleeding" may be interpreted differently in practice by different observers. Bleeding defined in terms of volume loss or hematocrit drops is more precise. Even if definitions are unambiguous, a problem remains if they are not uniformly used. In this context, a controlled vocabulary is one specified by those responsible for an information system and one that precludes users from adding unauthorized terms. Developing a controlled vocabulary and implementing it are long-term challenges. Several schemes have been developed to increase uniformity in the coding of patient history and physical results, medical diagnoses, or procedures. They go under a variety of abbreviations and acronyms (e.g., ICD-9-CM, CPT-4, SNOMEDIII) and are described in detail elsewhere (e.g., PPRC, 1988; IOM, 1991; AMA, 1993; CAP, 1993; Gibson and Middleton, 1994). To build on these efforts, the National Library of Medicine has developed a Uniform Medical Language System (UMLS) Metathesaurus to map terms used by such schemes. Conclusion This chapter has reviewed issues in measuring and evaluating critical outcomes for telemedicine and proposed general evaluation questions in four key areas: quality, access, cost, and patient and clinician perceptions and satisfaction. Depending on the application and clinical problem, the setting and patient population, the objectives of the program, and other factors, evaluations will differ in the outcomes of greatest interest and relevance. As stressed in Chapter 6, the earlier and more precisely evaluation objectives and questions are identified, the more likely it is that the program to be evaluated can be designed and implemented in ways that will help provide useful and credible answers. Although the questions about quality, access, cost, and patient and clinician perceptions are presented sequentially above, their interrelationships also warrant attention. For example, the timeliness of care—an element of access as defined here—may have important consequences for quality through earlier detection and better management of clinical problems. Similarly, economic analyses of telemedicine do not simply examine costs but attempt to relate the costs of an application to its benefits and to suggest bases for judging whether the benefits are worth the costs in comparison to other
OCR for page 193
--> alternatives. Judgments are typically based on a balancing of objectives that is contingent on a given evaluation's mix of effects on quality, access, and cost. For evaluations that are beyond the "test of concept" or formative phase, a central question will often be: What do the quality, access, cost, and other results suggest about whether and how the telemedicine program can be sustained beyond the evaluation stage?
Representative terms from entire chapter: