9
Methods of Quality Assessment and Assurance

A quality assurance program can have several purposes, each of which may be emphasized to varying degrees. In working toward its goals, a quality assurance program can try to prevent problems from occurring, detect and correct those problems that do occur, and encourage higher standards of care. It can attempt to remove or rehabilitate poor practitioners and providers, improve the average level of practice, reward excellence, or use some combination of those goals. The methods used in a quality assurance program may be as sharply focused as finding and reacting to isolated events involving a single patient and practitioner, such as a surgical mishap. They may be as broad as conducting continuing education, disseminating practice guidelines, initiating institution-wide “continuous improvement,” designing management information systems with uniform clinical data elements, and conducting research on effectiveness at a national level. Ideally, the choice of methods for a strategy for Medicare quality review and assurance should be based on an assessment of the burdens of harm from different quality problems (Chapter 7), an understanding of important features of health delivery systems that affect our current ability to measure care and effect change (Chapter 8), and the strengths and limitations of major methods of quality assessment.

INTRODUCTION

In this chapter we describe selected methods of quality assessment and assurance and discuss how well they meet the criteria for successful quality assurance efforts outlined in Chapter 2. (Chapter 6 in Volume II goes into more detail and differentiates methods by purpose, agent, and setting. It includes methods in use, methods derived from research studies, and meth-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I 9 Methods of Quality Assessment and Assurance A quality assurance program can have several purposes, each of which may be emphasized to varying degrees. In working toward its goals, a quality assurance program can try to prevent problems from occurring, detect and correct those problems that do occur, and encourage higher standards of care. It can attempt to remove or rehabilitate poor practitioners and providers, improve the average level of practice, reward excellence, or use some combination of those goals. The methods used in a quality assurance program may be as sharply focused as finding and reacting to isolated events involving a single patient and practitioner, such as a surgical mishap. They may be as broad as conducting continuing education, disseminating practice guidelines, initiating institution-wide “continuous improvement,” designing management information systems with uniform clinical data elements, and conducting research on effectiveness at a national level. Ideally, the choice of methods for a strategy for Medicare quality review and assurance should be based on an assessment of the burdens of harm from different quality problems (Chapter 7), an understanding of important features of health delivery systems that affect our current ability to measure care and effect change (Chapter 8), and the strengths and limitations of major methods of quality assessment. INTRODUCTION In this chapter we describe selected methods of quality assessment and assurance and discuss how well they meet the criteria for successful quality assurance efforts outlined in Chapter 2. (Chapter 6 in Volume II goes into more detail and differentiates methods by purpose, agent, and setting. It includes methods in use, methods derived from research studies, and meth-

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I ods described during site visits.) We have here focused on methods for preventing, detecting, and correcting quality problems for three settings of care: hospital-based care, office-based care, and home health care. Some approaches are directed at individuals; others are directed at institutions. Some are used primarily by health care organizations; others principally by external regulatory groups. Some have been developed for research projects; others have evolved in clinical and administrative departments in health care facilities. Methods of preventing problems described in this chapter include accreditation and licensure for health care organizations and licensure and board certification for individual practitioners. Other methods include patient management guidelines and clinical reminders. Approaches in detecting problems include analysis of administrative data bases, retrospective chart review, nonintrusive outcome measures, generic screening for adverse events using medical records, clinical indicators, and assessments of patient outcomes (such as health status and satisfaction). Detection methods based on aggregate data include the use of administrative data bases for analyzing outcomes such as mortality and complication rates. Individual-case sources of information about quality include autopsy, case conferences, and patient complaints. Our discussion of methods of correcting problems emphasizes factors that are thought to impede or enhance the effectiveness of interventions intended to change behavior. Some interventions may be quite informal, for example telephone conversations with individual practitioners. Others, such as financial sanctions, are more formal. Interventions based on poor practice patterns include remedial education, restrictions on practice, and penalties. The final section of this chapter reviews current thinking about the advantages and disadvantages of educational approaches, incentives (including rewards), and disincentives (including penalties) for individual physicians both for improving average and outlier practitioners. Because there has been little evaluation of methods of intervention, this section does not lend itself well to discussions of known strengths and limitations. Therefore, we confine our discussion to a description of factors and variables that are thought to influence ways of changing behavior in health care organizations. Important Attributes of Methods When considering the strengths and limitations of quality assurance methods, one should consider several features. Among these are reliability and validity. Reliability refers to consistency in results, that is the degree to which measures of quality agree either when repeated over time or when applied by different people or in different settings. To assess the reliability of a credentialing system, for instance, one might evaluate the consistency

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I of information obtained about applicants for hospital privileges or how often review committees agree in their recommendations. To determine the reliability of a method to detect quality problems, one might calculate how often chart reviewers identify the same adverse events. Reliability in correcting problems is more theoretical, but one might envision measuring whether comparable corrective action plans (e.g., continuing education courses or reading designated literature) consistently improve tested knowledge. Validity in this context refers to whether a method acts as intended. For one to consider board specialization a valid method of ensuring high quality, for instance, one would look for proof that those who are board certified provide a demonstrably higher quality of care than those who are not. Likewise, the validity of an outcome measure of quality could be assessed by determining whether patients with poor outcomes received deficient care and whether the deficiency produced the poor outcome. To demonstrate the validity of a method of correcting problems one would look for evidence that a specific intervention brought about the desired change. For instance, required consultation with a colleague before treating certain cases should result in fewer problem cases. Assessment methods may be valid in that they detect real problems in quality, but even valid tools may be inefficient (if they detect a great many events that are not quality problems) or ineffective (if they fail to detect many important quality problems). For virtually no method of assessment do we know the effect on provider behavior or the effect of practitioner change in behavior on patient outcomes. These are the ultimate tests of validity. Although methods may be accurate at identifying problems, they are valuable for quality assurance only if, or to the extent that, identification leads to changed behavior and to improved patient outcomes. Measures of these two demanding but critical factors are almost nonexistent, and this shortfall must temper any recommendations for specific approaches. Assessment methods have important attributes other than reliability and validity. These include their practicality, ease of application, lack of unintended negative effects, inclusion of patient views and preferences, and ability to detect poor technical quality, overuse, and underuse. It is also useful to consider whether various methods of assessment provide timely information to improve performance and whether they yield information that accords with ideas about how professionals learn. PREVENTION OF PROBLEMS Accreditation and Licensure for Organizations Hospitals, ambulatory care facilities, managed care organizations, and home health agencies can be accredited on a voluntary basis by the Joint Commission on Accreditation of Healthcare Organizations (Joint Commis-

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I sion). Approximately 77 percent of the approximately 7,000 Medicare participating hospitals have received Joint Commission accreditation. The remaining 1,600 hospitals that are not accredited are, for the most part, small rural institutions with 50 or fewer beds (see Chapter 5 in this volume and Chapter 7, Volume II for an extensive discussion of the evolution of the Joint Commission’s accreditation process). The accreditation manuals for each type of facility are designed for hospital use in self-assessment and for the Joint Commission to use for on-site surveys. For hospitals in “substantial compliance” such a survey occurs every three years. Scheduled at least four weeks in advance, the survey is conducted by a physician, nurse, and administrative surveyor over a three-day period using explicit scoring guidelines. After a concluding educational exit interview, the facility may receive full accreditation or may be notified that accreditation is contingent on its carrying out a plan of correction. A hospital with contingencies may submit written evidence or may undergo a return site visit. It may then may be fully accredited or, in due course, nonaccredited. In 1981, the Joint Commission replaced their prescriptive, structure-oriented standards and numerical audit requirements with a standard requiring ongoing, facility-wide monitoring of care. Monitoring was intended to permit the identification of problems and ways to improve the delivery of care and to promote solutions to any problems identified. Nevertheless, structural standards designed to prevent problems and to ensure the capacity of the hospital to operate safely are still in effect. Three such areas of emphasis include (1) a standard specifying that the governing body is to hold the medical staff responsible for establishing quality assurance mechanisms, (2) medical staff standards requiring regular review, evaluation and monitoring of the quality and appropriateness of services provided by the medical staff, and (3) a standard calling for the establishment of coordinated hospital-wide quality assurance activities. In addition to the Joint Commission, accreditation for ambulatory facilities can also be sought on a voluntary basis from the Accreditation Association for Ambulatory Health Care (AAAHC), for HMOs from the National Committee on Quality Assurance (NCQA), and for home health agencies from the National League for Nursing (NLN) through its Community Health Accreditation Program. To date nonhospital providers have sought accreditation infrequently. These accrediting organizations, however, have become increasingly active, and some states, such as Pennsylvania and Kansas, have determined that these accrediting groups are acceptable to provide external review for HMOs. Strengths Standards for accreditation are publicly available. If the standards are unambiguous, and if reviewers are consistent in applying them, then infor-

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I mation on accreditation status provides comparable information on health facilities. If accreditation standards were more widely accepted by external regulators (e.g., eligibility for third-party payers or state licensure boards), this might reduce overlapping requirements. Because accreditation is conferred voluntarily by a body representing the kind of facilities being reviewed, it represents a quasi-internal process that is, at least in theory, responsive to member organizations, yet accountable to the industry as a whole and to the public. Depending on the perceived value of accreditation and the stringency of the review process, the organization may make substantial efforts to comply with standards. A variable rating system in accreditation could recognize outstanding performance. Limitations Accreditation is evidence that certain quality assurance efforts such as requiring specific credentials, staffing policies, or grievance procedures are being pursued. However, unless the accreditation process is itself evaluated and found to be based on reliable and valid methods, it cannot be relied on as a method of ensuring quality, and it may divert resources from more effective approaches. Accreditation can be very expensive and cumbersome, and this may discourage its voluntary use. Credentials, Licensure, and Specialty Certification1 The examination of credentials is regularly used as a method of assuring high quality. The process is used (1) by state boards in granting licenses to practice, (2) by specialty and subspecialty boards in granting certification, (3) by hospital committees in reviewing applications to the medical staff, and (4) by payers in determining eligibility to be paid for services (Chassin et al., 1989a). The decisions of these groups may themselves constitute credentials. Licensure and board certification are particularly important. Physician Licensure Each state has statutes regulating the practice of medicine through physician licensure. Most of these laws define the practice of medicine and prohibit those who are unlicensed from engaging in it. State medical practice acts are administered by state boards of medical examiners. Those who apply for licensure are judged on the basis of their education, postgraduate training, experience, results on licensing examinations, and moral character. Applicants for licensure must be graduates of schools of medicine or osteopathy that are accredited by the Liaison Committee on Medical Education, with special provisions being made for graduates of foreign medical schools. A postgraduate internship of one year is required by approximately three-quarters of the states, and applicants must

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I successfully pass a licensing examination. All states currently use the Federation Licensing Examination (FLEX), prepared by the National Board of Medical Examiners (NBME) for the Federation of State Medical Boards. Most states will also accept the so-called National Boards, prepared by NBME or by the National Board of Examiners for Osteopathic Physicians and Surgeons. These examinations are administered in three stages as the student progresses through his or her education (Havighurst, 1988). Some states have reciprocity agreements, whereby licenses granted by one state are recognized in another state. Some states require that applicants go through the procedures specified in their medical practice acts regardless of whether they are already licensed elsewhere (Havighurst, 1988). Strengths. Licensure provides a minimum standard of quality for the individual health care practitioner. It does not meet any of the other goals of quality assurance listed in Chapter 2. Limitations. The authority to practice medicine, once licensure has been obtained, is legally constrained only by criminal and medical malpractice law. Physician licensure is generally for life, and where licenses must be renewed, no new demonstration of competence is required. Many states have instituted certain continuing medical education (CME) requirements as a condition of license renewal. Attendance at approved CME is sufficient to meet the statutory mandate; those attending need not take and pass any examinations or show any other sign of accomplishment (Davis et al., 1984; Havighurst, 1988). The physician’s license is also unlimited in scope, permitting the physician to engage in areas of practice for which he or she may have little training (Havighurst, 1988). This lack of limits stands in sharp contrast to the strict limitations placed on other health professionals subject to licensure. Licensure in no way guarantees competence across the wide range of medical practice or over time. Specialty Certification and Recertification The American Board of Medical Specialties (ABMS) recognizes 23 specialty boards that certify physicians as medical specialists in carefully delineated areas of practice. Several other entities also certify physicians, but because the ABMS system is so dominant, “board certification” is generally understood to mean certification in a medical specialty by a board recognized by ABMS (Havighurst and King, 1983). For a board to achieve accreditation status, it must be sponsored both by a professional group, such as a specialty society, and the appropriate scien-

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I tific section of the American Medical Association (AMA). All the boards are evaluated for recognition according to the ABMS “Essentials for Approval of Examining Boards in Medical Specialties.” Each board thus requires similar levels of training and experience. The residency program must be approved by the Accreditation Council for Graduate Medical Education (ACGME), an organization composed of members of the ABMS, the AMA, and other concerned organizations. Together with appropriate specialty boards, the ACGME develops accreditation standards for each specialty residency program. These are regularly modified in conjunction with changing specialty board requirements and must be approved by the AMA’s Council on Medical Education (Havighurst and King, 1983). Ultimately, candidates must also pass comprehensive examinations administered by the specialty board. Candidates for certification must receive and complete specialty training in an approved graduate medical program, the length and extent of which vary somewhat among the specialties. A majority of physicians in the United States identify themselves as specialists, but only about one-half are actually certified by an ABMS board. The number seeking certification has grown and continues to grow rapidly. Almost all physicians newly entering practice now seek some sort of certification. Of those who designate themselves as specialists, an increasing number are actually board certified. Strengths. Certification in a medical specialty is widely accepted as an indication that certified physicians possess a superior level of training and skill in their area of specialization. Information on certification is readily available from such sources as county medical societies, the ABMS, AMA, American Medical Directory, and the AMA Physician Masterfile. Certification has been endorsed by the Joint Commission as an “excellent benchmark for the delineation of clinical privileges” (Joint Commission, 1989, p. 106). Limitations. Ramsey and his co-workers (1989) compared the performance of board-certified and noncertified practitioners in internal medicine using measures of knowledge, judgment, communications skills, and humanistic qualities. Scores of board-certified internists on a written examination were significantly higher than those of noncertified internists, but ratings by professional associates, patient satisfaction scores, and performance in the care of common illnesses (as measured by medical record review) showed few differences. There were modest differences in preventive care and patient outcomes that favored the certified physicians. The Office of Technology Assessment (OTA, 1988) reviewed 13 studies on the adequacy of physician specialization as a measure of quality and found little evidence that board certification accurately predicts high-qual-

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I ity care. Studies that use process criteria tend to show that specialists trained in their area of practice (the modal specialist) provide higher quality than those who have not been so trained, but this higher quality of process has not been linked to superior patient outcomes. Nor has a relationship been established between specialist care and patient satisfaction (Chassin et al., 1989a). Even if superior performance is associated with specialty training or board certification in one area, such evidence would not necessarily be generalizable to other specialties, diagnoses, or procedures (OTA, 1988). In the past, boards granted certification for unlimited periods. There has been a move over the past 10 years toward recertification requirements, so that 15 of the 23 specialty boards have now adopted or decided to adopt time-limited certification with intervals between revaluations ranging from six to 10 years. One board offers voluntary recertification, and seven specialty boards have no recertification procedures (Havighurst and King, 1983). Some experts have recommended that the certification and recertification processes should shift from one that is knowledge-based to a more “performance-based” assessment that reflects actual practice such as a review of a sample of records or observation. This, it is believed, will reflect more accurately the physician’s practice and thereby increase the validity of board certification (Havighurst and King, 1983). Appropriateness and Patient Management Guidelines2 In medicine, and particularly in organized ambulatory care practices, guidelines serve many purposes, but they are intended primarily for education. They may specify appropriate and inappropriate uses of medical interventions, act as reminders for relatively simple tasks (e.g., provision of vaccinations), or serve as shorthand reminders for complex clinical decision making. For this last use they are sometimes called patient care algorithms. In all these applications, practice guidelines can help to forestall the occurrence of problems in patient care. In modified formats, they can also be used for retrospective quality review. Numerous groups, including medical specialty groups, have formulated such appropriateness and patient management guidelines. They are also frequently developed by interested clinicians within health care facilities and by health services researchers. They take on a variety of formats, depending on their highly individualized purpose. Strengths Patient management guidelines can be viewed as the translation of a medical text into a focused, often graphic, and sometimes computerized format. The use of branched reasoning and flow diagrams allows for great

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I complexity and logically complete presentations. Well-constructed guidelines can allow patient preferences to be elicited and taken into account. Limitations Few (perhaps no) algorithms in use today are based entirely on scientific evidence of effectiveness. Generally, some or all of the available evidence is augmented by the clinical experience of the formulators. Many guidelines are nothing more than lists of ambiguous or vague statements about appropriate care that lack any guidance on their implementation. Guidelines are frequently put into practice with no or only haphazard pretesting or evaluation. Often they lack provisions for updating or modification based on new knowledge, on their usefulness to clinicians, or on their impact on care. Guidelines may be of limited use for patients with multiple chronic conditions because the formats rapidly become too complex for easy reference. Clinical Reminder Systems Clinical reminder systems are computerized methods used in some managed care plans, clinics, and office practices to remind clinicians of preventive tests that should be performed, of laboratory monitoring that is due for patients with chronic disease, and of potential drug interactions (McDonald, 1976; Barnett et al., 1978, 1983; McDonald et al., 1984; Tierney et al., 1986). For instance, when printing out a list of scheduled patients a reminder system may use an age, sex, and risk-adjusted algorithm to specify screening tests or laboratory monitoring for individual patients. Other reminders may be used interactively to warn of possible drug interactions or to query the physician and advise on appropriate antibiotic prescriptions. Strengths A computerized reminder system can alert a practitioner to patient needs and potential problems at the time patient care is provided, making it a truly concurrent quality assurance system. Such systems can be tailored to individual risk factors and previous medical history. Clinical reminder systems can incorporate probabilities of various outcomes and references to journal citations for further information and can be updated frequently. Their value in improving clinical process has been well demonstrated. Limitations These systems require readily available computer equipment, a rapid response time, and enough practitioner familiarity with the software to be

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I feasible for use during practice hours. Their relationship to improved patient outcomes remains unevaluated. DETECTION OF PROBLEMS Use of Large Administrative Data Sets3 Large data sets refer to claims-based administrative data bases such as those for Medicare Part A and Part B claims. Roos et al. (1989) distinguish three types of data bases and the kinds of studies that are feasible with each. A Level 1 data base contains only hospital discharge abstracts and will permit aggregate studies of, for instance, in-hospital mortality rates and lengths of stay, by geographic region or over time. A Level 2 data base contains, in addition, unique patient-identifying numbers. It can be used to study, for instance, short-term readmissions and volume and outcome relationships at a hospital-specific level. A Level 3 data base (the most comprehensive) will also have information from health program enrollment files, including when eligibility begins and ends. This data base permits the highest quality longitudinal studies, short- and long-term outcomes studies, and population-based (system-wide coverage) studies. Studies can include outcomes for intervention-free individuals and for poor outcomes or other complications that are not recorded as part of the hospital stay. Weiner et al. (1989) have provided examples of quality-of-care indicators that might be developed from ambulatory care data bases. These include system measures such as the rate of hospitalizations, of readmissions, and “avoidable disease” or disease first diagnosed at an advanced stage. Other examples include (1) preventive-care indicators, such as the percentage of eligible persons receiving a recommended number of periodic screening tests or exams within a given time period and the documented incidence of newly diagnosed disease versus the expected incidence; (2) diagnostic indicators, such as the number or proportion of patients who receive unnecessary diagnostic tests or procedures; and (3) treatment indicators, such as the percentage of patients with a given diagnosis who receive the appropriate medication, the percentage of patients undergoing ambulatory surgical procedures who experience complications including hospitalizations, and the percentage of all visits to the patient’s primary provider. Strengths All large administrative data bases have several theoretical advantages for quality assessment. First, the accuracy of various types of data (e.g., medications, previous hospitalizations, and numbers of physician visits and medical conditions treated) is unaffected by errors in patient or practitioner

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I recall. Second, the use of these data bases is unobtrusive; patient consent for individual studies is not required, and no bias is introduced from individuals’ knowing that they are being studied. Unobtrusiveness may also contribute to their acceptability to practitioners and health care facilities. Third, assessors can create and test different statistical models or approaches to risk adjustment. They can also alter study designs or use several different study designs to test findings; for instance, they can use both cohort and case-control designs to examine the effect of different intervention periods. Fourth, the same files can be applied in different ways, for instance, tracking outcomes of surgery, computer modeling of readmission, examining changes in complication rates over time, or studying outcomes of care for patients in different geographic areas. Fifth, investigators can accurately assess risks as well as benefits associated with treatment, especially for areas of medical uncertainty. The data bases can provide inputs for clinical decision making by allowing calculation of the probability of complications of treatment or of mortality at varying lengths of time after treatment. Sixth, the use of administrative data bases is relatively inexpensive in comparison to methods that require large-scale primary data collection. An important strength of Level 3 data bases is that they contain population data, and thus they permit some assessment of population access and outcomes. Comparative studies should be able to identify possible areas of underuse. Limitations Administrative data bases have considerable drawbacks for quality assessment. First, data bases may exclude important information such as certain events, information on location of service and provider, or costs, and may assemble the elements in ways that complicate linkage to other files. Second, the precision of the coding schemes (primarily the ICD-9-CM4 and CPT systems) is of great concern, particularly for medical conditions that encompass a broad range of clinical severity and contain important clinical subgroups, such as congestive heart failure and diabetes mellitus. The ICD-9-CM coding system does not distinguish procedures performed on the right side of the body from those performed on the left. For this reason, a data base with ICD-9-CM codes will not allow a reviewer to determine whether a second hip replacement, for instance, is a reoperation or a new operation. Of equal concern is the poor ability of data bases to distinguish the order of events during a single episode of care (e.g., a pulmonary embolus that was present at the time of admission versus one that developed after surgery). Although administrative data bases record the occurrence of events such as x-rays and diagnostic tests, the results of these

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I both scarce and crucial, and it points to the need for educational reform in building the capability of professionals to act with confidence. We return to this point in Chapter 11. Changing Practitioner Behavior Changing professional behavior in the long run requires persuading the professional of the need to change. The most persuasive data are those that are credible, complete, timely, and pertinent to an individual’s practice. Educational Approaches Good education and training are regarded by the health care professions to be the foundation for good practice. Beyond a lengthy and rigorous initial experience, a lifetime characterized by continuous learning is the ideal. Professionals hold educational interventions to be the preferred method for obtaining behavior change, especially if the method does not single out individuals. Although education may be the most useful first approach to changing professionals’ behavior, the kinds of problems or practitioners that are most amenable to educational interventions are not clear. Respected clinicians providing feedback in relatively informal settings may be the most effective agents for change. Lohr et al. (1981, p. vii), in referring to technology diffusion, state that “in general, professional colleagues are considered more potent legitimizing agents than any other single influence, and the most effective force for physicians’ adoption of medical innovations is professional, face-to-face contact with recognized peers.” The same can also be said for adoption of new practice behaviors (Eisenberg, 1986; Schroeder, 1987; Davidoff et al., 1989). After reviewing the literature, Eisenberg (1986) reported that some studies show continuing education to be effective, others show it to be ineffective, and many other studies are inconclusive because of deficiencies in their methods or ambiguities in their findings. Guided by this picture and principles of adult education, he surmised that successful approaches to modifying physician behavior should have the following features. First, a practitioner should have accepted (presumably on the basis of valid evidence) that he or she has a need to learn. Second, the educational content should be specific to the need already identified. Third, education should be conducted face-to-face. Fourth, if possible, it should be conducted one-to-one. Fifth, it should be conducted by an “influential” person—a person the practitioner trusts and respects. Presumably, many educational efforts fail because one or more of these conditions are not met. Education is called for when insufficient knowledge or skill are at least

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I in part the reason for deficient care. How often ignorance and ineptitude are a cause of poor care, and to what degree, is not known, but their contribution can be expected to vary considerably from setting to setting. McDonald and his co-workers (McDonald, 1976; McDonald et al., 1984; Tierney et al., 1986) concluded after a controlled study that errors in ambulatory care occurred more often because of overload of tasks and information than because of lack of knowledge. To the extent that mistakes are caused by lack of access to current knowledge, online computer-aided management, warning, and reminder systems may hold promise for affecting physician behavior. We do not, however, know a great deal about the circumstances in which these systems are used or are useful, and we recognize that available tools and attitudes may be changing dramatically as the professionally trained population becomes increasingly computer literate. The conditions detailed above underscore the importance of establishing a clearly defined link between quality monitoring and continuing education, as typified by the “bi-cycle” model proposed many years ago by Brown and Uhl (1970) and repeatedly advanced since.7 Donabedian (1989) believes that the relative effectiveness of alternative ways of linking monitoring to education, and of conducting the educational effort itself, should be high on the agenda of research on the effectiveness of quality assurance through monitoring. Incentives and Disincentives Professional behavior can also be changed by directive. The net effect of such an approach on the health of patients and the morale of professionals has not been explored. Likewise, little is known about the effects of positive versus negative incentives or ways to link informal professional incentives to quality assurance activities. Feedback and education are meant to appeal to internalized values and to mobilize the personal resources of practitioners. Various factors in the environment may well enhance or diminish these efforts. Much depends on the implicit expectations and informal understandings of medical colleagues, but a great deal may also depend on the structure of a more formalized system of rewards and penalties. The relative impotence of quality assurance efforts in directly modifying practitioner behavior may be attributable to the absence of a clearly defined, consistently operative link between the results of monitoring and the career prospects of practitioners. Thus, the formal system of incentives and disincentives deserves particular attention in any analysis of effectiveness. Incentives are commonly regarded as rewards and disincentives as penalties, but not receiving a reward when a system of rewards has been instituted can be a disincentive, and not being penalized when a system of

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I penalties has been established can be an incentive. A system based on recognizing and rewarding, rather than ferreting out and punishing error, would very likely differ in its acceptability to professionals and perhaps also in its effectiveness. One might also hypothesize that a system having features of both approaches could be the most effective. Rewards and penalties might also be distinguishable by whether they are professional, financial, or both, and by whether they are generalized or particularized. For example, a promotion connotes both professional and financial rewards that are not necessarily related to any particular meritorious action; rather, a general pattern of laudable behavior is being recognized. By contrast, withholding payment for an unapproved procedure is a particularized penalty. A proposal that physicians be awarded part of the savings that accrue from their maintaining a lower-than-average length of hospital stay is an positive financial reward related to a pattern of behavior. The traditional incentives of the professional culture such as career advancement, salary, risk-sharing and bonus arrangements, and esteem of colleagues are more individualized. The magnitude of the rewards or penalties might affect their impact. This may be especially true if penalties are matched to the seriousness of the offense (Vladeck, 1988), to the credibility and legitimacy of the judgments that lead to them, to the presence of procedural and legal safeguards against arbitrary action, and to evidence that penalties are used fairly and consistently. Eisenberg (1986) reached two conclusions about the use of penalties. First, penalties do modify physician behaviors. Second, they are deeply resented and may have unexpected or unwanted consequences—a “backlash,” as he calls it. Such backlash undoubtedly would create political and administrative problems, but its effect on the quality of care is not clear. Changing Systems and Organizations Berwick (1989) has stated that “flaws come more often from impaired systems than from impaired people.” Many quality assurance professionals concurred with this view during our site visits, and this viewpoint has led many to focus not only on average practice rather than outliers but on organizational factors that affect quality and on ways of changing them. Despite a voluminous literature on planned organizational change (Johnson, 1989), its principles have not been extensively applied to the health care organization. In particular, the implementation of continuous improvement models (discussed in Chapter 2) has not yet progressed far enough to demonstrate their effectiveness in modifying clinical and organizational behavior. In a discussion of barriers limiting implementation of quality assurance

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I programs, Luke and Boss (1981, p. 148) stated that the ineffectiveness of educational strategies results from a “failure to conceptualize quality assurance primarily as a problem of organizational and behavioral change.” They further asserted that “…the real barriers to quality assurance are not the impediments to data acquisition and analysis but the points of resistance to change within health institutions.” They identified 10 barriers to change that must be recognized and addressed if interventions are to be effective. These barriers are: (1) autonomy expectations of health professionals; (2) collective benefits of stability; (3) calculated opposition to change; (4) programmed behavior; (5) tunnel vision; (6) resource limitations; (7) sunk costs; (8) accumulations of official constraints on behavior; (9) unofficial and unplanned constraints on behavior; and (10) interorganizational agreements. Many organizational factors may influence the effectiveness of quality assurance efforts. Particularly important may be the collaborative nature of medical practice and its dependence on institutional (mainly hospital) support (Knaus et al., 1986). Physicians, as a group, may be able to control only a part of what is done for patients; any one physician can control even less. Palmer et al. (1985) found that physicians in ambulatory practices were more likely to improve their care in response to failures revealed by monitoring when the change to be made was more directly under their own control. Organizational Factors Influencing the Form and Effectiveness of Quality Assurance Quality monitoring is most likely to occur when care is provided in or through institutions or organized programs. Thus, the forms it takes as well as its effectiveness can be expected to reflect the characteristics of these organizations. Variations in quality among institutions are probably at least partly attributable to the fact some institutions have better developed and functioning quality monitoring systems than others. When care is made more “visible” to colleagues (for example, through sharing responsibility for care, consultation, teaching rounds, clinical conferences, and the like), then the quality of that care is likely to be higher (Neuhauser, 1971; Shortell et al., 1976). Other advantages in quality have been attributed to controls over recruitment and staffing, to equipment and material resources, to direct supervision of professional work, and to more subtle attributes such as coordination, communication, and tightness of organizational control (Georgopoulos and Mann, 1962; Scott et al., 1976; Flood and Scott, 1978). The role of formal monitoring mechanisms, as a separable organizational feature, in influencing the quality of care provided is as yet unexplored. Several features of the organizational environment might influence the

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I implementation and effectiveness of quality monitoring. These include ideology, leadership, and baseline performance. Ideology The importance accorded to quality, both in absolute terms and relative to competing objectives (particularly cost containment), may be an important determinant of the effectiveness of quality monitoring. The sources of concern for quality may derive from the perception of a social responsibility, a professional imperative, a prudent yielding to coercion, or the prospect of a profitable response to market forces. All these factors may also bear on the effectiveness of quality assurance, particularly if all motivations impel in the same direction (Donabedian, 1989). The relative importance given to technical care as compared to the interpersonal process may also influence the form and success of quality assurance. To some extent this choice is ideological; it reflects or is influenced by the views of the organization’s leadership, the values and traditions of the major constituencies to which the organization is answerable, and the functions an organization serves. For instance, the quality of technical care is likely to be the dominant concern of a major teaching center. In contrast, a long-term care facility under religious auspices is impelled, in part, by the values of its sponsors to emphasize the amenities and the interpersonal aspects of care. In the first instance, the interpersonal process may be at risk, whereas in the second situation technical care may be in jeopardy. Leadership Leadership as a component of the organizational environment is least amenable to control and often dependent on serendipity. Donabedian (1989) points out that whether leadership is provided by a member of the governing board or a senior administrator, he or she must be a trusted and respected colleague who is directly involved in the program. Although the evidence is weak, it seems to suggest that quality monitoring is more effective in altering physician behavior when clinical leaders participate in it and alter their own behavior in response to its findings (Palmer et al., 1988). Baseline Performance The baseline level of clinical performance that characterizes an organization may be an important determinant of the perceived need for quality monitoring and may affect both the design of the monitoring enterprise and its effectiveness. In this regard, the shared perceptions of the level of performance may be as important as the actual level, more objectively assessed.

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I When the actual or perceived level of performance is exceedingly high, formal monitoring may seem redundant. When such review is externally imposed, it may be resented and at best perfunctorily performed. Major teaching institutions may be particularly prone to these behaviors. When actual performance is at an uncommonly low level, quality monitoring may be regarded as a threat. Where monitoring is introduced, it may be ineffective because poor practice is usually the consequence of deepseated organizational pathology. Disbelief, defensiveness, and low expectations may lead to weak internal criteria and standards that fail to challenge. Even when external criteria and standards are held out as an example, they are likely to be countered by a host of arguments seeking to show why the criteria do not apply to the peculiarities of the local situation. SUMMARY The variety of techniques for quality review and assurance used in the United States is enormous and rich. Some activities are intended to prevent quality problems; some are designed to detect them; and still others are efforts to correct problems once they are identified. This chapter has provided a highly selective sampler of methods for quality review and assurance. It illustrates the considerable range of efforts beyond those of the federal PRO program and shows there is much to learn from the professional and provider communities’ own efforts. All these methods have strengths and limitations, which we have cited here. We have, however, taken no position on the quality of those efforts. The techniques and approaches described are not necessarily the best, although some may well be state-of-the-art. Our review here of quality assessment methods currently in use (and the descriptions of methods in various settings in Volume II) reveals inadequacies, in particular, the weak focus on the continuum of care across multiple providers for patients, especially those with chronic illness, who move from one setting of care to another. Review tends to focus on single events and single settings rather than episodes of care. By and large, review techniques look at the technical quality of care and specifically at the physician component of decision making. For those receiving care, undertreatment, and to a lesser degree, overuse may be identified. The quality of interpersonal care and the use of patient outcomes in evaluating care are only now beginning to be incorporated into quality review efforts. This chapter highlights the need for a much better understanding of how effectively to bring about change in provider performance and practice patterns. Available intervention methods include information feedback on performance, financial incentives and disincentives and penalties, and organizational development and change techniques. The emphasis is often on

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I overcoming deficiencies in knowledge or skills, indifference, or impairment. The limited repertoire of practitioner-oriented interventions (e.g., education, exhortation, surveillance, and sanctions) is insufficient to address what may be far more complex reasons for inappropriate or poor technical care. These include the nature of medical work, its collaborative nature, the lack of physician control over many aspects of health services, and the organizational environment of practice. Most quality assurance professionals have at one time or another become discouraged at the results of providing information on practice and hoping that a change in behavior would result. The likelihood of change may be linked to the seriousness of the deficiency, the relevance of the practitioner’s behavior to that deficiency, and the ease of changing behavior. The most promising strategies for changing individual behavior are likely to be those that act in concert with the training and practice characteristics of doctors in conforming to the medical culture, that help the already good practitioners as well as the outliers, and that recognize the limited ability of any given practitioner to change the delivery system. Leadership and commitment in concert with other organizational goals may be the most important factors in organizational change. Organization and funding of quality assurance programs likely influence their effectiveness. The baseline level of performance and the external regulatory environment are also likely to influence an organization’s response to purported deficiencies. NOTES 1.   Some of the discussion of licensure and board certification has been drawn from a paper, “Medicare Quality Assurance Mechanisms and the Law,” by A.H.Smith and M.J.Mehlman prepared for this study and referred to hereafter as Smith and Mehlman, 1989. 2.   See Chapter 10 for a more extended discussion of appropriateness (practice) guidelines, patient management criteria sets, and algorithms. 3.   Much of this discussion has been drawn from a paper by L.L.Roos, N.P.Roos, E.S.Fisher, and T.A.Bubolz prepared for this study and hereafter referred to as Roos et al., 1989. Some of the material will appear in Roos et al. (forthcoming) and Roos, 1989. 4.   International Classification of Diseases, ninth revision, clinical modification. The Medicare hospital (Part A) files for instance, use ICD-9-CM codes, and diagnosis-related groups (DRGs) are based on them as well. 5.   The development and testing of coding dictionaries that manage multiple medical terms such as those in use with the Computer Stored Ambulatory Record (COSTAR) have demonstrated promise in improving coding accuracy in ambulatory care. 6.   Calculations done from data supplied by HCFA for the Second Scope of Work through February 1989 of review of more than 6.1 million records show the

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I     following: Of all records reviewed, nearly 24 percent failed at least one screen. Of those that failed at least one screen, physician advisors confirmed quality problems in 30 percent, or about 7 percent of all records reviewed (HCFA, 1989). 7.   The bi-cycle model is one in which problems are identified, analyzed, attacked, and then reevaluated. In the case of problems caused by lack of practitioner knowledge or skills the methods of correction should involve highly targeted, pertinent continuing education. REFERENCES Barnett, G.O., Winickoff, R.N., Dorsey, J.L., et al. Quality Assurance Through Automated Monitoring and Concurrent Feedback Using a Computer-Based Medical Information System. Medical Care 16:962–970, 1978. Barnett, G.O., Winickoff, R.N., Morgan, M.M., et al. A Computer-Based Monitoring System for Follow-up of Elevated Blood Pressure. Medical Care 21:400–409, 1983. Berwick, D.M. The Ideal of Continuous Improvement. Reply to Letters to the Editor. New England Journal of Medicine 320:1425, 1989. Brown, C.R. Jr. and Uhl, H.S. Mandatory Medical Education: Sense or Nonsense? Journal of the American Medical Association 213:1660–1668, 1970. Borgiel, A.E. Assessing the Quality of Care in Family Physicians’ Practices by the College of Family Physicians of Canada. Pp. 63–72 in Quality of Care and Technology Assessment. Lohr, K.N. and Rettig, R.A. eds. Washington, D.C.: National Academy Press, 1988. Borgiel, A.E., Williams, J.I., Anderson, G.M., et al. Assessing the Quality of Care in Family Physicians’ practices. Canadian Family Physician 31:853–862, 1985. Chassin, M.R., Kosecoff, J., and Dubois, R. Value-Managed Health Care Purchasing. An Employer’s Guidebook Series. Volume II: Health Care Quality Assessment. Chicago, Ill.: Midwest Business Group on Health, 1989a. Chassin, M.R., Park, R.W., Lohr, K.N., et al. Differences Among Hospitals in Medicare Patient Mortality. Health Service Research 24:1–31, 1989b. Craddick, J.W. and Bader, B.S. Medical Management Analysis: A Systematic Approach to Quality Assurance and Risk Management. Vol. I. Auburn, Calif.: J.W.Craddick, 1983. Davidoff, F., Goodspeed, R., and Clive, J. Changing Test Ordering Behavior: A Randomized Controlled Trial Comparing Probabilistic Reasoning With Cost-Containment Education. Medical Care 27:45–58, 1989. Davis, D., Haynes, R.B., Chambers, L., et al. The Impact of CME: A Methodologic Review of the Continuing Medical Education Literature. Evaluation and the Health Professions 7:251–284, 1984. Davies, A.R. and Ware, J.E. Involving Consumers in Quality of Care Assessment. Health Affairs 7:33–48, Spring 1988. Donabedian, A. Reflections on the Effectiveness of Quality Assurance. Paper prepared for the Institute of Medicine Study to Design a Strategy for Quality Review and Assurance in Medicare, 1989.

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I Dubois, R.W. Hospital Mortality as an Indicator of Quality. Pp. 107–131 in Providing Quality Care: The Challenge to Clinicians. Goldfield, N. and Nash, D.B., eds. Philadelphia, Pa.: American College of Physicians, 1989. Eisenberg, J.M. Doctor’s Decision and the Cost of Medical Care. Ann Arbor, Mich.: Health Administration Press, 1986. Flood, A.B. and Scott, W.R. Professional Power and Professional Effectiveness: The Power of the Surgical Staff and the Quality of Surgical Care in Hospitals. Journal of Health and Social Behavior 19:240–254, 1978. GAO (General Accounting Office). DOD HEALTH CARE. Occurrence Screen Program Undergoing Changes, but Weaknesses Still Exist. GAO/HRD-89–36. Washington, D.C.: General Accounting Office, January 1989. Geller, S. Autopsy. Scientific American 248:124–135, 1983. Georgopoulos, B.S. and Mann, F.C. The Community General Hospital. New York, N.Y.: MacMillan, 1962. Gerbert, B., Stone, G., Stulbarg, M., et al. Agreement Among Physician Assessment Methods: Searching for the Truth Among Fallible Methods. Medical Care 26:519–535, 1988. Greenfield, S., Cretin, S., Worthman, L.G., et al. Comparison of a Criteria Map to a Criteria List in Quality-of-Care Assessment for Patients with Chest Pain: The Relation of Each to Outcome. Medical Care 19:255–272, 1981. Greenfield, S. Measuring the Quality of Office Practice. Pp. 183–198 in Providing Quality Care: The Challenge to Clinicians. Goldfield, N. and Nash, D., eds. Philadelphia, Pa.: American College of Physicians, 1989. Havighurst, C.C. and King, N.M. Private Credentialing of Health Care Personnel: An Antitrust Perspective. American Journal of Law & Medicine 9:131–201, 1983. Havighurst, C.C. Health Care Law and Policy: Readings, Notes, and Questions. Westbury, N.Y.: Foundation Press, 1988. Haynes, R.B., Taylor, D.W., and Sackett, D.L., eds. Compliance in Health Care. Baltimore, Md.: Johns Hopkins University Press, 1979. Hiatt, H.H., Barnes, B.A., Brennan, T.A., et al. Special Report: A Study of Medical Injury and Medical Malpractice. New England Journal of Medicine 320:480–483, 1989. Johnson, K.W. Knowledge Utilization and Planned Change: An Empirical Assessment of the A VICTORY Model. Knowledge in Society 2:57–79, 1989. Joint Commission. 1990 AMH. Accreditation Manual for Hospitals. Chicago, Ill.: Joint Commission, 1989. Kessner, D.M., Kalk, C.E., and Singer, J. Assessing Health Quality: The Case for Tracers. New England Journal of Medicine 288:189–194, 1973. Knaus, W.A., Draper, E.A., Wagner, D.P., et al. An Evaluation of Outcome from Intensive Care in Major Medical Centers. Annals of Internal Medicine 104:410–418, 1986. Landefeld, C.S. and Goldman, L. The Autopsy in Quality Assurance: History, Current Status, and Future Directions. Quality Review Bulletin 15:42–48, 1989. Larson, E., Oram, L.F., and Hedrick, E. Nosocomial Infection Rates as an Indicator of Quality. Medical Care 26:676–684, 1988.

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I Lehmann, R. Forum on Clinical Indicator Development: A Discussion of the Use and Development of Indicators. Quality Review Bulletin 15:223–227, 1989. Lohr, K.N., Winkler, J.D., and Brook, R.H. Peer Review and Technology Assessment in Medicine. R-2820-OTA. Santa Monica, Calif.: The RAND Corporation, 1981. Luke, R.D. and Boss, R.W. Barriers Limiting the Implementation of Quality Assurance Programs. Health Services Research 16:305–314, 1981. McDonald, C.J. Protocol-Based Computer Reminders, the Quality of Care and the Non-Perfectibility of Man. New England Journal of Medicine 295:1351–1355, 1976. McDonald, C.J., Hui, S.L., Smith, D.M., et al. Reminders to Physicians From an Introspective Computer Medical Record: A Two-Year Randomized Trial. Annals of Internal Medicine 100:130–138, 1984. Meyer, W., Clinton, J.J., and Newhall, D. A First Report of the Department of Defense External Civilian Peer Review of Medical Care. Journal of the American Medical Association 260:2690–2693, 1988. MMWR. Morbidity and Mortality Weekly Report. Autopsy frequency—United States (1980–1985). 37:191–194, 1988. Nelson, A.R. Orphan Data and the Unclosed Loop: A Dilemma in PSRO and Medical Audit. New England Journal of Medicine 295:617–619, 1976. Nelson, E.C. and Berwick, D.M. The Measurement of Health Status in Clinical Practice. Medical Care 27: S77-S90, (March Supplement) 1989. Nelson, E.C., Wasson, J.H., Kirk, J.W., et al. Assessment of Function in Routine Clinical Practice: Description of the COOP Chart Method and Preliminary Findings. Journal of Chronic Diseases 40:55S-63S, (Supplement) 1987. Neuhauser, D. The Relationship Between Administration Activities and Hospital Performance. Chicago, Ill.: The University of Chicago Center for Health Administrative Studies, Research Series 29, 1971. O’Leary, D. Future Trends in Evaluating Quality Care. Lecture delivered at the McCormick Center Hotel, Chicago, Ill., May 13, 1988. OTA (Office of Technology Assessment). The Quality of Medical Care. Information for Consumers. OTA-H-386. Washington, D.C.: U.S. Government Printing Office, June 1988. Palmer, R.H., Strain, R., Maurer, J.V., et al. Quality Assurance in Eight Adult Medicine Group Practices. Medical Care 22:632–643, 1984. Palmer, R.H., Louis, T.A., Hsu, L.N., et al. A Randomized Control Trial of Quality Assurance in Sixteen Ambulatory Care Practices. Medical Care 23:751–768, 1985. Palmer, R.H., Hall, J.A., Hargraves, L., et al. Internal Leadership versus External Review for Ambulatory Care Quality. Outline of Talk (Draft) September 28, 1988. Paul-Shaheen, P., Clark, J.D., and Williams, D. Small Area Analysis: A Review and Analysis of the North American Literature. Journal of Health Politics, Policy and Law 12:741–809, 1987. PPRC (Physician Payment Review Commission). Annual Report to Congress. 1989. Washington, D.C.: Physician Payment Review Commission, 1989.

OCR for page 265
Medicare: A Strategy for Quality Assurance - Volume I Roos, L.L. Nonexperimental Data Systems in Surgery. International Journal of Technology Assessment in Health Care 5:341–386, 1989. Roos, L.L., Roos, N.P., Fisher, E.S., et al. Strengths and Weaknesses of Health Insurance Data Systems for Assessing Outcomes. Paper prepared for the Institute of Medicine Study to Design a Strategy for Quality Review and Assurance in Medicare, 1989. Roos, L.L., Sharp, S.M., Cohen, M.M., et al. Risk Adjustment in Claims-based Research: The Search for Efficient Approaches. Journal of Clinical Epidemiology, 42:1193–1206, 1989. Ramsey, P.G., Carline, J.D., Inui, T.S., et al. Predictive Validity of Certification by the American Board of Internal Medicine. Annals of Internal Medicine 110:719–726, 1989. Rutstein, D.D., Berenberg, W.B., Chalmers, T.C., et al. Measuring the Quality of Medical Care (Tables Revised, 9/1/77) A Clinical Method. New England Journal of Medicine 294:582–588, 1976. Scott, W.R., Forrest, W.H. Jr., and Brown, B.V. Hospital Structure and Postoperative Mortality and Morbidity. Pp. 72–89 in Organizational Research in Hospitals. Shortell, S.M. and Brown, M., eds. Chicago, Ill.: Blue Cross Association, 1976. Schroeder, S.A. Strategies for Reducing Medical Costs by Changing Physicians’ Behavior. International Journal of Technology Assessment in Health Care 3:39–50, 1987. Shortell, S.M., Becker, S.W., and Neuhauser, D. The Effects of Managerial Practices on Hospital Efficiency and Quality of Care. Pp. 90–107 in Organizational Research in Hospitals. Shortell, S.M. and Brown, M. eds. Chicago, Ill.: Blue Cross Association, 1976. Smith, A.H. and Mehlman, M.J. Medicare Quality Assurance Mechanisms and the Law. Paper prepared for the Institute of Medicine Study to Design a Strategy for Quality Review and Assurance in Medicare, 1989. Stewart, A.L., Hays, R.D., and Ware, J.E. The MOS Short-form General Health Survey: Reliability and Validity in a Patient Population. Medical Care 26:724–732, 1988. Stewart, A.L., Greenfield, S., Hays, R.D., et al. Functional Status and Well-being of Patients with Chronic Conditions: Results from the Medical Outcomes Study. Journal of American Medical Association 262:907–943, 1989. Tarlov, A.R., Ware, J.E., Greenfield, S., et al. The Medical Outcomes Study. An Application of Methods for Monitoring the Results of Medical Care. Journal of the American Medical Association 262:925–930, 1989. Tierney, W.M., Hui, S.L., and McDonald, C.J. Delayed Feedback of Physician Performance Versus Immediate Reminders to Perform Preventive Care: Effects on Physician Compliance. Medical Care 24:659–666, 1986. Vladeck, B.C. Quality Assurance through External Control. Inquiry 25:100–107, 1988. Weiner, J., Powe, N., Steinwachs, D., et al. Quality of Care Indicators for Potential Application to Insurance Claims/Encounter Data. Report to the Cigna Foundation. Johns Hopkins University Research and Development Center, 1989.