Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 70
_, Methods of Technology Assessment As Chapter 1 indicates, technology as- a foundation for building a system of tech- sessment offers the essential bridge be- nology assessment for the nation. tween basic research and development and prudent practical application of medical technology. We have a substantial body of methods that can be applied to the various tasks of assessment, and their availability makes possible the acceptance, modifica- tion, or rejection of new technologies on a largely rational basis. That rationality, however, depends on many factors that go well beyond safety and efficacy, including, among other components, economics, eth- ics, preferences of patients, education of physicians, and diffusion of information. The methods that have been developed can take some account of most of these compo- nents, although combining the results for the components is a major task and one that is far from settled or solved. The exis- tence of these assessment methods provides The outline, introduction, and conclusions of this chapter were developed by Frederick Mosteller. The various sections of the chapter were drafted primarily by other authors identified at the opening of each sec- tion. 70 Most innovations in health care technol- ogy rest on some theoretical ideas held by the innovators. These ideas inevitably range in strength from very well informed to hopeful speculation. Beyond this, a few innovations are purely empirical in the sense that someone has noticed that the technology seemed to work, even though no underlying mechanism was proposed or understood. In considering medical tech- nologies, no matter how strong or weak the theoretical justification, experience must be decisive. If in practice the innovation is clearly better or clearly worse than existing technologies, then the innovation deserves adoption or rejection. It is known from much experience that merely having a good idea, a good theory, or a constructive observation is not enough because there are so many unexpected interfering variables that may thwart the innovation and the in- novator. Learning from controlled experi- ence is central to progress in health care. Learning from experience itself without formal planning often presents great diffi- culties and sometimes leads to long-main-
OCR for page 71
METHODS OF TECHNOLOGY ASSESSMENT tained fallacies, partly because of the lack of control of variables. This method is slow and expensive unless the effects are huge. Planning and analysis and scientific testing provide ways to strengthen the learning process. This chapter describes a number of techniques or methodologies that help to systematize learning from experience in health care technology. Few people are acquainted with more than a few of the methods used for assess- ment. Usually investigators are acquainted with the few methods most frequently used in their own specialties. Consequently, it seems worthwhile to give a brief descrip- tion of the more widely used methods and what they are most useful for studying. For direct attack on evaluation through data acquisition, clinical trials are highly regarded. For generating hypotheses, the case study and the series of cases have spe- cial value. Registries and data bases some- times produce hypotheses, sometimes they help evaluate hypotheses, and sometimes they aid directly in the treatment of pa- tients. Sample surveys excel in describing collections of patients, health workers, transactions, and institutions. Epidemiological and surveillance stud- ies, although not synonymous, are well adapted to identifying rare events that may be caused by adverse effects of a tech- nology. Quantitative synthesis (meta-analysis) and group judgment methods give us ways to summarize current states of knowledge and sometimes to predict the future. Simi- larly, cost-effectiveness analysis (CEA) and cost-benefit analysis (CBA) offer ways of introducing costs and economics into these assessments. Modeling provides a way to simulate the future and still include com- plicated features of the real life process and to see what variables or parameters seem to produce the more substantial effects. When backed with strong, although lim- ited, empirical investigation, it may add much breadth to an evaluation. 71 Sometimes what is learned to be true in a scientific laboratory may not, at first, be successfully applied in practical circum- stances. Myriad reasons can explain this: the new technique is not correctly applied, or to the right kinds of cases, or it is not ap- plied assiduously enough, or too assidu- ously, etc. This idea in medical contexts is captured in the terms efficacy and effec- tiveness. Efficacy refers to what a method can accomplish in expert hands when cor- rectly applied to an appropriate patient; effectiveness refers to its performance in more Unmoral routine applications. The rel- evance of these ideas here is that some of the methods presented below are more nat- urally adaptable to assessing one of these or the other. The reader will probably appre- ciate, for example, that surveillance and data banks point toward assessing effec- tiveness, and most randomized clinical trials point toward assessing efficacy. Although randomized clinical trials of- fer the strongest method of assessing the ef- ficacy of a new therapy, it is recognized that it is not possible to have randomized trials for every version of every innovation. However desirable that might be, it is not feasible. Consequently, other methods of assessment are often going to be depended on; of course, some technologies actually require other methods. This in turn means that steps need to be taken to strengthen the other methods. These steps have two forms. First, where possible, apply the known ways of improving studies, such as observational studies (for example, have a careful protocol, use random samples, use blindness where possible, and so on). Sec- ond, many of these methods could be im- proved if research were carried out to find new ways to improve them. Therefore, specific research that could lead to getting stronger results from the weaker methods is often suggested. Possibly, research will find that particu- lar methodologies are best when applied to special classes of treatments. For example,
OCR for page 72
72 perhaps noninvasive drugs and devices could be handled in one way and invasive methods in another. Perhaps data banks and registries could offer good results from some class of problems. Answers to such questions are not now available. At the same time that the need for im- proving the weaker methods is recognized, it is also recognized that the methods al- ready in existence are not sufficiently often applied. The Office of Health Technology Assessment (OHTA) evaluates the safety and effectiveness of new or as yet unestab- lished medical technologies and proce- dures that are being considered for cover- age under Medicare. Requests for these evaluations come from the Health Care Fi- nancing Administration (HCFA). OHTA carries out its evaluations by reviewing the literature and by getting advice from vari- ous agencies and professional organiza- tions. The information so acquired is syn- thesized to reach some conclusion. OHTA does not gather primary data itself. Again and again, it turns out, and OHTA notes, that the primary data are almost nonexis- tent and that primary data would be re- quired to reach a well-informed conclu- sion. In advising HCFA about coverage for various medical technologies, OHTA pre- pared 65 reports in the years 1982, 1983, and 1984. Lasch (1985) reviewed these re- ports to see what the state of the informa- tional base on safety and efficacy seemed to be (K. E. Lasch, Synthesizing in HRST Re- ports, unpublished report, Harvard School of Public Health, 1985~. Lasch sorted the reports into four categories, as follows: 1. The technology enjoyed widespread use and was considered an established technology. 2. The data base for the technology was insufficient; there was a call for more stud- ies and better research designs, or accuracy was questioned for diagnostic tests. 3. The data base was sufficient; the technology was not recommended. ASSESSING MEDICAL TECHNOLOGY 4. The technology was outmoded, not routinely used, and not an established therapy. After the studies were categorized for the 3 years, Lasch found the results shown in Table 3-1. The percentage values of the results are similar from year to year. The category of insufficient data stands out. In noting that 69 percent of these assess- ments have insufficient data to reach a sat- isfactory conclusion, it should not be as- sumed that the technologies in the other categories have always been evaluated on the basis of strong data. The categories were chosen to generate a clear set when the evidence was inadequate. The first cat- egory of widespread use may also include poorly evaluated technologies. This study, then, offers a clear message that many technologies that physicians wish to use have not been adequately evaluated. Simi- larly, at the consensus conferences, speak- ers frequently point out the lack of primary data (National Institutes of Health tNIH], 1983, 1984~. Thus, the most important need is to gather more primary data. As we report later in this chapter, the Office of Technology Assessment (OTA; 1980a) polled data analysts who conduct cost-effectiveness and cost-benefit analyses of health care technologies and found lack of information to be a uniformly signifi- cant problem. More primary research is needed, and this will have to be led in part by research physicians with training in quantitative methods and supported by doctoral-level epidemiologists and biostatisticians. All three groups are in short supply (National Academy of Sciences, 1978, 1981, 1983~. At the least the development of methods will also require epidemiologists and bio- statisticians. Therefore, on both grounds, we will need funds for training research personnel. Many assessment methods are described in some detail in the sections that consti-
OCR for page 73
METHODS OF TECHNOLOGY ASSESSMENT 73 TABLE 3-1 Distribution of Technologies for Years 1982, 1983, and 1984 into Four Typesa when Reviewed by OHTA for HCFA Percentages for Each Yearb Category 1982 1983 1984 Total Widespread use 16 (4) 19 (4) 21 (4) 18 (12) Insufficient data 68 (17) 76 (16) 63 (12) 69 (45) Data sufficient; technology not effective 4 (1) 0 (O) O (O) 2 (1) Technology not used or outmoded 12 (3) is (1) 16 (3) 11 (7) Totals 100 (25) 100 (21) 100 (19) 100 (65) aEach of the 65 reports was assigned to one of the above categories based on a reading of the summary and discussion sections. Coding of the 65 reports revealed that the four categories were mutually exclusive; each report fell neatly into one of the categories. bNumbers of studies shown in parentheses. tote the main body of this chapter. Unless explicitly interested in research methods, some readers may wish to scan cursorily through the chapter. Most sections follow a pattern that opens with a brief description of the method, fol- lowed by typical purposes and uses and by a subsection addressing capabilities and limitations, including some remarks on ways of strengthening the method in prac- tical use. Sometimes a final subsection dis- cusses research that could be done that might lead to improvements in the method. RANDOMIZED CLINICAL TRIALS* The randomized clinical trial (RCT) is a method of comparing the relative merits (and shortcomings) of two or more treat- ments tested in human subjects. A well-de- signed and -executed RCT is widely re- garded as the most powerful and sensitive tool for the comparison of therapies, diag- nostic procedures, and regimens of care. More broadly, the RCT can be regarded as an unusually reliable method for learn- ing from experience; its success lies in structuring that experience so as to fore- *This section was drafted by Lincoln E. Moses. close many sources of ambiguity. In the health sciences the method is applied not only in comparing therapies but also diag- nostic methodologies, ways of imparting information to patients, and regimens of care (e.g., home care versus critical care units for certain heart patients). In gen- eral, if alternative ways of accomplishing an aim are in competition, the RCT may be the best technique for resolving their relative merits. Notice that comparison is at the heart of the method. A clinical trial is not a device for ascertaining the health consequences of a toxic substance in food or for elucidating the etiology of a disease. It is a method for comparing interventions that are applied and controlled by the investigator. The clinical trial becomes an RCT if there is a deliberate introduction of randomness into the assignment of patients (eligible for both, or all, of the treatments) to treat- ment A, treatment B. etc. The reasons for such a method of assignment are discussed below. Hereafter, when referring to an RCT, it is contemplated that it satisfies these two conditions: 1. No subject is admitted without hav- ing been judged to be equally suitable to
OCR for page 74
74 receive any one of the treatments being of- fered to the subject's class of patients. 2. No subject is admitted without hav- ing volunteered to receive either treat- ment, as may be assigned. Practical Problems of Comparing Treatments Two factors make it intrinsically diffi- cult to compare different treatments. First, the subjects receiving the treatments usu- ally are different people, so differences found between the treatments could be due to differences among the subjects in the groups. If the groups differ in any system- atic way (whether recognized or not), the treatment comparison may be biased; bias can exaggerate, nullify, or reverse true dif- ferences. Second, even if the treatments could be compared in the same patients (as sometimes happens), the contrast between the treatments will vary from one patient to another, producing uncertainty in the overall assessment. This is the problem of variability. Large samples can reduce the disturbance of variability but do not help with bias. If two treatment groups are differently constituted, then bias in the treatment comparison must be regarded as likely. The phrase "differently constituted" ap- plies, for example, where the treatment groups are (1) admitted to the study by dif- ferent means, (2) treated in different places, at different times, or by different sets of practitioners; (3) assessed by differ- ent groups; or (4) analyzed and reported by different teams. Randomization in a clinical trial is aimed at preventing bias. Two characteris- tic features are essential to realizing that aim. First, the study is conducted under a protocol that makes explicit exactly what questions are to be studied, what treat- ments are to be applied; and how, to what kind of patients, when, and where. It also specifies how assessment of outcomes will ASSESSING MEDICAL TECHNOLOGY be done and how statistical analyses will be conducted. Second, the RCT calls for assignment of the respective treatments to each eligible patient admitted to the study by means of a random choice. The effect of this is to en- sure that the two treatment groups are not "differently constituted", indeed, they are brought into being as random subsets of a singly constituted group which is opera- tionally defined by the protocol. The protocol-controlled RCT is even stronger whenever knowledge of which treatment a patient has received is screened from participants (patients, treat- ing physicians, outcome assessors). A result of such "blinding" is to ensure that placebo effects remain randomly assorted to the treatments. Another result is to prevent differential decisions about care during the study. It is especially important that those assessing outcomes be blind to the type of treatment—unless the outcome is entirely objective, e.g., length of survival. In some cases, blinding of physicians may not be possible, such as when a medical modality is being compared with a surgical one. The Protocol The protocol is a written prescriptive document that spells out the purposes and rationale of the trial and how it will be conducted. Specifics include the criteria of eligibility for inclusion of patients in the trial and criteria for exclusion—and de- scription of treatments, adjuvant therapy, outcome measurements, patient follow- up, and statistical analyses to be per- formed. The protocol also specifies the numbers of patients to be entered and the mechanics of randomization. The protocol is both a planning document and a proce- dures manual. The aim is to provide trust- worthy answers at the end of the study to the following questions: What treatments were applied, to what kinds of patients, with what results? What do the results mean?
OCR for page 75
METHODS OF TECHNOLOGY ASSESSMENT Provisions for blindness and for the or- der in which processes are to be performed can be central to the validity of a study and to the value of the protocol that governs it. If the decision to enter each patient into the trial is made in the knowledge of which treatment the next patient will receive, then ample opportunity for building up noncomparable treatment groups is at hand, so the protocol should not use alter- nate-patient assignment to the treatments. If a rather subjective diagnostic test W as- sesses a condition thought to be related to another test V, then W measured after V is not the same as W measured before V; it may be important that the protocol specify the order in which they are to be done. The careful protocol attempts to specify in ad- vance all procedural steps that may mate- rially affect execution of the trial and inter- pretation of its results. A well-conducted RCT requires not only a good protocol but also that the trial be carried out in accordance with it. The pro- tocol may call for specific steps to check on (and promote) protocol adherence. Staging and laboratory analyses may be checked by introducing (blindly) occasional standard specimens. Samples of study records may be checked back to more basic clinical rec- ords. Visits by monitors, combined with audit, may be routinely conducted in multicenter studies. The protocol also has the character of a compact among the participating investi- gators, relevant human subjects commit- tees, and funding sponsors. This contrac- tual character lends stability to a study over its lifetime, helping to supply definite answers to the questions concerning what was done, to what kinds of patients, and with what results. Random Assignment to Treatment The primary reason for random assign- ment is to prevent bias by breaking any possible systematic connection of one treat- ment or the other with favorable values of 75 interfering variables (whether recognized or not). A fuller appreciation of this princi- ple may be gained by considering two al- ternative modes of treatment comparison that are sometimes advocated. The first is the use of historical controls, the second is the use of statistical procedures to adjust for treatment group differences in the im- portant interfering variables. The historically controlled trial (HCT) compares outcomes on a new treatment to outcomes in previous (historical) cases from the same setting. The motivation is to arrive at decisions sooner by assigning all eligible patients rather than only half of them to the new treatment. But because the treatment and control groups come from different time periods, they are "dif- ferently constituted groups." This raises the spectre of bias and sometimes the ac- tuality. The drop in cardiovascular deaths and the decrease in perinatal mortality over the last decade are both not really un- derstood, and both exemplify temporal shifts in control levels of the sort that viti- ate historical controls. Time changes all things, including the patients' characteris- tics at a hospital, the effectiveness of adju- vant treatments not under study, the skill of surgeons with a new operation, and the skill of physicians with a new drug. Thus, it is hard to know when an HCT does reach a valid conclusion. There are successes and there are failures. An example of what seems to be a suc- cessful HCT is that of a changing policy by an institution toward stab wounds. Origi- nally, the policy had been to perform an exploratory laparotomy on all patients pre- senting with abdominal stab wounds. On the basis of advances in handling wounds and some data from refusals to give con- sent, the institution decided to change to a policy allowing surgical judgment to be ex- ercised. This reduced considerably the number of laparotomies performed (92 to 40 percent) and also the numbers of infec- tions (Nance and Cohn, 1969~. The overall complication rate dropped from 27 to 12
OCR for page 76
76 percent, and no complications occurred in 72 unexplored patients. Byar et al. (1976) call attention to an RCT comparing placebo and estrogen therapy for prostate cancer in which the survival of placebo controls admitted! in the first 2.5 years was significantly shorter (p = .01) than the survival of those admit- ted in the second 2.5 years, although ad- mission criteria, in a fixed setting, were un- changed. They point out that the use of the early placebo group (as historical controls) would have falsely led to the conclusion that estrogen therapy (in the second pe- riod) was effective. It is possible to consider the use of histor- ical controls whenever the variation in suc- cessive control levels is statistically taken into account. However, it may be difficult or impossible to estimate that variation; that is a practical difficulty. Furthermore, there is a theoretical principle that applies. The work of Meter (1975), and later Po- cock (1976), show that for a given standard deviation in batch-to-batch random bias, there is a minimum study size number (the number of experimental subjects) beyond which relying on historical controls, no matter how numerous they are, is inferior to dividing the sample into two equal groups, half experimental and half control. In summary, historical control trials are inferior to RCTs because (1) differently constituted groups are inherently likely to produce bias; (2) if the historical controls were comparable and if the random bias of successive batches of controls had variabil- ity that was exactly known, then reliance on the historical control data would be preferable to randomization only for stud- ies below a certain threshhold size; and (3) knowledge of variability of the random bias is often not available. One often sees the argument that the need for randomization can be circum- vented by making statistical adjustments for differently constituted subgroups, cor- recting for differences in the influential ASSESSING MEDICAL TECHNOLOGY variables that affect outcomes, and render- ing the subgroups comparable. It is easy to find statisticians who place little credence in this trust of statistical adjustment, and for cogent reasons. First, some of the most influential variables may not even be rec- ognized as important. Second, the ones that are recognized as important may not have been measured, or they may not have been measured comparably. Third, just how to make the adjustment can be very unclear; mutually influential variables can be interrelated in ways that both are im- portant and poorly understood. Random- ization avoids these difficulties by ensuring that whatever the critical variables may be and however they may conspire together to affect the outcomes, they cannot systemat- ically benefit one treatment over the other, beyond those vagaries of chance for which the significance test specifically makes al- lowances. This approach avoids the effort of trying to unravel the Gordian knot of causation and cuts through it at one stroke, by random assignment. Before leaving the subject of random as- signment, the idea of randomization within strata should be addressed. If some pretreatment variable, say stage of disease, is known to be strongly related to outcome, then it can be wise to design the study so that (nearly) equal numbers of both treat- ments occur at each level of that pretreat- ment variable. This kind of design is quite natural for multi-institutional studies, when each institution is treated as a stra- tum. Refining the randomization to be done separately within strata does not give added protection against bias, but it may increase the efficiency of a study, i.e., in- crease its effective sample size (usually only moderately). Limitations of RCTs The method, powerful as it is, is hard to apply under certain circumstances. If out- comes mature after decades, then comple-
OCR for page 77
METHODS OF TECHNOLOGY ASSESSMENT tion of the RCT requires long-term main- tenance of protocol-controlled follow-up, which is difficult and expensive. If a sufficiently rare outcome is the end- point of interest, then detection of treat- ment differences may call for unworkably large sample sizes. One example was con- cern about the safety of the anesthetic, halothane. Detection of differences in sur- gical death rates (about 2 percent overall) that might relate to anesthetic choice would amount to trying to distinguish be- tween death rates such as 1.9 percent and 2.1 percent a task calling at least for hun- dreds of thousands of patients. The retro- spective study that was done did arrive at conclusions, but they were expressed with diffidence made necessary by the possible existence of unrecognized biases. Sometimes it is objected that an RCT is not applicable because treatments are too variable to be controlled with the speci- ficity that an RCT demands. This objec- tion is sometimes false; for example, a treatment may be defined to allow modifi- cation as indications arise in the course of therapy. In other cases, the objection is simply specious, for it asserts the impossi- bility of answering the question "What is the treatment?" That impossibility would block any kind of objective assessment of it. A rather more difficult limitation to deal with grows out of the possibility that a new procedure started in an RCT may, outside that trial, evolve into a superior modified version of the treatment. Then, continua- tion of the RCT is at risk of being irrelevant or unethical. There is a real problem here, and it deserves more study; the question is how the use of protocol and randomization can help to speed sound evolution of new therapies. One proposal has been to "ran- domize the first patient." (See, for in- stance, Chalmers, 1975, 1981.) Inherent in the concept of randomizing the first pa- tient is a fluid protocol that allows a change in the details of a new treatment as 77 the investigators improve their perfor- mance (the "learning curve") or as other information appears. It has not found wide agreement. The definitive treatment of these issues is not yet at hand. The sample size of an RCT may have been planned to resolve differences of a stated size, but when it is completed, ques- tions about treatment comparisons in cer- tain subclasses of patients cannot be re- solved. This is not a limitation of the RCT per se, for more questions always can be asked of a body of data than can be an- swered by it, but one should be warned to think at the planning stage about choosing sample sizes large enough to support ade- quate treatment comparisons in particu- larly salient subgroups. It is sometimes argued that RCTs are too costly. The cost of disciplined, careful, checked medical work is of course high; the advantages of the protocol are not cheaply bought. But in many medical centers with already high standards of recordkeeping, diagnosis, etc., the incremental cost of the protocol might not be great. The incre- mental cost of randomization is negligible. Costs can be high when the base costs of bed, drugs, tests, and care are all loaded onto the RCT budget. Most of these costs would have been incurred anyway, re- gardless of how the patients were treated. Failure to distinguish between total costs, which include those that would be incurred anyway, from incremental costs of RCTs is inherently misleading and could lead to grievous policy errors. Good mea- surements of incremental costs of RCTs are needed. This will involve both conceptual effort and data gathering. Better informa- tion concerning actual incremental costs of RCTs is a topic that should receive system- atic research attention. Two other limitations of RCTs also are drawbacks to any investigational method. The first is that dispute may grow around unwelcome conclusions and hinder adop- tion of the findings. The second is that the
OCR for page 78
78 RCT may give a clear verdict in patients of the kind used in the trial, but leave unan- swered the question of efficacy in different kinds of subjects. This issue, dubbed exter- nal validity sometimes is readily dealt with; thus, the Salk vaccine trials showed the vaccine to be effective in first-, second-, and third-grade children. No difficulty was found in generalizing the conclusion to both older and younger children. Some- times things are harder they may even demand further RCTs. External validity is of course a problem whenever we under- tal~e to learn from one body of experience and then apply the results to other experi- ence; it is not a peculiar difficulty of RCTs. We do not know as much as we could af- ford to about designing studies with an eye on external validity. This is another area that deserves further research effort. Strengthening RCTs The primary paths to good quality lie in designing a strong protocol and executing it faithfully. The paper by Goodman in Appendix 3-B of this committee's report gives a systematic treatment of most of the key features of a strong protocol. Extensive accounts of RCT protocol are given in works by Friedman et al. (1981) and Sha- piro and Louis (1983~. Some additional ideas on pre- and post-protocol execution deserve comment here. First, the study should be large enough; if it is too small to have a good chance of establishing the existence of a plausibly sized actual improvement, then it needs to be made larger or to be abandoned. Other- wise, work, money, and time will be de- voted to an effort that lacks a good chance of producing a useful finding. Statistical methods for assessing adequacy of planned study size (power calculations) are well es- tablished and should be used. (Sometimes, however, the opportunity to do a study is too good to be missed even if it is too small ASSESSING MEDICAL TECHNOLOGY to be definitive. This should be reported with the study in hope that results of other studies can be combined with these and to- gether they may reach firm conclusions.) Second, the participating investigators should fully understand and be fully sup- portive of the investigation. Persons with initial convictions about relative merits of the treatments may prove to be encum- brances to successful execution of the pro- tocol. Third, in planning for the time and number of cooperating centers that will be needed to carry the study through, be real- istically guarded about the flow of eligible patients that can be anticipated. Seasoned RCT veterans recommend safety factors of two, five, even ten. The foregoing suggestions all relate to the planning phase. A final way of strengthening the RCT applies to the com- pletion phase. Write about and report it well. In par- ticular, the operational definitions of all terms should be clear. Thus, the reader should not be left with doubts about how the subjects were defined and selected, how they were assigned to treatments, what treatments were applied, or how out- comes were measured. In addition, the re- port should specify whether study staff were blind to treatment allocation at key steps like enrollment in study, determina- tion of eligibility, interpretation of diag- nostic tests, measurement of outcome, etc. These issues were prominent among those that DerSimonian et al. (1982) checked in reviewing reports of clinical trials in four leading medical journals and that Emerson et al. (1984) checked in reviewing reports in six leading surgical journals. Both stud- ies answered five questions: (1) What were the eligibility criteria for admission to the study? (2) Was admission to the study done prior to allocation of treatment? (3) Was allocation to treatment done at random? (4) What was the method of randomiza-
OCR for page 79
METHODS OF TECHNOLOGY ASSESSMENT tion? (5) Were outcomes assessed by per- sons who were blind to treatment? Good reporting will also explain the quality control measures that were an- plied, methods of follow-up used, and au- dit checks employed. Not only should the reader be told what was done, and how, but also what hap- pened. Summary statistics should have the aim of revealing information to the reader. The methods of statistical analysis should be explained. The best way to do this is topic by topic. The analysis was ac- tually done in such a pattern; it should be reported that way: for understanding, for specificity, and, incidentally, for ease of writing. Sometimes one finds a published paper which lists statistical procedures in the methods section. "We used chi- squared, the l-test, the F-test, and Jonck- heere's test." The use of this style of report- ing for banquet recipes would list all the ingredients in all the dishes together and report the use of stove, mixer, oven, meat grinder, egg beater, and double boiler. In addition to showing the data, or gen- erously detailed summaries of them, the statistical analysis should state each of the principal questions that motivated the study and what light the data shed on those questions. (Note that this is not the same thing at all as reporting just those results that are statistically significant.) To lend understanding both to significant and non- significant results, it is wise to use confi- dence intervals whenever feasible and to report the power of statistical tests that are applied (Freiman et al., 1978~. Interesting statistical results that arise out of studying the data (rather than from studying the principal questions that motivated the study) are necessarily on a different, and somewhat ambiguous, logical footing. It is usually wise to regard such outcomes with considerable reserve, more as hypotheses turned up than as facts established. It is es- pecially important to be candid about the 79 nature and amount of "data dredging" that has accompanied the analysis. A Final Remark The protocol has been described as a compact; its construction is typically a col- legial exercise. This entails some advan- tages. Of course, deliberation and consul- tation give opportunities for better planning. Sometimes a sequence of RCTs leads to cumulative expertise and strategiz- ing. But, some of the greatest advantages may lie in the ethical domain. The use in human beings of a new treat- ment with only partially understood prop- erties raises certain problems of ethical portent. (This is true whether that new treatment is tried in an RCT or in any other way.) Among these questions are the following: How strong is the evidence that this new treatment may be at least as good as the best available current therapy? How shall we know when we should stop using both treatments and prefer only one of them? Who shall be able to receive this new treatment, and who shall not? Each of these questions is likely to be better an- swered when decided by a group of profes- sionals, acting explicitly and consul- tatively, in a process open to review. Wishful thinking blooms wherever Homo sapiens is found, but group consultation tends more often than not to restrain it. Another advantage of the collegial building of the protocol is that investiga- tors who already believe they know which treatment is superior have the opportunity to drop out, leaving to the trial's execution investigators able to proceed in good con- science to participate themselves and to in- vite their patients to participate.
OCR for page 80
80 EVALUATING DIAGNOSTIC TECHNOLOGIES* Accurate diagnosis is central to good medical practice. Diagnostic technology provides the physician with diagnostic in- formation. However, all diagnostic tests and procedures have associated costs and risks. Thus, persons involved with medical care must determine whether an individ- ual test or procedure provides significant new diagnostic information and whether the information provided and its impact on subsequent medical care offset the costs and risks of the technology. For each diag- nostic test, these and related questions re- quire assessment of (l) the diagnostic infor- mation provided and (2) the impact of the resulting therapy on patient outcome. Such assessments of diagnostic technology rarely are performed. Most diagnostic technology undergoes only narrow and limited evaluation. The lack of more com- prehensive assessment severely limits the efficient and optimal use of diagnostic tests and procedures. Fineberg et al. (1977) has formulated a hierarchy of evaluation of diagnostic tech- nologies: 1. Technical capacity—Does the device or procedure perform reliably and deliver accurate information? 2. Diagnostic accuracy Does the test contribute to making an accurate diagno- . ~ slsr 3. Diagnostic impact Does the test result influence the pattern of subsequent diagnostic testing? Does it replace other di- agnostic tests or procedures? 4. Therapeutic impact Does the test result influence the selection and delivery of therapy? Is more appropriate therapy used after application of the diagnostic test *This section was contributed by I. Sanford Schwartz. ASSESSING MEDICAL TECHNOLOGY than would be used if the test was not available? 5. Patient outcome—Does performance of the test contribute to improved health of the patient? Clearly, if diagnostic technology fails ut- terly at any step in this chain, then it can- not be successful at any later stage. If it succeeds at some stage, this implies success in the prior stages (even if they have not been explicitly tested) but does not tell what success may be attached to later stages. Thus, an accurate test may or may not lead to more accurate diagnosis, which in turn may or may not lead to better ther- apy, and that in turn may or may not even- tuate in better health of the patient. Be- cause many tests may be involved, it can require carefully designed studies to gauge success or failure of any particular one at stages 2 through 5. Present Evaluation Methods The first step in the hierarchy of evaluat- ing diagnostic tests and procedures is deter- mination of the technical performance of the test. Several factors are involved in this evaluation. The first deals with the ability of the test actually to measure what it claims to measure. Replicability and bias of test results are important measures of test performance. Replicability (i.e., preci- sion) reflects the variance in a test result that occurs when the test is repeated on the same specimen. A highly precise test ex- hibits little variance among repeated mea- surements, an imprecise test exhibits great variance. The greater this variation, the less faith one may have in a single test's results. However, a precise test is not nec- essarily a good test. A test may exhibit a high level of replicability yet be in error. A good test must be reliable (i.e., unbiased); that is, it must exhibit agreement between the mean test result and the true value of the biologic variable being measured in the
OCR for page 165
METHODS OF TECHNOLOGY ASSESSMENT where I is the interval of time since the last Pap smear (in this example 1 year), F(t) is the cumulative distribution for the length of time from the moment a lesion is first detect- able by a Pap smear until it becomes an inva- sive cancer, P(t) is the cumulative distribu- tion for the length of time from the first moment of invasion to the appearance of signs and symptoms that would cause the pa- tient to seek care in the absence of screening, rots is the instantaneous incidence rate of in- vasive cancers tr(O) is the rate in 40-year-old average-risk women], and EN is the random false-negative rate of the Pap smear. Each of the elements in Equation 1 has an intuitive interpretation. The variable of integration, t, denotes the possible times that the woman might develop an invasive cancer of the cervix (t = 0 is now). By inte- grating from negative infinity to positive infinity, this formula considers all the pos- ~ _ loo O ~ O' > ~ Lll — 7= ~' J ~ ~ Z E is o G 50 IL Z a: ~ o 25 165 sible times that an invasive cancer might occur. For any particular time that an in- vasive cancer might occur (call this time t ' ), the expression 1 - PI - ~ ' ~ gives the probability that the woman is currently asymptomatic and has not yet detected or sought care for signs or symptoms of the cancer. F(t' + 1) - F(t') gives the proba- bility that the cancer was not potentially detectable until after the last Pap smear was done a year ago. The expression 1 - F(t' + 1) gives the probability that the le- sion was detectable before last year's Pap smear. This last expression must be multi- plied by FN, the chance that that Pap smear was falsely negative and missed it. The expression rat') expel - it r~x)dx] is the probability that this woman will in fact de- velop an invasive cancer at the time t'. A formula for the second probability is the same as Equation 1 except that Equa- ~ cnCC m11115 `< _~o ~ A>) 10 X c,) _ J ~ 5 80 _ ~6 5 4 3 2 1 60 _ \ Frequency (years between tests) 40 _ 20 _ 1 1 50 100 150 200 250 300 FINANCIAL COSTS FIGURE 3-6 Effect of Pap test frequency on financial cost and three measures of benefit for a 20-year-old average-risk woman. Main assumptions are as follows: (1) testing is begun at age 20; (2) a woman will have a checkup every 3 years for other malignant diseases from ages 20 to 40, and then annually thereafter; (3) the marginal cost of a Pap test is $10; (4) Pap test-detect- able dysplasia and carcinoma in situ precede invasive cervical carcinoma by an average of 17 years (range, O to 34 years); (5) 2.5 percent of invasive cervical cancers develop very rapidly, requiring less than 2 years to pass through dysplasia and CIS; (6) no cases of dysplasia or CIS regress spontaneously; (7) no Pap tests are falsely read as positive or suspicious; and (8) 5-year relative survival rates from time or detection (lead time adjusted) are dysplasia and CIS, 98 percent; local invasive, 78 percent; and regional invasive, 43 percent. If. a woman must also pay a $25 office visit fee for the separate visits for the Pap test, the costs increase to about $700 for an annual Pap test and $1,700 for a biannual Pap test (Eddy, 1981~.
OCR for page 166
166 tion 1 must be multiplied by l - EN, the probability that the Pap smear will not be falsely negative. In similar fashion formu- las can be written for the other important probabilities. These formulas are more complicated if one wants to consider the use of more than one type of test, a series of previous examinations done at various fre- quencies, and other factors, but the con- cepts are similar. To estimate the value of a Pap smear done at various frequencies one can apply formulas to calculate the probabilities of important clinical and economic outcomes relating to cervical cancer for each year in a woman's life, constantly updating the parameters of the formulas to keep track of the woman's changing age and screening history. The calculations can be performed for each screening strategy being evalu- ated: for example, no screening at all, screening every year, screening every 3 years, screening every year for three nega- tive examinations and then every 3 years, and so forth. Parameters for the equations, such as age-specific incidence rates [ret)] and parameters for the functions P(t) and F(t), are estimated from the data collected in clinical and epidemiological studies. The results of an analysis using parame- ter values estimated from such studies are illustrated in Figure 3-6 (Eddy, 1981~. This figure shows the estimated effect of screen- ing a woman with a Pap smear at various frequencies from age 20 to 75. The figure indicates three measures of benefit: the de- crease in the probability that the woman will die of cervical cancer; the increase in her life expectancy, given that the woman is destined to get invasive cancer; and the increase in life expectancy for the average- risk woman who may (with about a 1 per- cent probability) or may not get invasive cervical cancer. The horizontal axis gives the present value (at age 20) of a lifetime series of screening examinations minus the present value of expected savings in treat- ment costs. ASSESSING MEDICAL TECHNOLOGY The calculations indicate that the 3-year Pap smear is about 99 percent as effective as an annual Pap smear. If the 40-year-old, average-risk woman in the original exam- ple postponed her Pap smear another 2 years, the increased annual risk she would run of dying of cervical cancer would be on the order of l per lOO,000, about the same as the risk of death from one round-trip transcontinental airplane flight. REFERENCES American College of Surgeons, Commission on Cancer. 1974. Cancer Program Manual. Chicago: American College of Surgeons. American Public Health Association. 1981. Con- trol of Communicable Diseases in Man, A. Beneson, ed. Washington, D.C. Arnstein, S. R. 1977. Technology assessment: Op- portunities and obstacles. IEEE Trans. Syst. Man and Cybern. SM-7:571-582. Averill, R. F., and L. F. McMahon. 1977. A cost- benefit analysis of continued stay certification. Med. Care 15:158. Bailar J. C. 1970a. Periodical incidence surveys: I. Organization. Seminar on Cancer Registries in Latin America, Pan American Health Organization-World Health Organization, 41-100. Bailar J. C. 1970b. Periodical incidence surveys: II. Basis for the selection of survey areas. Seminar on Cancer Registries in Latin America, Pan American Health Organization-World Health Organization, 101-110. Bailar, J. C., III, T. A. Louis, P. W. Lavori, and M. Polansky. 1984. A classification for biomedical re- search reports. N. Engl. J. Med. 311:1482-1487. Banta, D. H., C. J. Behney, and J. S. Willems. 1981. Costs and their evaluation. In Toward Rational Technology in Medicine. New York: Springer Pub- lishing. Barron, B. A., and R. M. Richart. 1981. Screening protocols for cervical neoplastic disease. Gynecol. On- col. 12:S156. Baum, M. L., D. S. Anish, T. C. Chalmers, et al. 1981. A survey of clinical trials of antibiotic prophy- laxis in colon surgery: Evidence against further use of no-treatment controls. N. Engl. J. Med.305: 795-798. Bearman, J. E., R. B. Lowenson, and W. H. Gul- len. 1974. Muench~s Postulates, Laws and Corollaries, Biometrics Note #4, National Eye Institute, DHEW, Bethesda. Bell, R. L., and E. O. Smith. 1982. Clinical trials in post-marketing surveillance of drugs. Controlled Clinical Trials 3:61-68.
OCR for page 167
METHODS OF TECHNOLOGY ASSESSMENT Berk, A. A., and T. C. Chalmers. 1981. Cost and efficacy of the substitution of ambulatory for inpa- tient care. N. Engl. J. Med. 304:393-397. Bernstein, L. M., E. R. Siegel, and C. M. Gold- stein. 1980. The hepatitis knowledge base: A proto- type information transfer system. Ann. Intern. Med. 93: 169-181. Berwick, D. M., and A. L. Komaroff. 1982. Cost effectiveness of lead screening. N. Engl. J. Med. 306: 1392-1398. Blum, R. L. 1982. Discovery, confirmation, and incorporation of causal relationships from a large time-oriented clinical data base: The RX project. Comp. Biomed. Res. 15:164-187. Boruch, R. F. 1985. Enhancing the usefulness of longitudinal data by coupling longitudinal surveys and randomized experiments. Draft Report for DOL, Center for Statistics and Probability, Northwestern University. Boruch, R. F., and J. S. Cecil. 1979. Assuring con- fidentiality of social research data. Philadelphia: Uni- versity of Pennsylvania Press. Boston Collaborative Drug Surveillance Program. 1973. Oral contraceptives and venous thromboembol- ic disease, surgically confirmed gallbladder disease and breast tumors. Lancet 1:1399-1404. Braakman, R. 1978. Data bank of head injuries in three countries. Scott. Med. J. 23:107-108. Bruce, R. A., et al. 1974. Seattle Heart Watch: Ini- tial clinical circulation and electrocardiographic re- sponses to maximal exercise. Am. J. Cardiol. 33:459- 469. Bruce, R. A., et al. 1981. A computer terminal pro- gram to evaluate cardiovascular functional limits and estimate coronary event rates. West. J. Med. 135:342- 350. Bunker,J, P., B. J. Barnes, and F. Mosteller. 1977. Costs, Risks and Benefits of Surgery. New York: Ox- ford University Press. Burke, J. F., H. S. Jordan, C. B. Boyle, and E. Vanner. 1981. An Impact Study of the 1978 Consen- sus Conference on Supportive Therapy in Burn Care. Massachusetts Health Research Institute. Byar, D. P., R. M. Simon, W. T. Friedewald, et al. 1976. Randomized clinical trials: Perspectives on some recent ideas. N. Engl. J. Med. 295:74-80. Campbell, D. T., and J. C. Stanley. 1963. Experi- mental and Quasi-Experimental Designs for Re- search. Chicago: Rand McNally College Publishing. Centers for Disease Control. 1979. Abortion Sur- veillance 1977. Atlanta, Gal: Department of Health, Education, and Welfare. Centers for Disease Control. 1982a. Annual Sum- mary, 1981. Reported morbidity and mortality in the United States. Morbid. Mortal. Weekly Rep. 30~54~. Centers for Disease Control. 1982b. Annual Sum- mary, 1981. Morbid. Mortal. Weekly Rep. 30:126- 127. 167 Centor, R. A., and J. S. Schwartz. In press. Calcu- lation of the area under a ROC curve using microcom- puters. Med Decision Making. Chaitman, B. R., et al. 1981. Angiographic preva- lence of high-risk coronary artery disease in patient subsets (CASS). Circulation 64:360-367. Chalmers, T. C. 1975. Randomizing the first pa- tient. Med. Clin. North Am. 59:1035-1038. Chalmers, T. C. 1981. The Clinical Trial. Milbank Mem. Fund Q. 59:324-339. Cochran, W. G. 1954. The combination of esti- mates from different experiments. Biometrics 10: 101- 129. Cohen, S. N., M. F. Armstrong, R. L. Briggs, et al. 1974. Computer-based monitoring and reporting of drug interactions. Medinfo 1974:889-894. Collen, J. F., R. Feldman, A. Siegelaub, and D. Crawford. 1970. Dollar cost per positive test for auto- mated multiphasic screening. N. Engl. J. Med. 283:459-463. Collen, M. F. 1979a. A case study of mam- mography. In Medical Technology and the Health Care System: A Study of the Diffusion of Equipment- Embodied Technology. Prepared by the Committee on Technology and Health Care, National Academy of Sciences, Washington, D.C. Collen, M. F. 1979b. A study of multiphasic health testing. In Medical Technology and the Health Care System: A Study of the Diffusion of Equipment- Embodied Technology. Prepared by the Committee on Technology and Health Care, National Academy of Sciences, Washinggon, D.C. Collen, M. F. 1979c. A guideline matrix for tech- nological system evaluation. J. Med. Systems 2:249- 254. Collen, M. F. 1979d. Cost effectiveness of auto- mated laboratory testing. In Clinician and Chemist, D. S. Young, et al., ea., Proceedings of the First A.O. Beckman Conference in Clinical Chemistry, Ameri- can Association of Clinical Chemists, Washingon, D.C. Collen, M. F. 1983. Utilization of Diagnostic X-ray Examinations. HHS Pub. FDA 83-8208. Washington, D.C.: U.S. Government Printing Office. Collen, M. F., L. G. Dales, G. D. Friedman, et al. 1973. Multiphasic health checkup evaluation study. 4. Preliminary cost benefit analysis for middle-aged men. Prev. Med. 2:236-246. Collen, M. F., S. R. Garfield, R. H. Richart, et al. 1977. Cost analyses of alternative health examination modes. Arch. Int. Med. 137:73-79. Commission on Cancer. 1974. Cancer Program Manual. Chicago: American College of Surgeons. Cook, T. D., and L. C. Leviton. 1980. Reviewing the literature: A comparison of traditional methods with meta-analysis. J. Pers. 48:449-472. Cooper, B. S., and D. P. Rice. 1976. The economic cost of illness revisited. Soc. Secur. Bull. 39:21-36.
OCR for page 168
168 Cowley, R. A., W. J. Sacco, W. Gill, et al. 1974. A prognostic index for severe trauma. J. Trauma 14: 1029-1035. Cox, E. B., and W. Stanley. 1979a. Schema driven time-oriented record on minicomputer. Comp. Biomed. Res. 12:503-516. Cox, E. B., J. Laszlo, and A. Freiman. 1979b. Classification of cancer patients: Beyond TNM. J. Am. Med. Assoc. 242:2691-2695. Cretin, S. 1977. Cost-benefit analysis of treatment and prevention of myocardial infarction. Health Serv. Res. 12:174. Cutler, S. J., J. Scotto, S. S. Devesa, and R. R. Connelly. 1974. Third National Cancer Survey—An overview of available information. J. Natl. Cancer Inst. 53:1565-1575. Dalkey, N. C. 1969. The Delphi Method: An Ex- perimental Study of Group Opinion. Santa Monica, Calif.: Rand Corporation. Dannenberg, A. L., R. Shapiro, and J. F. Fries. 1979. Enhancement of clinical predictive ability by computer consultation. Meth. Inform. Med. 18:10- 14. Deane, R., and A. Ulene. 1977. Hysterectomy or tubal ligation for sterilization: A cost-effectiveness analysis. Inquiry 14:73. Delbecq, A., A. H. Van de yen, and D. H. Gustaf- son. 1975. Group Techniques for Program Planning. Glenview, Ill.: Scott, Foresman. Demographic Analysis Section, Biometry Branch, National Cancer Institute. 1976. Code Manual—The SEER Program. DHEW Pub. No. (NIH)-79-1999. Bethesda, Md.: National Cancer Institute. DerSimonian, R., L. J. Charrette, B. McPeek, and F. Mosteller. 1982. Reporting on methods in clinical trials. N. Engl. J. Med. 306:1332-1337. DerSimonian, R., and N. Laird. 1982. Evaluating the effectiveness of coaching for SAT exams, a meta- analysis. Pp. 1-15 in Harvard Educational Review. Devine, E. C., and T. D. Cook. 1983. Effects of psychoeducational intervention on lengh of hospital stay: A meta-analytic review of 34 studies, American Journal of Nursing. Reproduced in R. J. Light, ed. 1983. Evaluation Studies, Review Annual, Vol. 8, pp. 417-432. Beverly Hills, Calif.: Sage Publications. Dixon, R. E. 1978. Effects of infections on hospital care. Ann. Intern. Med. 89 (part 2~: 749-753. Drazen, E., and J. Metzger. 1981. Methods for evaluating costs of automated hospital information systems. DHHS Publ. No. (PHS) 81-3283. Washing- ton, D.C.: U.S. Government Printing Office. Dyke, F. J., F. A. Murphy, J. K. Murphy, et al. 1974. Effect of surveillance on the number of hyster- ectomies in the province of Saskatchewan. N. Engl. J. Med. 296:1326-1328. Eddy, D. M. 1980. Screening for Cancer: Theory, Analysis and Design. Englewood Cliffs, N.J.: Pren- tice-Hall. ASSESSING MEDICAL TECHNOLOGY Eddy, D. M. 1981. Appropriateness of cervical cancer screening. Gynecol. Oncol. 12:S168. Eddy, D. M. 1982. Clinical policies and the quality of clinical practice. N. Engl. J. Med. 307:343-347. Emerson, J. D., B. McPeek, and F. Mosteller. 1984. Reporting clinical trials in general surgical jour- nals. Surgery 95:572-579. Evans, R. W. 1983. Health care technology and the inevitability of resource allocation and rationing decisions. J. Am. Med. Assoc. 149:2208-2219. Evenson, R. C., H. Altman, I. W. Sletten, and D. W. Cho. 1975. Accuracy of actuarial and clinical pre- dictions for length of stay and unauthorized absence. Dis. New. Syst. 36:250-252. Eysenck, H. J. 1978. An exercise in mega-silliness. Am. Psychol. 33:517. Farber, M. E., and S. N. Finkelstein. 1979. A cost- benefit analysis of a mandatory premarital rubella- antibody screening program. N. Engl. J. Med. 300:856-859. Feigl, P., N. E. Breslow, J. Laszlo, et al. 1981. The U.S. centralized cancer patient data system for uni- form communication among cancer centers. J. Natl. Cancer Inst. 67:1017-1024. Feinstein, A. R. 1977. Clinical Biostatistics, pp. 214-226. St. Louis: C. V. Mosby. Feinstein, A. R. 1978. Clinical biostatistics XLIV. Clin. Pharmacol. Ther. 24:117-25. Ferguson, M. 1980. The Aquarian Conspiracy. Los Angeles: ]. P. Tarcher, Inc. Fineberg, H. V. 1979. Gastric freezing: A study of diffusion of a medical innovation. Pp. 173-200 in Medical Technology and the Health Care System: A Study of the Diffusion of Equipment-Embodied Technology. Washington, D.C.: National Academy Press. Fineberg, H. V., R. Bauman, and M. Sosman. 1977. Computerized cranial tomography: Effect on diagnostic and therapeutic plans. J. Am. Med. Assoc. 238:224-230. Fink, A., J. Kosecoff, M. Chassin, and R. H. Brook. 1984 Consensus methods: Characteristics and guidelines for use. Am. ]. Public Health 74:979-983. Finkler, S. A. 1982. The distinction between cost and charges. Ann. Intern. Med. 96:102-1099. Finney, D. J. 1965. The design and logic of a moni- tor of drug use. J. Chronic Dis. 18:77-98. Finney, D. J. 1966. Monitoring adverse reactions to drugs—its logic and its weakness. Pp. 198-207 in Medical Foundation, International Congress Series No. 115. Proceedings of the European Society for the Study of Drug Toxicity, Vol. VII. Fisher, R. A. 1938. Statistical Methods for Re- search Workers. London: Oliver & Boyd. Fisher, R: A. 1948. Combining independent tests of significance. Am. Stat. 2:30. Fletcher, R. H., and S. W. Fletcher. 1979. Clinical
OCR for page 169
METHODS OF TECHNOLOGY ASSESSMENT research in general medical journals: A 30 year per- spective. N. Engl. J. Med. 301:108-183. Foege, W. H., J. D. Millar, and J. M. Lane. 1971. Selective epidemiologic control in smallpox eradica- tion. Am. J. Epidemiol. 94:311-315. Freiman, J. A., T. C. Chalmers, H. Smith, Jr., and R. R. Kuebler. 1978. The importance of the type II error and the sample size in the design and interpre- tation of the randomized control trial. N. Engl. J. Med. 299: 690-694. Friedman, G. D. 1972. Screening criteria for drug monitoring: The Kaiser-Permanente drug reaction monitoring system. J. Chronic Dis. 25:11-20. Friedman, G. D. 1983a. Rauwolfia and breast cancer: No relation found in long term users age fifty and over. J. Chronic Dis. 36:367-370. Friedman, G. D., and H. K. Ury. 1983b. Screen- ing for possible drug carcinogenicity: Second report of findings. J. Natl. Cancer Inst. 71: 1165- 1175. Friedman, L. F., C. D. Furberg, and D. L. De- Mets. 1981. Fundamentals of Clinical Trials, ix, 225. Boston: John Wright, PSG. Fries, J. F., S. Weyl, and H. R. Holman. 1974. Es- timating prognosis in systemic lupus erythematosus. Am. J. Med. 57:561-565. Fuchs, V. R. 1974. Who Shall Live? Health, Eco- nomics and Social Choice. New York: Basic Books. Galbraith, S. L. 1978. Prognostic factors already known. Scott Med. J. 23:108-109. Galliher, H. P. 1981. Optimizing ages for cervical smear examinations in followed healthy individuals. Gynecol. Oncol. 12:S188. Garber, A. M., V. R. Fuchs, and J. F. Silverman. 1984. Case mix, costs, and outcomes: differences be- tween faculty and community services in a university hospital. N. Engl. J. Med. 310:1231-1237. Gershman, S. T., H. Barrett, J. T. Flannery, et al. 1976. Development of the Connecticut tumor regis- try. Conn. Med. 40:697-701. Gilbert, J. P., B. McPeek, and F. Mosteller. 1977. Progress in surgery and anesthesia: Benefits and risks of innovative therapy. In Costs, Risks, and Benefits of Surgery, J. Bunker, B. Barnes, and F. Mosteller, eds. New York: Oxford University Press. Gittelsohn, A. M., and J. Wennberg. 1977. On the incidence of tonsillectomy and other common surgical procedures. Pp. 91-106 in Costs, Risks and Benefits of Surgery, J. P. Bunker, B. A. Barnes, and F. Mosteller, eds. New York: Oxford University Press. Glaser, E. M. 1980. Using behavioral science strat- egies for defining the state-of-the-art. J. Appl. Behav. Sci. 16:79-82. Glass, G. V. 1976. Primary, secondary and meta- analysis of research. Educ. Res. 5:351-379. Glass, G. V., B. McGaw, and M. L. Smith. 1981. Meta-Analysis in Social Research. Beverly Hills, Calif.: Sage Publications. 169 Globe, S., G. N. Levy, and C. M. Schwartz. 1967. Science, Technology, and Innovation. Contract NSF- C667 (Battelle-Columbus Laboratories). Washing- ton, D. C.: National Science Foundation. Grace, N. D., H. Muench, and T. C. Chalmers. 1966. The present status of shunts for portal hyperten- sion in cirrhosis. Gastroenterology 50:684. Graham, G. F., and F. J. Wyllie. 1979. Prediction of gall-stone pancreatitis by computer. Br. Med. J. 1 :515-517. Groot, L. M. J. 1982. Advanced and Expensive Medical Technology in the Member States of the Eu- ropean Community: Legislation, Policy and Costs. Commissioned by the European Community. Mimeo- graph; Roermond, The Netherlands. Grossman, L. B. 1981. Beads and balloons. Pp. 2-6 in Pacemaker, Vol. 6, No. 3. Detroit: Harper Grace Hospitals. Gustafson, D. H., A. L. Delbecq, M. Hansen, and R. F. Myers. 1975. Design of a health policy research and development system. Inquiry 12:251-262. Haley, R. W., D. Quade, H. E. Freeman, et al. 1980. The Senic Project: Study on the efficacy of non- socomial infection control. Am. J. Epidemiol. 111 :472-485. Hanley, J. A., and B. J. McNeil. 1982. The mean- ing and use of the area under a receiver operating characteristic (ROC) curve. Diag. Radiol. 143:29-36. Health and Social Service Journal, May 30, 1980, pp. 702-704. Hedges, L. V., and I. Olkin. 1980. Vote-counting methods in research synthesis. Psychol. Bull. 88:359- 369. Herbert, T. T., and E. B. Yost. 1979. A compari- son of decision quality under nominal and interacting consensus group formats: The case of the structured problem. Decision Sci. 10:358-370. Herbst, A. L., M. Ulfelder, and D. C. Poskaner. 1971. Association of maternal stilbestrol therapy with tumor appearance in young women. N. Engl. J. Med. 284:878-881. Heyman, A., J. G. Burch, R. Rosati, et al. 1979. Use of a computerized information system in the man- agement of patients with transient cerebral ischemia. Neurology 29:214-221. Hiltz, S. R., and M. Turoff. 1978. The Network Nation. Reading, Mass: Addison-Wesley. Hoaglin, D. C., R. J. Light, B. McPeek, et al. 1982. Data for Decisions. Cambridge, Mass: Abt Books. Hodgkin, J. E., ed. 1979. Chronic Obstructive Pul- monary Diseases: Current Concepts in Diagnosis and Comprehensive Care. Park Ridge, Ill.: American Col- lege of Chest Physicians. Hodgkin, J. E., O. J. Balchum, I. Kass, et al. 1975. Chronic obstructive airway diseases: Current con- cepts in diagnosis and comprehensive care. J. Am. Med. Assoc. 232:1243-1260.
OCR for page 170
170 Hook, E. B., D. M. Schreinemachers, and P. K. Cross. 1981. Use of prenatal cytogenic diagnosis in New York State. N. Engl. J. Med. 305:1410-1413. Horn, S. D., and J. W. Williamson. 1977. Statisti- cal methods for reliability and validity testing: An ap- plication to nominal group judgments in health care. Med. Care 15:922-928. Hovrocks, J. C., D. E. Lambert, W. A. McAdams, et al. 1976. Transfer of computer aided diagnosis of dyspepsia from one geographical area to another. Gut 17:640-644. Hunter, J. E. 1982. Meta-Analysis: Cumulating Research Findings Across Studies. Beverly Hill, Calif.: Sage Publications. IMS America Ltd. 1978. Report of the Joint Com- mission on Prescription Drug Use, Contract 223-78- 3007. Department of Health, Education, and Wel- fare, Food and Drug Administration. Inglefinger, J. A., F. Mosteller, L. A. Thibodeau, and J. H. Ware. 1983. Biostatistics in Clinical Medi- cine, Section 11-6, pp. xiii, 316. New York: Macmil- lan. Institute of Medicine 1981. Evaluating Medical Technologies in Clinical Use. Washington, D.C.: Na- tional Academy Press. Jackson, G. B. 1980. Methods for integrative re- views. Rev. Educ. Res. 50:438-460. Jacoby, I. 1983. Biomedical technology: Informa- tion dissemination and the NIH consensus develop- ment process. Knowledge: Creation, Diffusion. Utili- zation 5:245-261. Jeans, W. D., and A. F. Morris. 1976. The accu- racy of radiological and computer diagnoses in small bowel examinations in children. Br. J. Radiol. 49:665-669, 1976. Jennett, B., G. Teasdale, R. Brackman, et al. 1976. Predicting outcome in individual patients after severe head trauma. Lancet 1: 1031-1034. Jick, H., A. M. Walker, and C. Spriet-Pourra. 1979. Postmarketing follow-up. J. Am. Med. Assoc. 242:2310-2314. Jillson, I. A. 1975. The national drug-abuse policy delphi: Progress report and findings to date. In The Delphi Method: Techniques and Applications, H. Linstone and M. Turoff, eds. Reading, Mass.: Addi- son-Wesley. Kennedy, M. M. 1979. Generalizing from single case studies. Eval. Q. 3:661-678. Kimball, A. M., S. B. Thacker, and M. E. Levy. 1980. Shigella surveillance in a large metropolitan area: Assessment of a passive reporting system. Am. J. Public Health 70:164-166. Klarman, H. E. 1973. Application of cost-benefit analysis of health systems technology. In Technology and Health Care Systems in the 1980's, M. F. Collen, ed. DHEW Publ. (HMS) 73-3016. Washington, D.C.: U.S. Government Printing Office. ASSESSING MEDICAL TECHNOLOGY Knill-Jones, R. 1978. New statistical approach to prediction. Scott. Med. J. 23:102-110. Kolata, G. 1985. Heart panel's conclusions ques- tioned. Science 227:40-41. Koplan, J. P., and L. S. Farer. 1980. Choice of preventive treatment for Isoniazid-resistant tubercu- lous infection. J. Am. Med. Assoc. 244:2736-2740. Koplan, J. P., and S. R. Preblud. 1982. A benefit- cost analysis of mumps vaccine. Am. J. Dis. Child. 136:362-364. Koplan, J. P., S. C. Schoenbaum, M. C. Wein- stein, et al. 1979. Pertussis vaccine: An analysis of benefits, risks and costs. N. Engl. J. Med. 301:906- 911. Kruskal, W., and F. Mosteller. 1980. Representa- tive sampling IV: The history of the concept in statis- tics, 1895-1939. Int. Stat. Rev. 48:169-195. Kurland, L. T., and C. A. Molgaard. 1981. The patient record in epidemiology. Sci. Am. 245:54-63. Lane, J. M., F. L. Ruben, J. M. Neff, et al. 1969. Complications of smallpox vaccination, 1968. Na- tional surveillance in the United States. N. Engl. J. Med. 281:1201-1208. Laszlo, J. 1984. In press. Two long-range clinical data bases are terminated—bang, whimper, or rip- ple? J. Clin. Oncol. Laszlo, J. Healthy registry and clinical data base terminology, with special reference to cancer-regis- tries. Submitted for publication. Laszlo, J., C. Angle, and E. Cox. 1976. The hospi- tal tumor registry: Present status and future prospects. Cancer 38:395. Lave, J., A. Dobson, and C. Walton. 1983. The potential use of Health Care Financing Administra- tion data tapes for health care services research. Health Care Financing Rev. 5:93-98. Layde, P. M., S. D. Von Allman, and G. P. Oak- ley. 1979. Maternal serum alpha-fetoprotein screen- ing: A cost-benefit analysis. Am. J. Publ. Health 69:566-573. Lembcke, P. A. 1956. Medical auditing by scien- tific methods, illustrated by major female pelvic sur- gery. J. Am. Med. Assoc. 162:646-655. Lenfant, C., B. Rifkind, and I. Jacoby. 1985. Heart panel's conclusions (letter). Science 227:582- 583. Light, R. J., ed. 1983. Evaluation Studies Review Annual, Vol. 8. Beverly Hills: Sage Publications. Light, R. J., and P. V. Smith. 1971. Accumulating evidence: Procedures for resolving contradictions among different research studies. Harv. Ed. Rev. 41:429-471. Linstone, H., and M. Turoff, eds. 1975. The Delphi Method: Techniques and Applications. Read- ing, Mass.: Addison-Wesley. Lips, K. J. M., V. D. S. Veer, A. Struyvenberg, and R. A. Geerdink. 1982. Genetic predisposition to cancer in man. Am. J. Med. 73:305-307.
OCR for page 171
METHODS OF TECHNOLOGY ASSESSMENT Louis, P. 1836. Researches on the Effects of Blood Letting on Some Inflammatory Disease and on the In- fluence of Tartarized Antimony and Vesication in Pneumonia (translated by C. G. Putman). Boston: Hilliard Gray and Company. Louis, T. A., H. Fineberg, and F. Mosteller. 1985. Findings for public health from meta-analysis, pp. 1- 20. Annual Review of Public Health. Palo Alto, Cali- fornia: Annual Reviews, Inc. Lusted, L. B. 1969. Perception of roentgen image: Applications of signal detection theory. Bad. Clinics N. Am. 7:435-445. Maclennan, R., C. Muir, and A. Winkler. 1978. Cancer Registration and Its Techniques. Lyon: Inter- national Association for Cancer Research. Mantel, N., and W. Haenszel. 1959. Statistical as- pects of the analysis of data from retrospective studies of disease. J. Natl. Cancer Inst. 22:719-748. McNeil, B. J. 1979. Pitfalls in and Requirements for Evaluations of Diagnostic Technologies in Medical Technology. DHEW Publ. No. (PHS) 79-3254. Urban Institute Conference. McNeil, B. J., P. D. Varady, B. A. Burrows, and S. J. Adelstein. 1975. Cost-effectiveness calculations in the diagnosis and treatment of hypertensive renovas- cular disease. N. Engl. J. Med. 293:216-221. McShane, D. J., J. Porta, and J. F. Fries. 1978. Comparison of therapy in severe systemic lupus erythematosus employing stratification techniques. J. Rheumatol. 5:51-58. Meier, P. 1975. Statistics and medical experimen- tation. Biometrics 31:511-530. Metz, C. E. 1978. Basic principles of ROC analy- sis. Semin. Nucl. Med. 7:283-298. Metz, C. E., D. J. Goodenaugh, and K. Rossman. 1973. Evaluation of receiver operating characteristic curve data in terms of information theory, with appli- cations in radiology. Radiology 109:297-303. Moscovice, I., P. Armstrong, S. Shortell, and R. Bennet. 1977. Health services research for decision- makers: The use of the Delphi technique to determine health priorities. J. Health Politics, Policy and Law 2:388-410. Mosteller, F. M., and R. R. Bush. 1954. Selected quantitative techniques. In Handbook of Social Psy- chology, Vol. I, Theory and Method, G. Lindzey, ed. Cambridge, Mass.: Addison-Wesley. Mulley, A. G., M. D. Silverstein, and J. L. Dien- stag. 1982. Indications for use of hepatitis B vaccine, based on cost-effectiveness analysis. N. Engl. J. Med. 307:644-652. Mullner, R. M., C. S. Byre, and C. L. Kil- lingsworth. 1983. An inventory of the U.S. health care data bases. Review of Public Data Use 11:79- 192. Naisbitt, J. 1982. Megatrends. New York: Warner Books. 171 Nance, F. C., and I. Cohn, Jr. 1969. Surgical judg- ment in the management of stab wounds of the abdo- men. A retrospective and prospective analysis based on a study of 600 stabbed patients. Annals of Surgery 170:569. National Academy of Sciences. 1978. Personnel Needs and Training for Biomedical and Behavioral Research Personnel. Washington, D. C.: National Academy Press. National Academy of Sciences. 1981. Personnel Needs and Training for Biomedical and Behavioral Research. Washington, D. C.: National Academy Press. National Academy of Sciences/Institute of Medi- cine. 1983. Personnel Needs and Training for Bio- medical and Behavioral Research. Washington, D.C.: National Academy Press. National Academy of Sciences. 1985. Sharing Re- search Data, S. E. Fineberg, M. E. Martin, and M. L. Straf, eds. Washington, D.C.: National Academy Press. National Center for Health Care Technology. 1981. Coronary artery bypass surgery. J. Am. Med. Assoc. 2246:1643. National Center for Health Statistics. 1980a. Cata- log of Public Use Data Tapes from the National Cen- ter for Health Statistics. Public Health Service, DHHS Publ. No. (PHS) 81-1213. Washington, D.C.: U.S. Government Printing Office. National Center for Health Statistics. 1980b. Data Systems of the National Center for Health Statistics. Pp. 28-30 in Public Health Service, DHHS Publ. No. (PHS) 80-1247. Washington, D.C.: U.S. Government Printing Office. National Center for Health Statistics. 1980c. Long- Term Health Care: Minimum Data Set. Report of the National Committee on Vital and Health Statistics. Public Health Service, DHHS Publ. No. (PHS) 80- 1158. Washington, D.C.: U.S. Government Printing Office. National Center for Health Statistics. 1980d. The collection and processing of drug information, Na- tional Ambulatory Medieal Care Survey, United States. Prepared by H. Koeh. In Vital and Health Sta- tisties, Series 2, No. 90. Publie Health Serviee. Wash- ington, D.C.: U.S. Government Printing Offiee. National Center for Health Statisties. 1980e. Trends and variations in eesarean section delivery. Prepared by P. J. Placek, S. M. Taffel, and J. C. Kleinman. Pp. 73-36 in Health, United States, 1980. Public Health Service, DHHS Publ. No. (PHS) 81- 1232. Washington: U.S. Government Printing Office. National Center for Health Statistics. 1980f. Uni- form Hospital Discharge Data: Minimum Data Set. Report of the National Committee on Vital and Health Statisties. Publie Health Serviee, DHEW
OCR for page 172
172 Publ. No. (PHS) 80-1157. Hyattsville, Md.: U.S. Gov- ernment Printing Office. National Center for Health Statistics. 1981a. Inpa- tient utilization of short-stay hospitals by diagnosis, United States, 1978. Prepared by E. McCarthy. P. 24 in Vital and Health Statistics, Series 13, No. 55. Pub- lic Health Service, DHHS Publ. No. (PHS) 81-1716. Washington, D.C.: U.S. Government Printing Of- fice. National Center for Health Statistics. 1981b. NMCES Household Interview Instruments: Instru- ments and Procedures 1. Prepared by G. S. Bonham and L. T. Corder. Public Health Service, DHHS Publ. No. (PHS) 81-3280. Washington, D.C.: U.S. Government Printing Office. National Center for Health Statistics. 1981c. Uni- form Ambulatory Medical Care: Minimum Data Set. Report of the National Committee on Vital and Health Statistics. Public Health Service, DHHS Publ. No. (PHS) 81-1161. Hyattsville, Md.: U.S. Govern- ment Printing Office. National Center for Health Statistics. 1983a. Data Systems of the National Center for Health Statistics, pp. 63-75. Public Health Service, DHHS Publ. No. (PHS) 80-1247. Washington, D.C.: U.S. Government Printing Office. National Center for Health Statistics. 1983b. Drug utilization in office-based practices: Summary of find- ings. National Ambulatory Medical Care Survey. United States, 1980. Prepared by H. Koch. In Vital and Health Statistics, Series 13, No. 65. Public Health Service, DHHS Publ. No. (PHS) 83-1726. Washing- ton, D.C.: U.S. Government Printing Office. National Center for Health Statistics. 1983c. Health, United States, 1983, p. 13. Public Health Ser- vice, DHHS Publ. No. (PHS) 84-1232. Washington, D.C.: U.S. Government Printing Office. ASSESSING MEDICAL TECHNOLOGY National Commission for the Protection of Human Subjects. 1978. Report and Recommendations. Wash- ington, D.C.: U.S. Government Printing Office. National Institutes of Health. 1983. Liver trans- plantation consensus development. Conference Sum- mary. Volume 4, Number 7. National Institutes of Health. 1984. Diagnostic ul- trasound imaging in pregnancy. Consensus Develop- ment Conference. Volume 5, Number 1. NHLBI Angioplasty, Kent et al. 1982. Percuta- neous transluminal coronary angioplasty: Report from the registry of the National Heart, Lung, and Blood Institute. Am. J. Cardiol. 49:2011-2018. Needleman, H. L, S. K. Geiger, and R. Frank. 1985. Lead and IQ scores: a reanalysis. Science 227:701-702. Needleman, H. L., C. Gunnoe, A. Leviton, et al. 1979. Deficits in psychologic and classroom perfor- mance of children with elevated dentine lead levels. N. Engl. J. Med. 300:689-695. Neustadt, R., and H. V. Fineberg. 1983. The Epi- demic That Never Was: Decision Making in the Swine Flu Scare. 1983. New York: Vintage Books. Neutra, R. R., S. E. Fienberg, S. Greenland, and E. A. Friedman. 1978. Effect of fetal monitoring on neonatal death rates. N. Engl. J. Med. 299:324-326. Office of Health Research Statistics and Technol- ogy. 1981. U.S. Department of Health and Human Services. Transsexual surgery. Assessment Report Se- ries 1(4). Washington, D.C.: U.S. Department of Health and Human Services. Office of Technology Assessment, U.S. Congress. 1978a. Assessing the efficacy and safety of medical technologies. Washington, D.C.: U.S. Government Printing Office. Office of Technology Assessment, U.S. Congress. 1978b. Policy implications of the computed tomo- graphy (CT) scanner. GPO Stock No. 052-003-00565- 4. Washington, D.C.: U.S. Government Printing Of- fice. Office of Technology Assessment, U.S. Congress. 1980a. The implications of cost-effectiveness analysis of medical technology. Stock No. 051-003-00765-7. Washington, D.C.: U.S. Government Printing Of- National (~enter for Health Statistics. lusted. Pro- cedures and questionnaires of the National Medical Care Utilization and Expenditure Survey. Prepared by G. S. Bonham. In Series A, Methodological Report No. 1. Public Health Service, DHHS Publ. No. 83- 20001. Washington, D.C.: U.S. Government Printing Office. Notional Center for Health Statistics. 1983e. Utili- fice. zation of shortstay hospitals: United States, 1981 an- Office of Technology Assessment, U.S. Congress. nual summary. In Vital and Health Statistics, Series 1980b. The implications of cost-effectiveness analysis 13, No. 72. Public Health Service, DHHS Publ. No. of medical technology/background paper #1: Method- (PHS) 83-1733. Washington D.C.: U.S. Government ological issues and literature review. Washington, Printing Office. D.C.: U.S. Government Printing Office. National Center for Health Statistics. 1983f. Varia- Office of Technology Assessment, U.S. Congress. tion in use of obstetric technology. Prepared by J. C. 1980c. The implications of cost-effectiveness analysis Kleinman, M. Cooke, S. Machlin, and S. S. Kessel. of medical technology/background paper #3: The effi- Pp. 63-75 in Health, United States, 1983. Public cacy and cost effectiveness of psychotherapy. Wash- Health Service, DHHS Publ. No. (PHS) 84-1232. ington, D.C.: U.S. Government Printing Office. Washington, D.C.: U.S. Government Printing Of- Office of Technology Assessment, U.S. Congress. rice. 1981a. Cost effectiveness of influenza vaccination.
OCR for page 173
METHODS OF TECHNOLOGY ASSESSMENT Washington, D.C.: U.S. Government Printing Of- fice. Office of Technology Assessment, U.S. Congress. 1981b. The implications of cost-effectiveness analysis of medical technology/background paper #2: Case studies of medical technologies/care study #2: The feasibility of economic evaluation of diagnostic proce- dures: The case of CT scanning. Washington, D.C.: U.S. Government Printing Office. Office of Technology Assessment, U.S. Congress. 1981c. The implications of cost-effectiveness analysis of medical technology/background paper #2. Case studies of medical technologies/case study #1: Formal analysis, policy formulation, and end-stage renal dis- ease. Washington, D.C.: U.S. Government Printing Office. Office of Technology Assessment, U.S. Congress. 1981d. The implications of cost-effectiveness analysis of medical technology/background paper #2: Case studies of medical technologies/case study #3: Screen- ing for colon cancer: A technology assessment. Wash- ington, D.C.: U.S. Governmment Printing Office. Office of Technology Assessment, U.S. Congress. 1982a. Postmarketing Surveillance of Prescription Drugs. Washington, D.C.: U.S. Government Print- ing Office. Office of Technology Assessment, U.S. Congress. 1982b. Strategies for Medical Technology Assessment. Washington, D.C.: U.S. Government Printing Of- fice. Office of Technology Assessment, U.S. Congress. 1982c. Technology and Handicapped People, p. 22. OTA-H-179. Washington, D.C.: U.S. Government Printing Office. Office of Technology Assessment, U.S. Congress. 1983a. Abstracts of Case Studies in the Health Tech- nology Case Study Series. OTA-P-225. Washington, D.C.: U.S. Government Printing Office. Office of Technology Assessment, U. S. Congress. 1983b. Variations in Hospital Length of Stay: Their Relationship to Health Outcomes. Health Technology Case Study 24. OTA-HCS-24. Washington, D.C.: U.S. Government Printing Office. Olsen, S. A. 1982. Group Planning and Problem Solving Methods in Engineering Management. New York: John Wiley & Sons. Patrick, K. M., and R. Woolley. 1981. A cost-ben- efit analysis of immunization for pneumococcal pneu- monia. J. Am. Med. Assoc. 245:473-477. Pauker, S., and J. Kassirer. 1975. Therapeutic de- cision making: A cost-benefit analysis. N. Engl. J. Med. 293:229-234. Pearson, E. S. 1938. The probability integral trans- formation for testing goodness of fit and combining independent tests of significance. Biometrika 30: 134- 148. Pearson, E. S. 1950. On questions raised by the 173 combination of tests based on discontinuous distribu- tions. Biometrika 37:383-398. Perry, S., and J. T. Kalberer. 1980. The NIH Con- sensus-Development Program and the assessment of health-care technologies: The first two years. N. Engl. J. Med. 303:169-172. Pillemer, D. B., and R. J. Light. 1980. Synthekiz- ing outcomes: How to use research evidence from many studies. Harvard Educ. Rev. 50:176-195. Pocock, S. J. 1976. The combination of random- ized and historical controls in clinical trials. J. Chronic Dis. 29:175-188. Policy Research Incorporated. 1975a. National Drug Abuse Policy Delphi Study: Questionnaires and Summaries of Results. Baltimore. Policy Research Incorporated. 1975b. National Drug Abuse Policy Report: Final Report. Baltimore. Policy Research Incorporated. 1977. A Compre- hensive Study of the Ethical, Legal, and Social Impli- cations of Advances in Biomedical and Behavioral Re- search and Technology: Summary of the Final Report. Baltimore. Policy Research Incorporated. 1979a. Medical Practice Information Demonstration Project: Depres- sion Project: First Series of Instruments. Baltimore. Policy Research Incorporated. 1979b. Medical Practice Information Demonstration Project: Bipolar Disorder: A State-of-the-Science Report. Baltimore. Pollack, E. S. 1982a. Monitoring cancer incidence and patient survival in the United States. Proceedings, Social Statistics Section. Washington, D.C.: Ameri- can Statistical Association. Pollack, E. S. 1982b. SEER cost study. Printed for private distribution, Natonal Cancer Institute. President's Commission for the Study of Ethical Problems in Medicine and Biomedical and Behavioral Research. 1983. Summing up: Final Report on Studies of the Ethical and Legal Problems in Medicine and Biomedical and Behavioral Research. Washington, D.C.: U.S. Government Printing Office. Principal investigators of CASS and their associ- ates. 1981. National Heart, Lung, and Blood Institute Coronary Artery Surgery Study. Circulation 63(suppl. I):1-81. Queram, C. J. 1977. Cancer Registries and Report- ing Systems in the United States. Madisdn, Wis.: De- partment of Health and Social Services, Bureau of Health Statistics, Division of Health. Rand Corporation. 1983. Submission to the Office of Management and Budget of Supporting Statement and Data Collection Instruments for Assessing the Ef- fectiveness of the NIH Consensus Development Pro- gram. Ransahoff, D. F., and A. R. Feinstein. 1978. Prob- lems of spectrum and bias in evaluating the efficacy of diagnostic tests. N. Engl. J. Med. 299:926-930. Re, R., R. Novelline7 M. T. Escourrou, et al. 1978.
OCR for page 174
174 Inhibition of angiotensin-converting enzyme for diag- nosis of renal artery stenosis. N. Engl. J. Med. 298:582-586. Remington, R. D. 1976. Recommendations. Pp. 141-150 in Assessing Drug Reactions—Adverse and Beneficial, Vol. 7, Philosophy and Technology of Drug Assessment, F. N. Allen, ed. Washington, D.C.: The Interdisciplinary Communications Associates. Rennie, D. 1981. Consensus Statements. N. Engl. J. Med. 304:665-666. Richart, R. H. 1974. Evaluation of a hospital com- puter system, in Hospital Computer Systems, M. Col- len, ed. New York: John Wiley & Sons, Inc. Richart, R. M. 1981. Discussion of session III: Screening of cervical neoplasia. Gynecol. Oncol. 12:S212. Robertson, A., and M. H. Zweig. 1981. Use of re- ceiver operating characteristic curves to evaluate the clinical performance of analytical systems. Clin. Chem. 77:1568-1574. Romm, F. J., and B. S. Hulka. 1979. Developing criteria for quality of care assessment: Effect of the Delphi technique. Health Services Res. 14:309-312. Boos, L. L., Jr. 1979. Alternative designs to study outcomes. Med. Care 17:1069-1087. Rosati, R. A., A. G. Wallace, and E. A. Stead. 1973. The way of the future. Arch. Intern. Med. 131:285. Rosenthal, R. 1978a. Combining results of inde- pendent studies. Psychol. Bull. 85:185-193. Rosenthal, R., and D. B. Rubin. 1978b. Interper- sonal expectancy effects: The first 345 studies. Behav- ioral and Brain Sciences 3:377-415. Russell, L. B. 1979. Technology in Hospitals: Med- ical Advances in Their Diffusion. Washington, D.C.: The Brookings Institution. Sackman, H. 1975. Delphi Critique. Lexington, Mass.: Lexington Books. Sanders, C. A. 1979. Medical technology and the health care system: A study of the diffusion of equip- ment-embodied technology, by the Committee on Technology and Health Care. Washington, D.C.: National Academy of Sciences. Schaffarzick, R., cited by Wennberg and Git- telsohn, 1973. Schneiderman, M. A. 1966. Looking backward: Is it worth the crick in the neck? Or: pitfalls in using ret- rospective data. Am. J. Roentgenol. Radium Ther. Nucl. Med. 96:230-235. Schoenbaum, S., J. N. Hyde, L. Bartoshesky, and K. Crampton. 1976a. Benefit-cost analysis of rubella vaccination policy. N. Engl. J. Med. 294:306-310. Schoenbaum, S. C., B. J. McNeil, and J. Kavet. 1976b. The swine-influenza decision. N. Engl. J. Med. 295:759-765. Schwartz, J. S., P. J. Weinbaum, C. Nesler, et al. 1983. Assessment of tests to identify infants at high ASSESSING MEDICAL TECHNOLOGY risk of respiratory distress syndrome (RDS) using re- ceiver operating characteristic (ROC) curve analysis. Med. Decision Making 3:365. Scitovsky, A. A. 1979. Changes in the use of ancil- lary services for ~common' illness. Pp. 39-56 in Medi- cal Technology: The Culprit Behind Health Costs? Proceedings of the 1977 Sun Valley Forum on Na- tional Health Insurance, Stuart H. Altman and Robert Blendon, eds. DHEW Publ. No. (PHS) 79- 3216. New York: John Wiley. Shapiro, M., et al. 1983a. Benefit-cost analysis of antimicrobial prophylaxis in abdominal and vaginal hysterectomy. J. Am. Med. Assoc. 249:1290-1294. Shapiro, S. H., and T. A. Louis. 1983. Clinical Trials: Issues and Approaches, pp. ix, 209. New York: Marcel Dekker. Sherins, R. J., C. L. Olivery, and J. L. Ziegler. 1978. Gynecomastia and gonadal dysfunction in ado- lescent boys treated with combination chemotherapy for Hodgkin's disease. N. Engl. J. Med. 299:12-16. Showstack, J., et al. 1981. Evaluating the costs and benefits of a diagnostic technology: The case of upper gastrointestinal endoscopy. Med. Care 19:498-509. Shur, R. D. 1981. Estimating the natural history of cancer from cross-sectional data. Ph.D. Dissertation, Engineering-Economic Systems, Stanford University. Shwartz, M. 1978. An analysis of the benefits of se- rial screening for breast cancer based upon a mathe- matical model of the disease. Cancer 51:1550. Siegel, E. R., 1980. Use of computer conferencing to validate and update NLM's hepatitis knowledge base, in The Future of Electronic Communications, M. M. Henderson, and M. J. MacNaughton, eds. Washington, D.C.: American Association for the Ad- vancement of Science. Snow, J. 1847. On the Inhalation of the Vapor of Ether in Surgical Operations. (Reproduced by Lea & Febiger, Philadelphia, 1959.) Stange, P., and A. T. Summer. 1978. Predicting treatment costs and life expectancy for end-stage renal disease. N. Engl. J. Med. 298:372-378. Stason, W,. and M. Weinstein. 1977. Allocation of resources to manage hypertension. N. Engl. J. Med. 296:732. Steinberg, D. 1985. Heart panel's conclusions (let- ter). Science 227:582. Swain, R. W. 1981. Health Systems Analysis. Co- lumbus, Ohio: Grid Publishing. Swets, J. A. 1979. ROC analysis applied to the evaluation of medical imaging techniques. Invest. Ra- diology 14:109-121. Swets, J. D., and R. M. Pickett. 1982. Evaluation of Diagnostic Systems. New York: Academic Press. Tatro, D. S., T. N. Moore, and S. N. Cohen. 1979. Computer-based system for adverse drug detection and prevention. Am. J. Hosp. Pharm. 36:198-201.
OCR for page 175
METHODS OF TECHNOLOGY ASSESSMENT Teasdale, G. 1978. Prediction in action. Scott. Med. J. 23:111. The Lancet Commission on Anesthesia. 1893. Lan- cet 1:629-638, 693-708, 761-776, 899-914, 971-978, 1236-1240, 1479-1498. Thornell, C. A. 1981. Comparison of Strategies for the Development of Process Measures in Emergency Medical Services, NCHSR Research Summary Series. Hyattsville, Md.: National Center for Health Services Research. Turoff, M., and S. R. Hiltz. 1982. Computer sup- port for group versus individual decisions. IEEE Trans. Comm. COM-30(1~:82-91. U.S. Consumer Product Safety Commission. 1982. National Electronic Injury Surveillance System. NEISS Data Highlights 6:1-4. U.S. Public Health Service. 1964. Smoking and Health. Report of the Advisory Committee to the Sur- geon General of the Public Health Service. Washing- ton, D.C.: U.S. Department of Health, Education, and Welfare. Warner, K. E., and R. C. Hutton. 1981. Cost- benefit and cost-effectiveness analysis in health care. Med. Care 19:498-509. Warner, K. E., and B. R. Luce. 1982. Cost-Benefit and Cost-Effectiveness Analysis in Health Care. Ann Arbor, Mich.: Health Administration Press. Weinstein M., and W. Stason. 1976. Hyperten- sion: A Policy Perspective. Cambridge, Mass.: Har- vard University Press. Weinstein, M. C., and H. V. Fineberg. 1980. Clin- ical Decision Analysis. Philadelphia: W. B. Saunders. Weinstein, M. C., and B. Stason. 1977. Founda- tions of cost-effectiveness analysis for health and med- ical practices. N. Engl. J. Med. 296:716-721. Wennberg, J. 1984. Dealing with medical practice variations: A proposal for action. Health Affairs 3:6- 32. Wennberg, J., and A. Gittlesohn. 1973. Small area variations in health care delivery. Science 182:1102- 1108. Wennberg, J. E. 1981. A strategy for the post- marketing surveillance and evaluation of health care technology. Department of Community and Family Medicine, Dartmouth Medical School. Wennberg, J. E., J. P. Bunker, and B. Barnes. 1980. The need for assessing the outcome of common medical practices. Ann. Rev. Public Health 1:277- 295. 175 White, C., and J. C. Bailar. 1956. Retrospective and prospective methods of striding association in medicine. Am. J. Public Health 46:35-44. White, J. J., and N. W. Axnick. 1975. The benefits from 10 years of measles immunization in the United States. Public Health Rep. 90:205-207. Willems, J. S., R. Sanders, A. Riddiough, and C. Bell. 1980. Cost-effectiveness of vaccination against pneumococcal pneumonia. N. Engl. J. Med. 303:553- 559. Williamson, J. W. 1978. Formulating priorities for quality assurance activity. J. Am. Med. Assoc. 239:631-637. Wilson, D. H. 1979a. The acute abdomen in the accident and emergency department. Practitioner 222:480-485. Wilson, S. 1979b. Explorations of the usefulness of case study evaluations. Evaluation Q. 3:446-59. World Health Organization. 1976a. WHO Hand- book for Standardized Cancer Registrations. Geneva. World Health Organization. 1976b. ICD-O, Inter- national Classification of Diseases for Oncology, 1st ed. Geneva. Wortman, P. M. 1983. Evaluation research: A methodological perspective. Ann. Rev. Psychol. 34:223-260. Wortman, P. M., and L. Saxe. 1982a. Assessment of medical technology: Methodological considerations (Appendix C). In Strategies for Medical Technology Assessment, prepared by the Office of Technology As- sessment, U.S. Congress. Washington, D.C.: U.S. Government Printing Office. Wortman, P. M., and A. Vinokur. 1982b. Evalua- tion of NIH Consensus Development Process. Phase I: Final Report. Ann Arbor, Mich.: Center for Research on Utilization of Scientific Knowledge, University of Michigan. Yin, R. K. 1984. Cast Study Research, Deisgn and Methods. Beverly Hills: Sage Publications. Young, J. L., Jr., A. Asire, and E. Pollock. 1976. SEER Program: Cancer Incidence and Mortality in the United Stats 1973-1976. DHEW Pub. No. (NIH) 78-1837. Bethesda, Md: National Cancer Institute. Young, J. L., C. L. Percy, A. J. Asire, eds. 1981. SEER Program: Incidence and Mortality Data, 1973- 77. National Cancer Institute Monograph 57, USG 80, NIH Pub. No. 81-2330. Washington, D.C.: Na- tional Cancer Institute.
Representative terms from entire chapter: