Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 70
_, Methods of Technology Assessment
As Chapter 1 indicates, technology as- a foundation for building a system of tech-
sessment offers the essential bridge be- nology assessment for the nation.
tween basic research and development and
prudent practical application of medical
technology. We have a substantial body of
methods that can be applied to the various
tasks of assessment, and their availability
makes possible the acceptance, modifica-
tion, or rejection of new technologies on a
largely rational basis. That rationality,
however, depends on many factors that go
well beyond safety and efficacy, including,
among other components, economics, eth-
ics, preferences of patients, education of
physicians, and diffusion of information.
The methods that have been developed can
take some account of most of these compo-
nents, although combining the results for
the components is a major task and one
that is far from settled or solved. The exis-
tence of these assessment methods provides
The outline, introduction, and conclusions of this
chapter were developed by Frederick Mosteller. The
various sections of the chapter were drafted primarily
by other authors identified at the opening of each sec-
tion.
70
Most innovations in health care technol-
ogy rest on some theoretical ideas held by
the innovators. These ideas inevitably
range in strength from very well informed
to hopeful speculation. Beyond this, a few
innovations are purely empirical in the
sense that someone has noticed that the
technology seemed to work, even though
no underlying mechanism was proposed or
understood. In considering medical tech-
nologies, no matter how strong or weak the
theoretical justification, experience must
be decisive. If in practice the innovation is
clearly better or clearly worse than existing
technologies, then the innovation deserves
adoption or rejection. It is known from
much experience that merely having a
good idea, a good theory, or a constructive
observation is not enough because there are
so many unexpected interfering variables
that may thwart the innovation and the in-
novator. Learning from controlled experi-
ence is central to progress in health care.
Learning from experience itself without
formal planning often presents great diffi-
culties and sometimes leads to long-main-
OCR for page 71
METHODS OF TECHNOLOGY ASSESSMENT
tained fallacies, partly because of the lack
of control of variables. This method is slow
and expensive unless the effects are huge.
Planning and analysis and scientific testing
provide ways to strengthen the learning
process. This chapter describes a number
of techniques or methodologies that help to
systematize learning from experience in
health care technology.
Few people are acquainted with more
than a few of the methods used for assess-
ment. Usually investigators are acquainted
with the few methods most frequently used
in their own specialties. Consequently, it
seems worthwhile to give a brief descrip-
tion of the more widely used methods and
what they are most useful for studying.
For direct attack on evaluation through
data acquisition, clinical trials are highly
regarded. For generating hypotheses, the
case study and the series of cases have spe-
cial value. Registries and data bases some-
times produce hypotheses, sometimes they
help evaluate hypotheses, and sometimes
they aid directly in the treatment of pa-
tients. Sample surveys excel in describing
collections of patients, health workers,
transactions, and institutions.
Epidemiological and surveillance stud-
ies, although not synonymous, are well
adapted to identifying rare events that
may be caused by adverse effects of a tech-
nology.
Quantitative synthesis (meta-analysis)
and group judgment methods give us ways
to summarize current states of knowledge
and sometimes to predict the future. Simi-
larly, cost-effectiveness analysis (CEA) and
cost-benefit analysis (CBA) offer ways of
introducing costs and economics into these
assessments. Modeling provides a way to
simulate the future and still include com-
plicated features of the real life process and
to see what variables or parameters seem to
produce the more substantial effects.
When backed with strong, although lim-
ited, empirical investigation, it may add
much breadth to an evaluation.
71
Sometimes what is learned to be true in a
scientific laboratory may not, at first, be
successfully applied in practical circum-
stances. Myriad reasons can explain this:
the new technique is not correctly applied,
or to the right kinds of cases, or it is not ap-
plied assiduously enough, or too assidu-
ously, etc. This idea in medical contexts is
captured in the terms efficacy and effec-
tiveness. Efficacy refers to what a method
can accomplish in expert hands when cor-
rectly applied to an appropriate patient;
effectiveness refers to its performance in
more Unmoral routine applications. The rel-
evance of these ideas here is that some of
the methods presented below are more nat-
urally adaptable to assessing one of these or
the other. The reader will probably appre-
ciate, for example, that surveillance and
data banks point toward assessing effec-
tiveness, and most randomized clinical
trials point toward assessing efficacy.
Although randomized clinical trials of-
fer the strongest method of assessing the ef-
ficacy of a new therapy, it is recognized
that it is not possible to have randomized
trials for every version of every innovation.
However desirable that might be, it is not
feasible. Consequently, other methods of
assessment are often going to be depended
on; of course, some technologies actually
require other methods. This in turn means
that steps need to be taken to strengthen
the other methods. These steps have two
forms. First, where possible, apply the
known ways of improving studies, such as
observational studies (for example, have a
careful protocol, use random samples, use
blindness where possible, and so on). Sec-
ond, many of these methods could be im-
proved if research were carried out to find
new ways to improve them. Therefore,
specific research that could lead to getting
stronger results from the weaker methods is
often suggested.
Possibly, research will find that particu-
lar methodologies are best when applied to
special classes of treatments. For example,
OCR for page 72
72
perhaps noninvasive drugs and devices
could be handled in one way and invasive
methods in another. Perhaps data banks
and registries could offer good results from
some class of problems. Answers to such
questions are not now available.
At the same time that the need for im-
proving the weaker methods is recognized,
it is also recognized that the methods al-
ready in existence are not sufficiently often
applied. The Office of Health Technology
Assessment (OHTA) evaluates the safety
and effectiveness of new or as yet unestab-
lished medical technologies and proce-
dures that are being considered for cover-
age under Medicare. Requests for these
evaluations come from the Health Care Fi-
nancing Administration (HCFA). OHTA
carries out its evaluations by reviewing the
literature and by getting advice from vari-
ous agencies and professional organiza-
tions. The information so acquired is syn-
thesized to reach some conclusion. OHTA
does not gather primary data itself. Again
and again, it turns out, and OHTA notes,
that the primary data are almost nonexis-
tent and that primary data would be re-
quired to reach a well-informed conclu-
sion. In advising HCFA about coverage for
various medical technologies, OHTA pre-
pared 65 reports in the years 1982, 1983,
and 1984. Lasch (1985) reviewed these re-
ports to see what the state of the informa-
tional base on safety and efficacy seemed to
be (K. E. Lasch, Synthesizing in HRST Re-
ports, unpublished report, Harvard School
of Public Health, 1985~. Lasch sorted the
reports into four categories, as follows:
1. The technology enjoyed widespread
use and was considered an established
technology.
2. The data base for the technology was
insufficient; there was a call for more stud-
ies and better research designs, or accuracy
was questioned for diagnostic tests.
3. The data base was sufficient; the
technology was not recommended.
ASSESSING MEDICAL TECHNOLOGY
4. The technology was outmoded, not
routinely used, and not an established
therapy.
After the studies were categorized for
the 3 years, Lasch found the results shown
in Table 3-1. The percentage values of the
results are similar from year to year. The
category of insufficient data stands out.
In noting that 69 percent of these assess-
ments have insufficient data to reach a sat-
isfactory conclusion, it should not be as-
sumed that the technologies in the other
categories have always been evaluated on
the basis of strong data. The categories
were chosen to generate a clear set when
the evidence was inadequate. The first cat-
egory of widespread use may also include
poorly evaluated technologies. This study,
then, offers a clear message that many
technologies that physicians wish to use
have not been adequately evaluated. Simi-
larly, at the consensus conferences, speak-
ers frequently point out the lack of primary
data (National Institutes of Health tNIH],
1983, 1984~. Thus, the most important
need is to gather more primary data.
As we report later in this chapter, the
Office of Technology Assessment (OTA;
1980a) polled data analysts who conduct
cost-effectiveness and cost-benefit analyses
of health care technologies and found lack
of information to be a uniformly signifi-
cant problem.
More primary research is needed, and
this will have to be led in part by research
physicians with training in quantitative
methods and supported by doctoral-level
epidemiologists and biostatisticians. All
three groups are in short supply (National
Academy of Sciences, 1978, 1981, 1983~.
At the least the development of methods
will also require epidemiologists and bio-
statisticians. Therefore, on both grounds,
we will need funds for training research
personnel.
Many assessment methods are described
in some detail in the sections that consti-
OCR for page 73
METHODS OF TECHNOLOGY ASSESSMENT
73
TABLE 3-1 Distribution of Technologies for Years 1982, 1983, and 1984 into Four Typesa
when Reviewed by OHTA for HCFA
Percentages for Each Yearb
Category 1982 1983 1984 Total
Widespread use 16 (4) 19 (4) 21 (4) 18 (12)
Insufficient data 68 (17) 76 (16) 63 (12) 69 (45)
Data sufficient;
technology not effective 4 (1) 0 (O) O (O) 2 (1)
Technology not used or outmoded 12 (3) is (1) 16 (3) 11 (7)
Totals 100 (25) 100 (21) 100 (19) 100 (65)
aEach of the 65 reports was assigned to one of the above categories based on a reading of the summary and
discussion sections. Coding of the 65 reports revealed that the four categories were mutually exclusive; each report
fell neatly into one of the categories.
bNumbers of studies shown in parentheses.
tote the main body of this chapter. Unless
explicitly interested in research methods,
some readers may wish to scan cursorily
through the chapter.
Most sections follow a pattern that opens
with a brief description of the method, fol-
lowed by typical purposes and uses and by
a subsection addressing capabilities and
limitations, including some remarks on
ways of strengthening the method in prac-
tical use. Sometimes a final subsection dis-
cusses research that could be done that
might lead to improvements in the
method.
RANDOMIZED CLINICAL TRIALS*
The randomized clinical trial (RCT) is a
method of comparing the relative merits
(and shortcomings) of two or more treat-
ments tested in human subjects. A well-de-
signed and -executed RCT is widely re-
garded as the most powerful and sensitive
tool for the comparison of therapies, diag-
nostic procedures, and regimens of care.
More broadly, the RCT can be regarded
as an unusually reliable method for learn-
ing from experience; its success lies in
structuring that experience so as to fore-
*This section was drafted by Lincoln E. Moses.
close many sources of ambiguity. In the
health sciences the method is applied not
only in comparing therapies but also diag-
nostic methodologies, ways of imparting
information to patients, and regimens of
care (e.g., home care versus critical care
units for certain heart patients). In gen-
eral, if alternative ways of accomplishing
an aim are in competition, the RCT may
be the best technique for resolving their
relative merits.
Notice that comparison is at the heart of
the method. A clinical trial is not a device
for ascertaining the health consequences of
a toxic substance in food or for elucidating
the etiology of a disease. It is a method for
comparing interventions that are applied
and controlled by the investigator. The
clinical trial becomes an RCT if there is a
deliberate introduction of randomness into
the assignment of patients (eligible for
both, or all, of the treatments) to treat-
ment A, treatment B. etc. The reasons for
such a method of assignment are discussed
below.
Hereafter, when referring to an RCT, it
is contemplated that it satisfies these two
conditions:
1. No subject is admitted without hav-
ing been judged to be equally suitable to
OCR for page 74
74
receive any one of the treatments being of-
fered to the subject's class of patients.
2. No subject is admitted without hav-
ing volunteered to receive either treat-
ment, as may be assigned.
Practical Problems of Comparing
Treatments
Two factors make it intrinsically diffi-
cult to compare different treatments. First,
the subjects receiving the treatments usu-
ally are different people, so differences
found between the treatments could be due
to differences among the subjects in the
groups. If the groups differ in any system-
atic way (whether recognized or not), the
treatment comparison may be biased; bias
can exaggerate, nullify, or reverse true dif-
ferences. Second, even if the treatments
could be compared in the same patients (as
sometimes happens), the contrast between
the treatments will vary from one patient
to another, producing uncertainty in the
overall assessment. This is the problem of
variability. Large samples can reduce the
disturbance of variability but do not help
with bias.
If two treatment groups are differently
constituted, then bias in the treatment
comparison must be regarded as likely.
The phrase "differently constituted" ap-
plies, for example, where the treatment
groups are (1) admitted to the study by dif-
ferent means, (2) treated in different
places, at different times, or by different
sets of practitioners; (3) assessed by differ-
ent groups; or (4) analyzed and reported
by different teams.
Randomization in a clinical trial is
aimed at preventing bias. Two characteris-
tic features are essential to realizing that
aim.
First, the study is conducted under a
protocol that makes explicit exactly what
questions are to be studied, what treat-
ments are to be applied; and how, to what
kind of patients, when, and where. It also
specifies how assessment of outcomes will
ASSESSING MEDICAL TECHNOLOGY
be done and how statistical analyses will be
conducted.
Second, the RCT calls for assignment of
the respective treatments to each eligible
patient admitted to the study by means of a
random choice. The effect of this is to en-
sure that the two treatment groups are not
"differently constituted", indeed, they are
brought into being as random subsets of a
singly constituted group which is opera-
tionally defined by the protocol.
The protocol-controlled RCT is even
stronger whenever knowledge of which
treatment a patient has received is
screened from participants (patients, treat-
ing physicians, outcome assessors). A result
of such "blinding" is to ensure that placebo
effects remain randomly assorted to the
treatments. Another result is to prevent
differential decisions about care during the
study. It is especially important that those
assessing outcomes be blind to the type of
treatment—unless the outcome is entirely
objective, e.g., length of survival. In some
cases, blinding of physicians may not be
possible, such as when a medical modality
is being compared with a surgical one.
The Protocol
The protocol is a written prescriptive
document that spells out the purposes and
rationale of the trial and how it will be
conducted. Specifics include the criteria of
eligibility for inclusion of patients in the
trial and criteria for exclusion—and de-
scription of treatments, adjuvant therapy,
outcome measurements, patient follow-
up, and statistical analyses to be per-
formed. The protocol also specifies the
numbers of patients to be entered and the
mechanics of randomization. The protocol
is both a planning document and a proce-
dures manual. The aim is to provide trust-
worthy answers at the end of the study to
the following questions: What treatments
were applied, to what kinds of patients,
with what results? What do the results
mean?
OCR for page 75
METHODS OF TECHNOLOGY ASSESSMENT
Provisions for blindness and for the or-
der in which processes are to be performed
can be central to the validity of a study and
to the value of the protocol that governs it.
If the decision to enter each patient into
the trial is made in the knowledge of which
treatment the next patient will receive,
then ample opportunity for building up
noncomparable treatment groups is at
hand, so the protocol should not use alter-
nate-patient assignment to the treatments.
If a rather subjective diagnostic test W as-
sesses a condition thought to be related to
another test V, then W measured after V is
not the same as W measured before V; it
may be important that the protocol specify
the order in which they are to be done. The
careful protocol attempts to specify in ad-
vance all procedural steps that may mate-
rially affect execution of the trial and inter-
pretation of its results.
A well-conducted RCT requires not only
a good protocol but also that the trial be
carried out in accordance with it. The pro-
tocol may call for specific steps to check on
(and promote) protocol adherence. Staging
and laboratory analyses may be checked by
introducing (blindly) occasional standard
specimens. Samples of study records may
be checked back to more basic clinical rec-
ords. Visits by monitors, combined with
audit, may be routinely conducted in
multicenter studies.
The protocol also has the character of a
compact among the participating investi-
gators, relevant human subjects commit-
tees, and funding sponsors. This contrac-
tual character lends stability to a study
over its lifetime, helping to supply definite
answers to the questions concerning what
was done, to what kinds of patients, and
with what results.
Random Assignment to Treatment
The primary reason for random assign-
ment is to prevent bias by breaking any
possible systematic connection of one treat-
ment or the other with favorable values of
75
interfering variables (whether recognized
or not). A fuller appreciation of this princi-
ple may be gained by considering two al-
ternative modes of treatment comparison
that are sometimes advocated. The first is
the use of historical controls, the second is
the use of statistical procedures to adjust
for treatment group differences in the im-
portant interfering variables.
The historically controlled trial (HCT)
compares outcomes on a new treatment to
outcomes in previous (historical) cases
from the same setting. The motivation is to
arrive at decisions sooner by assigning all
eligible patients rather than only half of
them to the new treatment. But because
the treatment and control groups come
from different time periods, they are "dif-
ferently constituted groups." This raises
the spectre of bias and sometimes the ac-
tuality. The drop in cardiovascular deaths
and the decrease in perinatal mortality
over the last decade are both not really un-
derstood, and both exemplify temporal
shifts in control levels of the sort that viti-
ate historical controls. Time changes all
things, including the patients' characteris-
tics at a hospital, the effectiveness of adju-
vant treatments not under study, the skill
of surgeons with a new operation, and the
skill of physicians with a new drug. Thus,
it is hard to know when an HCT does reach
a valid conclusion. There are successes and
there are failures.
An example of what seems to be a suc-
cessful HCT is that of a changing policy by
an institution toward stab wounds. Origi-
nally, the policy had been to perform an
exploratory laparotomy on all patients pre-
senting with abdominal stab wounds. On
the basis of advances in handling wounds
and some data from refusals to give con-
sent, the institution decided to change to a
policy allowing surgical judgment to be ex-
ercised. This reduced considerably the
number of laparotomies performed (92 to
40 percent) and also the numbers of infec-
tions (Nance and Cohn, 1969~. The overall
complication rate dropped from 27 to 12
OCR for page 76
76
percent, and no complications occurred in
72 unexplored patients.
Byar et al. (1976) call attention to an
RCT comparing placebo and estrogen
therapy for prostate cancer in which the
survival of placebo controls admitted! in
the first 2.5 years was significantly shorter
(p = .01) than the survival of those admit-
ted in the second 2.5 years, although ad-
mission criteria, in a fixed setting, were un-
changed. They point out that the use of the
early placebo group (as historical controls)
would have falsely led to the conclusion
that estrogen therapy (in the second pe-
riod) was effective.
It is possible to consider the use of histor-
ical controls whenever the variation in suc-
cessive control levels is statistically taken
into account. However, it may be difficult
or impossible to estimate that variation;
that is a practical difficulty. Furthermore,
there is a theoretical principle that applies.
The work of Meter (1975), and later Po-
cock (1976), show that for a given standard
deviation in batch-to-batch random bias,
there is a minimum study size number (the
number of experimental subjects) beyond
which relying on historical controls, no
matter how numerous they are, is inferior
to dividing the sample into two equal
groups, half experimental and half control.
In summary, historical control trials are
inferior to RCTs because (1) differently
constituted groups are inherently likely to
produce bias; (2) if the historical controls
were comparable and if the random bias of
successive batches of controls had variabil-
ity that was exactly known, then reliance
on the historical control data would be
preferable to randomization only for stud-
ies below a certain threshhold size; and (3)
knowledge of variability of the random
bias is often not available.
One often sees the argument that the
need for randomization can be circum-
vented by making statistical adjustments
for differently constituted subgroups, cor-
recting for differences in the influential
ASSESSING MEDICAL TECHNOLOGY
variables that affect outcomes, and render-
ing the subgroups comparable. It is easy to
find statisticians who place little credence
in this trust of statistical adjustment, and
for cogent reasons. First, some of the most
influential variables may not even be rec-
ognized as important. Second, the ones
that are recognized as important may not
have been measured, or they may not have
been measured comparably. Third, just
how to make the adjustment can be very
unclear; mutually influential variables can
be interrelated in ways that both are im-
portant and poorly understood. Random-
ization avoids these difficulties by ensuring
that whatever the critical variables may be
and however they may conspire together to
affect the outcomes, they cannot systemat-
ically benefit one treatment over the other,
beyond those vagaries of chance for which
the significance test specifically makes al-
lowances. This approach avoids the effort
of trying to unravel the Gordian knot of
causation and cuts through it at one stroke,
by random assignment.
Before leaving the subject of random as-
signment, the idea of randomization
within strata should be addressed. If some
pretreatment variable, say stage of disease,
is known to be strongly related to outcome,
then it can be wise to design the study so
that (nearly) equal numbers of both treat-
ments occur at each level of that pretreat-
ment variable. This kind of design is quite
natural for multi-institutional studies,
when each institution is treated as a stra-
tum. Refining the randomization to be
done separately within strata does not give
added protection against bias, but it may
increase the efficiency of a study, i.e., in-
crease its effective sample size (usually only
moderately).
Limitations of RCTs
The method, powerful as it is, is hard to
apply under certain circumstances. If out-
comes mature after decades, then comple-
OCR for page 77
METHODS OF TECHNOLOGY ASSESSMENT
tion of the RCT requires long-term main-
tenance of protocol-controlled follow-up,
which is difficult and expensive.
If a sufficiently rare outcome is the end-
point of interest, then detection of treat-
ment differences may call for unworkably
large sample sizes. One example was con-
cern about the safety of the anesthetic,
halothane. Detection of differences in sur-
gical death rates (about 2 percent overall)
that might relate to anesthetic choice
would amount to trying to distinguish be-
tween death rates such as 1.9 percent and
2.1 percent a task calling at least for hun-
dreds of thousands of patients. The retro-
spective study that was done did arrive at
conclusions, but they were expressed with
diffidence made necessary by the possible
existence of unrecognized biases.
Sometimes it is objected that an RCT is
not applicable because treatments are too
variable to be controlled with the speci-
ficity that an RCT demands. This objec-
tion is sometimes false; for example, a
treatment may be defined to allow modifi-
cation as indications arise in the course of
therapy. In other cases, the objection is
simply specious, for it asserts the impossi-
bility of answering the question "What is
the treatment?" That impossibility would
block any kind of objective assessment of
it.
A rather more difficult limitation to deal
with grows out of the possibility that a new
procedure started in an RCT may, outside
that trial, evolve into a superior modified
version of the treatment. Then, continua-
tion of the RCT is at risk of being irrelevant
or unethical. There is a real problem here,
and it deserves more study; the question is
how the use of protocol and randomization
can help to speed sound evolution of new
therapies. One proposal has been to "ran-
domize the first patient." (See, for in-
stance, Chalmers, 1975, 1981.) Inherent in
the concept of randomizing the first pa-
tient is a fluid protocol that allows a
change in the details of a new treatment as
77
the investigators improve their perfor-
mance (the "learning curve") or as other
information appears. It has not found wide
agreement. The definitive treatment of
these issues is not yet at hand.
The sample size of an RCT may have
been planned to resolve differences of a
stated size, but when it is completed, ques-
tions about treatment comparisons in cer-
tain subclasses of patients cannot be re-
solved. This is not a limitation of the RCT
per se, for more questions always can be
asked of a body of data than can be an-
swered by it, but one should be warned to
think at the planning stage about choosing
sample sizes large enough to support ade-
quate treatment comparisons in particu-
larly salient subgroups.
It is sometimes argued that RCTs are too
costly. The cost of disciplined, careful,
checked medical work is of course high; the
advantages of the protocol are not cheaply
bought. But in many medical centers with
already high standards of recordkeeping,
diagnosis, etc., the incremental cost of the
protocol might not be great. The incre-
mental cost of randomization is negligible.
Costs can be high when the base costs of
bed, drugs, tests, and care are all loaded
onto the RCT budget. Most of these costs
would have been incurred anyway, re-
gardless of how the patients were treated.
Failure to distinguish between total
costs, which include those that would be
incurred anyway, from incremental costs
of RCTs is inherently misleading and could
lead to grievous policy errors. Good mea-
surements of incremental costs of RCTs are
needed. This will involve both conceptual
effort and data gathering. Better informa-
tion concerning actual incremental costs of
RCTs is a topic that should receive system-
atic research attention.
Two other limitations of RCTs also are
drawbacks to any investigational method.
The first is that dispute may grow around
unwelcome conclusions and hinder adop-
tion of the findings. The second is that the
OCR for page 78
78
RCT may give a clear verdict in patients of
the kind used in the trial, but leave unan-
swered the question of efficacy in different
kinds of subjects. This issue, dubbed exter-
nal validity sometimes is readily dealt
with; thus, the Salk vaccine trials showed
the vaccine to be effective in first-, second-,
and third-grade children. No difficulty
was found in generalizing the conclusion to
both older and younger children. Some-
times things are harder they may even
demand further RCTs. External validity is
of course a problem whenever we under-
tal~e to learn from one body of experience
and then apply the results to other experi-
ence; it is not a peculiar difficulty of RCTs.
We do not know as much as we could af-
ford to about designing studies with an eye
on external validity. This is another area
that deserves further research effort.
Strengthening RCTs
The primary paths to good quality lie in
designing a strong protocol and executing
it faithfully. The paper by Goodman in
Appendix 3-B of this committee's report
gives a systematic treatment of most of the
key features of a strong protocol. Extensive
accounts of RCT protocol are given in
works by Friedman et al. (1981) and Sha-
piro and Louis (1983~. Some additional
ideas on pre- and post-protocol execution
deserve comment here.
First, the study should be large enough;
if it is too small to have a good chance of
establishing the existence of a plausibly
sized actual improvement, then it needs to
be made larger or to be abandoned. Other-
wise, work, money, and time will be de-
voted to an effort that lacks a good chance
of producing a useful finding. Statistical
methods for assessing adequacy of planned
study size (power calculations) are well es-
tablished and should be used. (Sometimes,
however, the opportunity to do a study is
too good to be missed even if it is too small
ASSESSING MEDICAL TECHNOLOGY
to be definitive. This should be reported
with the study in hope that results of other
studies can be combined with these and to-
gether they may reach firm conclusions.)
Second, the participating investigators
should fully understand and be fully sup-
portive of the investigation. Persons with
initial convictions about relative merits of
the treatments may prove to be encum-
brances to successful execution of the pro-
tocol.
Third, in planning for the time and
number of cooperating centers that will be
needed to carry the study through, be real-
istically guarded about the flow of eligible
patients that can be anticipated. Seasoned
RCT veterans recommend safety factors of
two, five, even ten.
The foregoing suggestions all relate to
the planning phase. A final way of
strengthening the RCT applies to the com-
pletion phase.
Write about and report it well. In par-
ticular, the operational definitions of all
terms should be clear. Thus, the reader
should not be left with doubts about how
the subjects were defined and selected,
how they were assigned to treatments,
what treatments were applied, or how out-
comes were measured. In addition, the re-
port should specify whether study staff
were blind to treatment allocation at key
steps like enrollment in study, determina-
tion of eligibility, interpretation of diag-
nostic tests, measurement of outcome, etc.
These issues were prominent among those
that DerSimonian et al. (1982) checked in
reviewing reports of clinical trials in four
leading medical journals and that Emerson
et al. (1984) checked in reviewing reports
in six leading surgical journals. Both stud-
ies answered five questions: (1) What were
the eligibility criteria for admission to the
study? (2) Was admission to the study done
prior to allocation of treatment? (3) Was
allocation to treatment done at random?
(4) What was the method of randomiza-
OCR for page 79
METHODS OF TECHNOLOGY ASSESSMENT
tion? (5) Were outcomes assessed by per-
sons who were blind to treatment?
Good reporting will also explain the
quality control measures that were an-
plied, methods of follow-up used, and au-
dit checks employed.
Not only should the reader be told what
was done, and how, but also what hap-
pened. Summary statistics should have the
aim of revealing information to the reader.
The methods of statistical analysis
should be explained. The best way to do
this is topic by topic. The analysis was ac-
tually done in such a pattern; it should be
reported that way: for understanding, for
specificity, and, incidentally, for ease of
writing. Sometimes one finds a published
paper which lists statistical procedures in
the methods section. "We used chi-
squared, the l-test, the F-test, and Jonck-
heere's test." The use of this style of report-
ing for banquet recipes would list all the
ingredients in all the dishes together and
report the use of stove, mixer, oven, meat
grinder, egg beater, and double boiler.
In addition to showing the data, or gen-
erously detailed summaries of them, the
statistical analysis should state each of the
principal questions that motivated the
study and what light the data shed on those
questions. (Note that this is not the same
thing at all as reporting just those results
that are statistically significant.) To lend
understanding both to significant and non-
significant results, it is wise to use confi-
dence intervals whenever feasible and to
report the power of statistical tests that are
applied (Freiman et al., 1978~. Interesting
statistical results that arise out of studying
the data (rather than from studying the
principal questions that motivated the
study) are necessarily on a different, and
somewhat ambiguous, logical footing. It is
usually wise to regard such outcomes with
considerable reserve, more as hypotheses
turned up than as facts established. It is es-
pecially important to be candid about the
79
nature and amount of "data dredging"
that has accompanied the analysis.
A Final Remark
The protocol has been described as a
compact; its construction is typically a col-
legial exercise. This entails some advan-
tages. Of course, deliberation and consul-
tation give opportunities for better
planning. Sometimes a sequence of RCTs
leads to cumulative expertise and strategiz-
ing. But, some of the greatest advantages
may lie in the ethical domain.
The use in human beings of a new treat-
ment with only partially understood prop-
erties raises certain problems of ethical
portent. (This is true whether that new
treatment is tried in an RCT or in any
other way.) Among these questions are the
following: How strong is the evidence that
this new treatment may be at least as good
as the best available current therapy? How
shall we know when we should stop using
both treatments and prefer only one of
them? Who shall be able to receive this
new treatment, and who shall not? Each of
these questions is likely to be better an-
swered when decided by a group of profes-
sionals, acting explicitly and consul-
tatively, in a process open to review.
Wishful thinking blooms wherever Homo
sapiens is found, but group consultation
tends more often than not to restrain it.
Another advantage of the collegial
building of the protocol is that investiga-
tors who already believe they know which
treatment is superior have the opportunity
to drop out, leaving to the trial's execution
investigators able to proceed in good con-
science to participate themselves and to in-
vite their patients to participate.
OCR for page 80
80
EVALUATING DIAGNOSTIC
TECHNOLOGIES*
Accurate diagnosis is central to good
medical practice. Diagnostic technology
provides the physician with diagnostic in-
formation. However, all diagnostic tests
and procedures have associated costs and
risks. Thus, persons involved with medical
care must determine whether an individ-
ual test or procedure provides significant
new diagnostic information and whether
the information provided and its impact on
subsequent medical care offset the costs
and risks of the technology. For each diag-
nostic test, these and related questions re-
quire assessment of (l) the diagnostic infor-
mation provided and (2) the impact of the
resulting therapy on patient outcome.
Such assessments of diagnostic technology
rarely are performed. Most diagnostic
technology undergoes only narrow and
limited evaluation. The lack of more com-
prehensive assessment severely limits the
efficient and optimal use of diagnostic tests
and procedures.
Fineberg et al. (1977) has formulated a
hierarchy of evaluation of diagnostic tech-
nologies:
1. Technical capacity—Does the device
or procedure perform reliably and deliver
accurate information?
2. Diagnostic accuracy Does the test
contribute to making an accurate diagno-
. ~
slsr
3. Diagnostic impact Does the test
result influence the pattern of subsequent
diagnostic testing? Does it replace other di-
agnostic tests or procedures?
4. Therapeutic impact Does the test
result influence the selection and delivery
of therapy? Is more appropriate therapy
used after application of the diagnostic test
*This section was contributed by I. Sanford
Schwartz.
ASSESSING MEDICAL TECHNOLOGY
than would be used if the test was not
available?
5. Patient outcome—Does performance
of the test contribute to improved health of
the patient?
Clearly, if diagnostic technology fails ut-
terly at any step in this chain, then it can-
not be successful at any later stage. If it
succeeds at some stage, this implies success
in the prior stages (even if they have not
been explicitly tested) but does not tell
what success may be attached to later
stages. Thus, an accurate test may or may
not lead to more accurate diagnosis, which
in turn may or may not lead to better ther-
apy, and that in turn may or may not even-
tuate in better health of the patient. Be-
cause many tests may be involved, it can
require carefully designed studies to gauge
success or failure of any particular one at
stages 2 through 5.
Present Evaluation Methods
The first step in the hierarchy of evaluat-
ing diagnostic tests and procedures is deter-
mination of the technical performance of
the test. Several factors are involved in this
evaluation. The first deals with the ability
of the test actually to measure what it
claims to measure. Replicability and bias
of test results are important measures of
test performance. Replicability (i.e., preci-
sion) reflects the variance in a test result
that occurs when the test is repeated on the
same specimen. A highly precise test ex-
hibits little variance among repeated mea-
surements, an imprecise test exhibits great
variance. The greater this variation, the
less faith one may have in a single test's
results. However, a precise test is not nec-
essarily a good test. A test may exhibit a
high level of replicability yet be in error. A
good test must be reliable (i.e., unbiased);
that is, it must exhibit agreement between
the mean test result and the true value of
the biologic variable being measured in the
OCR for page 165
METHODS OF TECHNOLOGY ASSESSMENT
where I is the interval of time since the last
Pap smear (in this example 1 year), F(t) is the
cumulative distribution for the length of
time from the moment a lesion is first detect-
able by a Pap smear until it becomes an inva-
sive cancer, P(t) is the cumulative distribu-
tion for the length of time from the first
moment of invasion to the appearance of
signs and symptoms that would cause the pa-
tient to seek care in the absence of screening,
rots is the instantaneous incidence rate of in-
vasive cancers tr(O) is the rate in 40-year-old
average-risk women], and EN is the random
false-negative rate of the Pap smear.
Each of the elements in Equation 1 has
an intuitive interpretation. The variable of
integration, t, denotes the possible times
that the woman might develop an invasive
cancer of the cervix (t = 0 is now). By inte-
grating from negative infinity to positive
infinity, this formula considers all the pos-
~ _ loo
O
~ O'
> ~
Lll — 7=
~' J ~ ~
Z E
is o
G 50
IL
Z
a:
~ o 25
165
sible times that an invasive cancer might
occur. For any particular time that an in-
vasive cancer might occur (call this time
t ' ), the expression 1 - PI - ~ ' ~ gives the
probability that the woman is currently
asymptomatic and has not yet detected or
sought care for signs or symptoms of the
cancer. F(t' + 1) - F(t') gives the proba-
bility that the cancer was not potentially
detectable until after the last Pap smear
was done a year ago. The expression 1 -
F(t' + 1) gives the probability that the le-
sion was detectable before last year's Pap
smear. This last expression must be multi-
plied by FN, the chance that that Pap
smear was falsely negative and missed it.
The expression rat') expel - it r~x)dx] is the
probability that this woman will in fact de-
velop an invasive cancer at the time t'.
A formula for the second probability is
the same as Equation 1 except that Equa-
~ cnCC
m11115
`<
_~o
~ A>) 10
X c,)
_ J ~ 5
80 _
~6 5 4 3 2 1
60 _ \
Frequency
(years between tests)
40 _
20 _
1 1
50 100 150 200 250 300
FINANCIAL COSTS
FIGURE 3-6 Effect of Pap test frequency on financial cost and three measures of benefit for a
20-year-old average-risk woman. Main assumptions are as follows: (1) testing is begun at age
20; (2) a woman will have a checkup every 3 years for other malignant diseases from ages 20 to
40, and then annually thereafter; (3) the marginal cost of a Pap test is $10; (4) Pap test-detect-
able dysplasia and carcinoma in situ precede invasive cervical carcinoma by an average of 17
years (range, O to 34 years); (5) 2.5 percent of invasive cervical cancers develop very rapidly,
requiring less than 2 years to pass through dysplasia and CIS; (6) no cases of dysplasia or CIS
regress spontaneously; (7) no Pap tests are falsely read as positive or suspicious; and (8) 5-year
relative survival rates from time or detection (lead time adjusted) are dysplasia and CIS, 98
percent; local invasive, 78 percent; and regional invasive, 43 percent. If. a woman must also
pay a $25 office visit fee for the separate visits for the Pap test, the costs increase to about $700
for an annual Pap test and $1,700 for a biannual Pap test (Eddy, 1981~.
OCR for page 166
166
tion 1 must be multiplied by l - EN, the
probability that the Pap smear will not be
falsely negative. In similar fashion formu-
las can be written for the other important
probabilities. These formulas are more
complicated if one wants to consider the
use of more than one type of test, a series of
previous examinations done at various fre-
quencies, and other factors, but the con-
cepts are similar.
To estimate the value of a Pap smear
done at various frequencies one can apply
formulas to calculate the probabilities of
important clinical and economic outcomes
relating to cervical cancer for each year in
a woman's life, constantly updating the
parameters of the formulas to keep track of
the woman's changing age and screening
history. The calculations can be performed
for each screening strategy being evalu-
ated: for example, no screening at all,
screening every year, screening every 3
years, screening every year for three nega-
tive examinations and then every 3 years,
and so forth. Parameters for the equations,
such as age-specific incidence rates [ret)]
and parameters for the functions P(t) and
F(t), are estimated from the data collected
in clinical and epidemiological studies.
The results of an analysis using parame-
ter values estimated from such studies are
illustrated in Figure 3-6 (Eddy, 1981~. This
figure shows the estimated effect of screen-
ing a woman with a Pap smear at various
frequencies from age 20 to 75. The figure
indicates three measures of benefit: the de-
crease in the probability that the woman
will die of cervical cancer; the increase in
her life expectancy, given that the woman
is destined to get invasive cancer; and the
increase in life expectancy for the average-
risk woman who may (with about a 1 per-
cent probability) or may not get invasive
cervical cancer. The horizontal axis gives
the present value (at age 20) of a lifetime
series of screening examinations minus the
present value of expected savings in treat-
ment costs.
ASSESSING MEDICAL TECHNOLOGY
The calculations indicate that the 3-year
Pap smear is about 99 percent as effective
as an annual Pap smear. If the 40-year-old,
average-risk woman in the original exam-
ple postponed her Pap smear another 2
years, the increased annual risk she would
run of dying of cervical cancer would be on
the order of l per lOO,000, about the same
as the risk of death from one round-trip
transcontinental airplane flight.
REFERENCES
American College of Surgeons, Commission on
Cancer. 1974. Cancer Program Manual. Chicago:
American College of Surgeons.
American Public Health Association. 1981. Con-
trol of Communicable Diseases in Man, A. Beneson,
ed. Washington, D.C.
Arnstein, S. R. 1977. Technology assessment: Op-
portunities and obstacles. IEEE Trans. Syst. Man and
Cybern. SM-7:571-582.
Averill, R. F., and L. F. McMahon. 1977. A cost-
benefit analysis of continued stay certification. Med.
Care 15:158.
Bailar J. C. 1970a. Periodical incidence surveys: I.
Organization. Seminar on Cancer Registries in Latin
America, Pan American Health Organization-World
Health Organization, 41-100.
Bailar J. C. 1970b. Periodical incidence surveys:
II. Basis for the selection of survey areas. Seminar on
Cancer Registries in Latin America, Pan American
Health Organization-World Health Organization,
101-110.
Bailar, J. C., III, T. A. Louis, P. W. Lavori, and
M. Polansky. 1984. A classification for biomedical re-
search reports. N. Engl. J. Med. 311:1482-1487.
Banta, D. H., C. J. Behney, and J. S. Willems.
1981. Costs and their evaluation. In Toward Rational
Technology in Medicine. New York: Springer Pub-
lishing.
Barron, B. A., and R. M. Richart. 1981. Screening
protocols for cervical neoplastic disease. Gynecol. On-
col. 12:S156.
Baum, M. L., D. S. Anish, T. C. Chalmers, et al.
1981. A survey of clinical trials of antibiotic prophy-
laxis in colon surgery: Evidence against further use of
no-treatment controls. N. Engl. J. Med.305: 795-798.
Bearman, J. E., R. B. Lowenson, and W. H. Gul-
len. 1974. Muench~s Postulates, Laws and Corollaries,
Biometrics Note #4, National Eye Institute, DHEW,
Bethesda.
Bell, R. L., and E. O. Smith. 1982. Clinical trials
in post-marketing surveillance of drugs. Controlled
Clinical Trials 3:61-68.
OCR for page 167
METHODS OF TECHNOLOGY ASSESSMENT
Berk, A. A., and T. C. Chalmers. 1981. Cost and
efficacy of the substitution of ambulatory for inpa-
tient care. N. Engl. J. Med. 304:393-397.
Bernstein, L. M., E. R. Siegel, and C. M. Gold-
stein. 1980. The hepatitis knowledge base: A proto-
type information transfer system. Ann. Intern. Med.
93: 169-181.
Berwick, D. M., and A. L. Komaroff. 1982. Cost
effectiveness of lead screening. N. Engl. J. Med.
306: 1392-1398.
Blum, R. L. 1982. Discovery, confirmation, and
incorporation of causal relationships from a large
time-oriented clinical data base: The RX project.
Comp. Biomed. Res. 15:164-187.
Boruch, R. F. 1985. Enhancing the usefulness of
longitudinal data by coupling longitudinal surveys
and randomized experiments. Draft Report for DOL,
Center for Statistics and Probability, Northwestern
University.
Boruch, R. F., and J. S. Cecil. 1979. Assuring con-
fidentiality of social research data. Philadelphia: Uni-
versity of Pennsylvania Press.
Boston Collaborative Drug Surveillance Program.
1973. Oral contraceptives and venous thromboembol-
ic disease, surgically confirmed gallbladder disease
and breast tumors. Lancet 1:1399-1404.
Braakman, R. 1978. Data bank of head injuries in
three countries. Scott. Med. J. 23:107-108.
Bruce, R. A., et al. 1974. Seattle Heart Watch: Ini-
tial clinical circulation and electrocardiographic re-
sponses to maximal exercise. Am. J. Cardiol. 33:459-
469.
Bruce, R. A., et al. 1981. A computer terminal pro-
gram to evaluate cardiovascular functional limits and
estimate coronary event rates. West. J. Med. 135:342-
350.
Bunker,J, P., B. J. Barnes, and F. Mosteller. 1977.
Costs, Risks and Benefits of Surgery. New York: Ox-
ford University Press.
Burke, J. F., H. S. Jordan, C. B. Boyle, and E.
Vanner. 1981. An Impact Study of the 1978 Consen-
sus Conference on Supportive Therapy in Burn Care.
Massachusetts Health Research Institute.
Byar, D. P., R. M. Simon, W. T. Friedewald, et
al. 1976. Randomized clinical trials: Perspectives on
some recent ideas. N. Engl. J. Med. 295:74-80.
Campbell, D. T., and J. C. Stanley. 1963. Experi-
mental and Quasi-Experimental Designs for Re-
search. Chicago: Rand McNally College Publishing.
Centers for Disease Control. 1979. Abortion Sur-
veillance 1977. Atlanta, Gal: Department of Health,
Education, and Welfare.
Centers for Disease Control. 1982a. Annual Sum-
mary, 1981. Reported morbidity and mortality in the
United States. Morbid. Mortal. Weekly Rep. 30~54~.
Centers for Disease Control. 1982b. Annual Sum-
mary, 1981. Morbid. Mortal. Weekly Rep. 30:126-
127.
167
Centor, R. A., and J. S. Schwartz. In press. Calcu-
lation of the area under a ROC curve using microcom-
puters. Med Decision Making.
Chaitman, B. R., et al. 1981. Angiographic preva-
lence of high-risk coronary artery disease in patient
subsets (CASS). Circulation 64:360-367.
Chalmers, T. C. 1975. Randomizing the first pa-
tient. Med. Clin. North Am. 59:1035-1038.
Chalmers, T. C. 1981. The Clinical Trial. Milbank
Mem. Fund Q. 59:324-339.
Cochran, W. G. 1954. The combination of esti-
mates from different experiments. Biometrics 10: 101-
129.
Cohen, S. N., M. F. Armstrong, R. L. Briggs, et al.
1974. Computer-based monitoring and reporting of
drug interactions. Medinfo 1974:889-894.
Collen, J. F., R. Feldman, A. Siegelaub, and D.
Crawford. 1970. Dollar cost per positive test for auto-
mated multiphasic screening. N. Engl. J. Med.
283:459-463.
Collen, M. F. 1979a. A case study of mam-
mography. In Medical Technology and the Health
Care System: A Study of the Diffusion of Equipment-
Embodied Technology. Prepared by the Committee
on Technology and Health Care, National Academy
of Sciences, Washington, D.C.
Collen, M. F. 1979b. A study of multiphasic health
testing. In Medical Technology and the Health Care
System: A Study of the Diffusion of Equipment-
Embodied Technology. Prepared by the Committee
on Technology and Health Care, National Academy
of Sciences, Washinggon, D.C.
Collen, M. F. 1979c. A guideline matrix for tech-
nological system evaluation. J. Med. Systems 2:249-
254.
Collen, M. F. 1979d. Cost effectiveness of auto-
mated laboratory testing. In Clinician and Chemist,
D. S. Young, et al., ea., Proceedings of the First A.O.
Beckman Conference in Clinical Chemistry, Ameri-
can Association of Clinical Chemists, Washingon,
D.C.
Collen, M. F. 1983. Utilization of Diagnostic X-ray
Examinations. HHS Pub. FDA 83-8208. Washington,
D.C.: U.S. Government Printing Office.
Collen, M. F., L. G. Dales, G. D. Friedman, et al.
1973. Multiphasic health checkup evaluation study.
4. Preliminary cost benefit analysis for middle-aged
men. Prev. Med. 2:236-246.
Collen, M. F., S. R. Garfield, R. H. Richart, et al.
1977. Cost analyses of alternative health examination
modes. Arch. Int. Med. 137:73-79.
Commission on Cancer. 1974. Cancer Program
Manual. Chicago: American College of Surgeons.
Cook, T. D., and L. C. Leviton. 1980. Reviewing
the literature: A comparison of traditional methods
with meta-analysis. J. Pers. 48:449-472.
Cooper, B. S., and D. P. Rice. 1976. The economic
cost of illness revisited. Soc. Secur. Bull. 39:21-36.
OCR for page 168
168
Cowley, R. A., W. J. Sacco, W. Gill, et al. 1974. A
prognostic index for severe trauma. J. Trauma
14: 1029-1035.
Cox, E. B., and W. Stanley. 1979a. Schema driven
time-oriented record on minicomputer. Comp.
Biomed. Res. 12:503-516.
Cox, E. B., J. Laszlo, and A. Freiman. 1979b.
Classification of cancer patients: Beyond TNM. J.
Am. Med. Assoc. 242:2691-2695.
Cretin, S. 1977. Cost-benefit analysis of treatment
and prevention of myocardial infarction. Health Serv.
Res. 12:174.
Cutler, S. J., J. Scotto, S. S. Devesa, and R. R.
Connelly. 1974. Third National Cancer Survey—An
overview of available information. J. Natl. Cancer
Inst. 53:1565-1575.
Dalkey, N. C. 1969. The Delphi Method: An Ex-
perimental Study of Group Opinion. Santa Monica,
Calif.: Rand Corporation.
Dannenberg, A. L., R. Shapiro, and J. F. Fries.
1979. Enhancement of clinical predictive ability by
computer consultation. Meth. Inform. Med. 18:10-
14.
Deane, R., and A. Ulene. 1977. Hysterectomy or
tubal ligation for sterilization: A cost-effectiveness
analysis. Inquiry 14:73.
Delbecq, A., A. H. Van de yen, and D. H. Gustaf-
son. 1975. Group Techniques for Program Planning.
Glenview, Ill.: Scott, Foresman.
Demographic Analysis Section, Biometry Branch,
National Cancer Institute. 1976. Code Manual—The
SEER Program. DHEW Pub. No. (NIH)-79-1999.
Bethesda, Md.: National Cancer Institute.
DerSimonian, R., L. J. Charrette, B. McPeek, and
F. Mosteller. 1982. Reporting on methods in clinical
trials. N. Engl. J. Med. 306:1332-1337.
DerSimonian, R., and N. Laird. 1982. Evaluating
the effectiveness of coaching for SAT exams, a meta-
analysis. Pp. 1-15 in Harvard Educational Review.
Devine, E. C., and T. D. Cook. 1983. Effects of
psychoeducational intervention on lengh of hospital
stay: A meta-analytic review of 34 studies, American
Journal of Nursing. Reproduced in R. J. Light, ed.
1983. Evaluation Studies, Review Annual, Vol. 8, pp.
417-432. Beverly Hills, Calif.: Sage Publications.
Dixon, R. E. 1978. Effects of infections on hospital
care. Ann. Intern. Med. 89 (part 2~: 749-753.
Drazen, E., and J. Metzger. 1981. Methods for
evaluating costs of automated hospital information
systems. DHHS Publ. No. (PHS) 81-3283. Washing-
ton, D.C.: U.S. Government Printing Office.
Dyke, F. J., F. A. Murphy, J. K. Murphy, et al.
1974. Effect of surveillance on the number of hyster-
ectomies in the province of Saskatchewan. N. Engl. J.
Med. 296:1326-1328.
Eddy, D. M. 1980. Screening for Cancer: Theory,
Analysis and Design. Englewood Cliffs, N.J.: Pren-
tice-Hall.
ASSESSING MEDICAL TECHNOLOGY
Eddy, D. M. 1981. Appropriateness of cervical
cancer screening. Gynecol. Oncol. 12:S168.
Eddy, D. M. 1982. Clinical policies and the quality
of clinical practice. N. Engl. J. Med. 307:343-347.
Emerson, J. D., B. McPeek, and F. Mosteller.
1984. Reporting clinical trials in general surgical jour-
nals. Surgery 95:572-579.
Evans, R. W. 1983. Health care technology and
the inevitability of resource allocation and rationing
decisions. J. Am. Med. Assoc. 149:2208-2219.
Evenson, R. C., H. Altman, I. W. Sletten, and D.
W. Cho. 1975. Accuracy of actuarial and clinical pre-
dictions for length of stay and unauthorized absence.
Dis. New. Syst. 36:250-252.
Eysenck, H. J. 1978. An exercise in mega-silliness.
Am. Psychol. 33:517.
Farber, M. E., and S. N. Finkelstein. 1979. A cost-
benefit analysis of a mandatory premarital rubella-
antibody screening program. N. Engl. J. Med.
300:856-859.
Feigl, P., N. E. Breslow, J. Laszlo, et al. 1981. The
U.S. centralized cancer patient data system for uni-
form communication among cancer centers. J. Natl.
Cancer Inst. 67:1017-1024.
Feinstein, A. R. 1977. Clinical Biostatistics, pp.
214-226. St. Louis: C. V. Mosby.
Feinstein, A. R. 1978. Clinical biostatistics XLIV.
Clin. Pharmacol. Ther. 24:117-25.
Ferguson, M. 1980. The Aquarian Conspiracy. Los
Angeles: ]. P. Tarcher, Inc.
Fineberg, H. V. 1979. Gastric freezing: A study of
diffusion of a medical innovation. Pp. 173-200 in
Medical Technology and the Health Care System: A
Study of the Diffusion of Equipment-Embodied
Technology. Washington, D.C.: National Academy
Press.
Fineberg, H. V., R. Bauman, and M. Sosman.
1977. Computerized cranial tomography: Effect on
diagnostic and therapeutic plans. J. Am. Med. Assoc.
238:224-230.
Fink, A., J. Kosecoff, M. Chassin, and R. H.
Brook. 1984 Consensus methods: Characteristics and
guidelines for use. Am. ]. Public Health 74:979-983.
Finkler, S. A. 1982. The distinction between cost
and charges. Ann. Intern. Med. 96:102-1099.
Finney, D. J. 1965. The design and logic of a moni-
tor of drug use. J. Chronic Dis. 18:77-98.
Finney, D. J. 1966. Monitoring adverse reactions
to drugs—its logic and its weakness. Pp. 198-207 in
Medical Foundation, International Congress Series
No. 115. Proceedings of the European Society for the
Study of Drug Toxicity, Vol. VII.
Fisher, R. A. 1938. Statistical Methods for Re-
search Workers. London: Oliver & Boyd.
Fisher, R: A. 1948. Combining independent tests
of significance. Am. Stat. 2:30.
Fletcher, R. H., and S. W. Fletcher. 1979. Clinical
OCR for page 169
METHODS OF TECHNOLOGY ASSESSMENT
research in general medical journals: A 30 year per-
spective. N. Engl. J. Med. 301:108-183.
Foege, W. H., J. D. Millar, and J. M. Lane. 1971.
Selective epidemiologic control in smallpox eradica-
tion. Am. J. Epidemiol. 94:311-315.
Freiman, J. A., T. C. Chalmers, H. Smith, Jr.,
and R. R. Kuebler. 1978. The importance of the type
II error and the sample size in the design and interpre-
tation of the randomized control trial. N. Engl. J.
Med. 299: 690-694.
Friedman, G. D. 1972. Screening criteria for drug
monitoring: The Kaiser-Permanente drug reaction
monitoring system. J. Chronic Dis. 25:11-20.
Friedman, G. D. 1983a. Rauwolfia and breast
cancer: No relation found in long term users age fifty
and over. J. Chronic Dis. 36:367-370.
Friedman, G. D., and H. K. Ury. 1983b. Screen-
ing for possible drug carcinogenicity: Second report of
findings. J. Natl. Cancer Inst. 71: 1165- 1175.
Friedman, L. F., C. D. Furberg, and D. L. De-
Mets. 1981. Fundamentals of Clinical Trials, ix, 225.
Boston: John Wright, PSG.
Fries, J. F., S. Weyl, and H. R. Holman. 1974. Es-
timating prognosis in systemic lupus erythematosus.
Am. J. Med. 57:561-565.
Fuchs, V. R. 1974. Who Shall Live? Health, Eco-
nomics and Social Choice. New York: Basic Books.
Galbraith, S. L. 1978. Prognostic factors already
known. Scott Med. J. 23:108-109.
Galliher, H. P. 1981. Optimizing ages for cervical
smear examinations in followed healthy individuals.
Gynecol. Oncol. 12:S188.
Garber, A. M., V. R. Fuchs, and J. F. Silverman.
1984. Case mix, costs, and outcomes: differences be-
tween faculty and community services in a university
hospital. N. Engl. J. Med. 310:1231-1237.
Gershman, S. T., H. Barrett, J. T. Flannery, et al.
1976. Development of the Connecticut tumor regis-
try. Conn. Med. 40:697-701.
Gilbert, J. P., B. McPeek, and F. Mosteller. 1977.
Progress in surgery and anesthesia: Benefits and risks
of innovative therapy. In Costs, Risks, and Benefits of
Surgery, J. Bunker, B. Barnes, and F. Mosteller, eds.
New York: Oxford University Press.
Gittelsohn, A. M., and J. Wennberg. 1977. On the
incidence of tonsillectomy and other common surgical
procedures. Pp. 91-106 in Costs, Risks and Benefits of
Surgery, J. P. Bunker, B. A. Barnes, and F. Mosteller,
eds. New York: Oxford University Press.
Glaser, E. M. 1980. Using behavioral science strat-
egies for defining the state-of-the-art. J. Appl. Behav.
Sci. 16:79-82.
Glass, G. V. 1976. Primary, secondary and meta-
analysis of research. Educ. Res. 5:351-379.
Glass, G. V., B. McGaw, and M. L. Smith. 1981.
Meta-Analysis in Social Research. Beverly Hills,
Calif.: Sage Publications.
169
Globe, S., G. N. Levy, and C. M. Schwartz. 1967.
Science, Technology, and Innovation. Contract NSF-
C667 (Battelle-Columbus Laboratories). Washing-
ton, D. C.: National Science Foundation.
Grace, N. D., H. Muench, and T. C. Chalmers.
1966. The present status of shunts for portal hyperten-
sion in cirrhosis. Gastroenterology 50:684.
Graham, G. F., and F. J. Wyllie. 1979. Prediction
of gall-stone pancreatitis by computer. Br. Med. J.
1 :515-517.
Groot, L. M. J. 1982. Advanced and Expensive
Medical Technology in the Member States of the Eu-
ropean Community: Legislation, Policy and Costs.
Commissioned by the European Community. Mimeo-
graph; Roermond, The Netherlands.
Grossman, L. B. 1981. Beads and balloons. Pp. 2-6
in Pacemaker, Vol. 6, No. 3. Detroit: Harper Grace
Hospitals.
Gustafson, D. H., A. L. Delbecq, M. Hansen, and
R. F. Myers. 1975. Design of a health policy research
and development system. Inquiry 12:251-262.
Haley, R. W., D. Quade, H. E. Freeman, et al.
1980. The Senic Project: Study on the efficacy of non-
socomial infection control. Am. J. Epidemiol.
111 :472-485.
Hanley, J. A., and B. J. McNeil. 1982. The mean-
ing and use of the area under a receiver operating
characteristic (ROC) curve. Diag. Radiol. 143:29-36.
Health and Social Service Journal, May 30, 1980,
pp. 702-704.
Hedges, L. V., and I. Olkin. 1980. Vote-counting
methods in research synthesis. Psychol. Bull. 88:359-
369.
Herbert, T. T., and E. B. Yost. 1979. A compari-
son of decision quality under nominal and interacting
consensus group formats: The case of the structured
problem. Decision Sci. 10:358-370.
Herbst, A. L., M. Ulfelder, and D. C. Poskaner.
1971. Association of maternal stilbestrol therapy with
tumor appearance in young women. N. Engl. J. Med.
284:878-881.
Heyman, A., J. G. Burch, R. Rosati, et al. 1979.
Use of a computerized information system in the man-
agement of patients with transient cerebral ischemia.
Neurology 29:214-221.
Hiltz, S. R., and M. Turoff. 1978. The Network
Nation. Reading, Mass: Addison-Wesley.
Hoaglin, D. C., R. J. Light, B. McPeek, et al.
1982. Data for Decisions. Cambridge, Mass: Abt
Books.
Hodgkin, J. E., ed. 1979. Chronic Obstructive Pul-
monary Diseases: Current Concepts in Diagnosis and
Comprehensive Care. Park Ridge, Ill.: American Col-
lege of Chest Physicians.
Hodgkin, J. E., O. J. Balchum, I. Kass, et al. 1975.
Chronic obstructive airway diseases: Current con-
cepts in diagnosis and comprehensive care. J. Am.
Med. Assoc. 232:1243-1260.
OCR for page 170
170
Hook, E. B., D. M. Schreinemachers, and P. K.
Cross. 1981. Use of prenatal cytogenic diagnosis in
New York State. N. Engl. J. Med. 305:1410-1413.
Horn, S. D., and J. W. Williamson. 1977. Statisti-
cal methods for reliability and validity testing: An ap-
plication to nominal group judgments in health care.
Med. Care 15:922-928.
Hovrocks, J. C., D. E. Lambert, W. A. McAdams,
et al. 1976. Transfer of computer aided diagnosis of
dyspepsia from one geographical area to another. Gut
17:640-644.
Hunter, J. E. 1982. Meta-Analysis: Cumulating
Research Findings Across Studies. Beverly Hill,
Calif.: Sage Publications.
IMS America Ltd. 1978. Report of the Joint Com-
mission on Prescription Drug Use, Contract 223-78-
3007. Department of Health, Education, and Wel-
fare, Food and Drug Administration.
Inglefinger, J. A., F. Mosteller, L. A. Thibodeau,
and J. H. Ware. 1983. Biostatistics in Clinical Medi-
cine, Section 11-6, pp. xiii, 316. New York: Macmil-
lan.
Institute of Medicine 1981. Evaluating Medical
Technologies in Clinical Use. Washington, D.C.: Na-
tional Academy Press.
Jackson, G. B. 1980. Methods for integrative re-
views. Rev. Educ. Res. 50:438-460.
Jacoby, I. 1983. Biomedical technology: Informa-
tion dissemination and the NIH consensus develop-
ment process. Knowledge: Creation, Diffusion. Utili-
zation 5:245-261.
Jeans, W. D., and A. F. Morris. 1976. The accu-
racy of radiological and computer diagnoses in small
bowel examinations in children. Br. J. Radiol.
49:665-669, 1976.
Jennett, B., G. Teasdale, R. Brackman, et al.
1976. Predicting outcome in individual patients after
severe head trauma. Lancet 1: 1031-1034.
Jick, H., A. M. Walker, and C. Spriet-Pourra.
1979. Postmarketing follow-up. J. Am. Med. Assoc.
242:2310-2314.
Jillson, I. A. 1975. The national drug-abuse policy
delphi: Progress report and findings to date. In The
Delphi Method: Techniques and Applications, H.
Linstone and M. Turoff, eds. Reading, Mass.: Addi-
son-Wesley.
Kennedy, M. M. 1979. Generalizing from single
case studies. Eval. Q. 3:661-678.
Kimball, A. M., S. B. Thacker, and M. E. Levy.
1980. Shigella surveillance in a large metropolitan
area: Assessment of a passive reporting system. Am. J.
Public Health 70:164-166.
Klarman, H. E. 1973. Application of cost-benefit
analysis of health systems technology. In Technology
and Health Care Systems in the 1980's, M. F. Collen,
ed. DHEW Publ. (HMS) 73-3016. Washington,
D.C.: U.S. Government Printing Office.
ASSESSING MEDICAL TECHNOLOGY
Knill-Jones, R. 1978. New statistical approach to
prediction. Scott. Med. J. 23:102-110.
Kolata, G. 1985. Heart panel's conclusions ques-
tioned. Science 227:40-41.
Koplan, J. P., and L. S. Farer. 1980. Choice of
preventive treatment for Isoniazid-resistant tubercu-
lous infection. J. Am. Med. Assoc. 244:2736-2740.
Koplan, J. P., and S. R. Preblud. 1982. A benefit-
cost analysis of mumps vaccine. Am. J. Dis. Child.
136:362-364.
Koplan, J. P., S. C. Schoenbaum, M. C. Wein-
stein, et al. 1979. Pertussis vaccine: An analysis of
benefits, risks and costs. N. Engl. J. Med. 301:906-
911.
Kruskal, W., and F. Mosteller. 1980. Representa-
tive sampling IV: The history of the concept in statis-
tics, 1895-1939. Int. Stat. Rev. 48:169-195.
Kurland, L. T., and C. A. Molgaard. 1981. The
patient record in epidemiology. Sci. Am. 245:54-63.
Lane, J. M., F. L. Ruben, J. M. Neff, et al. 1969.
Complications of smallpox vaccination, 1968. Na-
tional surveillance in the United States. N. Engl. J.
Med. 281:1201-1208.
Laszlo, J. 1984. In press. Two long-range clinical
data bases are terminated—bang, whimper, or rip-
ple? J. Clin. Oncol.
Laszlo, J. Healthy registry and clinical data base
terminology, with special reference to cancer-regis-
tries. Submitted for publication.
Laszlo, J., C. Angle, and E. Cox. 1976. The hospi-
tal tumor registry: Present status and future prospects.
Cancer 38:395.
Lave, J., A. Dobson, and C. Walton. 1983. The
potential use of Health Care Financing Administra-
tion data tapes for health care services research.
Health Care Financing Rev. 5:93-98.
Layde, P. M., S. D. Von Allman, and G. P. Oak-
ley. 1979. Maternal serum alpha-fetoprotein screen-
ing: A cost-benefit analysis. Am. J. Publ. Health
69:566-573.
Lembcke, P. A. 1956. Medical auditing by scien-
tific methods, illustrated by major female pelvic sur-
gery. J. Am. Med. Assoc. 162:646-655.
Lenfant, C., B. Rifkind, and I. Jacoby. 1985.
Heart panel's conclusions (letter). Science 227:582-
583.
Light, R. J., ed. 1983. Evaluation Studies Review
Annual, Vol. 8. Beverly Hills: Sage Publications.
Light, R. J., and P. V. Smith. 1971. Accumulating
evidence: Procedures for resolving contradictions
among different research studies. Harv. Ed. Rev.
41:429-471.
Linstone, H., and M. Turoff, eds. 1975. The
Delphi Method: Techniques and Applications. Read-
ing, Mass.: Addison-Wesley.
Lips, K. J. M., V. D. S. Veer, A. Struyvenberg,
and R. A. Geerdink. 1982. Genetic predisposition to
cancer in man. Am. J. Med. 73:305-307.
OCR for page 171
METHODS OF TECHNOLOGY ASSESSMENT
Louis, P. 1836. Researches on the Effects of Blood
Letting on Some Inflammatory Disease and on the In-
fluence of Tartarized Antimony and Vesication in
Pneumonia (translated by C. G. Putman). Boston:
Hilliard Gray and Company.
Louis, T. A., H. Fineberg, and F. Mosteller. 1985.
Findings for public health from meta-analysis, pp. 1-
20. Annual Review of Public Health. Palo Alto, Cali-
fornia: Annual Reviews, Inc.
Lusted, L. B. 1969. Perception of roentgen image:
Applications of signal detection theory. Bad. Clinics
N. Am. 7:435-445.
Maclennan, R., C. Muir, and A. Winkler. 1978.
Cancer Registration and Its Techniques. Lyon: Inter-
national Association for Cancer Research.
Mantel, N., and W. Haenszel. 1959. Statistical as-
pects of the analysis of data from retrospective studies
of disease. J. Natl. Cancer Inst. 22:719-748.
McNeil, B. J. 1979. Pitfalls in and Requirements
for Evaluations of Diagnostic Technologies in Medical
Technology. DHEW Publ. No. (PHS) 79-3254. Urban
Institute Conference.
McNeil, B. J., P. D. Varady, B. A. Burrows, and S.
J. Adelstein. 1975. Cost-effectiveness calculations in
the diagnosis and treatment of hypertensive renovas-
cular disease. N. Engl. J. Med. 293:216-221.
McShane, D. J., J. Porta, and J. F. Fries. 1978.
Comparison of therapy in severe systemic lupus
erythematosus employing stratification techniques. J.
Rheumatol. 5:51-58.
Meier, P. 1975. Statistics and medical experimen-
tation. Biometrics 31:511-530.
Metz, C. E. 1978. Basic principles of ROC analy-
sis. Semin. Nucl. Med. 7:283-298.
Metz, C. E., D. J. Goodenaugh, and K. Rossman.
1973. Evaluation of receiver operating characteristic
curve data in terms of information theory, with appli-
cations in radiology. Radiology 109:297-303.
Moscovice, I., P. Armstrong, S. Shortell, and R.
Bennet. 1977. Health services research for decision-
makers: The use of the Delphi technique to determine
health priorities. J. Health Politics, Policy and Law
2:388-410.
Mosteller, F. M., and R. R. Bush. 1954. Selected
quantitative techniques. In Handbook of Social Psy-
chology, Vol. I, Theory and Method, G. Lindzey, ed.
Cambridge, Mass.: Addison-Wesley.
Mulley, A. G., M. D. Silverstein, and J. L. Dien-
stag. 1982. Indications for use of hepatitis B vaccine,
based on cost-effectiveness analysis. N. Engl. J. Med.
307:644-652.
Mullner, R. M., C. S. Byre, and C. L. Kil-
lingsworth. 1983. An inventory of the U.S. health
care data bases. Review of Public Data Use 11:79-
192.
Naisbitt, J. 1982. Megatrends. New York: Warner
Books.
171
Nance, F. C., and I. Cohn, Jr. 1969. Surgical judg-
ment in the management of stab wounds of the abdo-
men. A retrospective and prospective analysis based
on a study of 600 stabbed patients. Annals of Surgery
170:569.
National Academy of Sciences. 1978. Personnel
Needs and Training for Biomedical and Behavioral
Research Personnel. Washington, D. C.: National
Academy Press.
National Academy of Sciences. 1981. Personnel
Needs and Training for Biomedical and Behavioral
Research. Washington, D. C.: National Academy
Press.
National Academy of Sciences/Institute of Medi-
cine. 1983. Personnel Needs and Training for Bio-
medical and Behavioral Research. Washington,
D.C.: National Academy Press.
National Academy of Sciences. 1985. Sharing Re-
search Data, S. E. Fineberg, M. E. Martin, and M. L.
Straf, eds. Washington, D.C.: National Academy
Press.
National Center for Health Care Technology.
1981. Coronary artery bypass surgery. J. Am. Med.
Assoc. 2246:1643.
National Center for Health Statistics. 1980a. Cata-
log of Public Use Data Tapes from the National Cen-
ter for Health Statistics. Public Health Service, DHHS
Publ. No. (PHS) 81-1213. Washington, D.C.: U.S.
Government Printing Office.
National Center for Health Statistics. 1980b. Data
Systems of the National Center for Health Statistics.
Pp. 28-30 in Public Health Service, DHHS Publ. No.
(PHS) 80-1247. Washington, D.C.: U.S. Government
Printing Office.
National Center for Health Statistics. 1980c. Long-
Term Health Care: Minimum Data Set. Report of the
National Committee on Vital and Health Statistics.
Public Health Service, DHHS Publ. No. (PHS) 80-
1158. Washington, D.C.: U.S. Government Printing
Office.
National Center for Health Statistics. 1980d. The
collection and processing of drug information, Na-
tional Ambulatory Medieal Care Survey, United
States. Prepared by H. Koeh. In Vital and Health Sta-
tisties, Series 2, No. 90. Publie Health Serviee. Wash-
ington, D.C.: U.S. Government Printing Offiee.
National Center for Health Statisties. 1980e.
Trends and variations in eesarean section delivery.
Prepared by P. J. Placek, S. M. Taffel, and J. C.
Kleinman. Pp. 73-36 in Health, United States, 1980.
Public Health Service, DHHS Publ. No. (PHS) 81-
1232. Washington: U.S. Government Printing Office.
National Center for Health Statistics. 1980f. Uni-
form Hospital Discharge Data: Minimum Data Set.
Report of the National Committee on Vital and
Health Statisties. Publie Health Serviee, DHEW
OCR for page 172
172
Publ. No. (PHS) 80-1157. Hyattsville, Md.: U.S. Gov-
ernment Printing Office.
National Center for Health Statistics. 1981a. Inpa-
tient utilization of short-stay hospitals by diagnosis,
United States, 1978. Prepared by E. McCarthy. P. 24
in Vital and Health Statistics, Series 13, No. 55. Pub-
lic Health Service, DHHS Publ. No. (PHS) 81-1716.
Washington, D.C.: U.S. Government Printing Of-
fice.
National Center for Health Statistics. 1981b.
NMCES Household Interview Instruments: Instru-
ments and Procedures 1. Prepared by G. S. Bonham
and L. T. Corder. Public Health Service, DHHS
Publ. No. (PHS) 81-3280. Washington, D.C.: U.S.
Government Printing Office.
National Center for Health Statistics. 1981c. Uni-
form Ambulatory Medical Care: Minimum Data Set.
Report of the National Committee on Vital and
Health Statistics. Public Health Service, DHHS Publ.
No. (PHS) 81-1161. Hyattsville, Md.: U.S. Govern-
ment Printing Office.
National Center for Health Statistics. 1983a. Data
Systems of the National Center for Health Statistics,
pp. 63-75. Public Health Service, DHHS Publ. No.
(PHS) 80-1247. Washington, D.C.: U.S. Government
Printing Office.
National Center for Health Statistics. 1983b. Drug
utilization in office-based practices: Summary of find-
ings. National Ambulatory Medical Care Survey.
United States, 1980. Prepared by H. Koch. In Vital
and Health Statistics, Series 13, No. 65. Public Health
Service, DHHS Publ. No. (PHS) 83-1726. Washing-
ton, D.C.: U.S. Government Printing Office.
National Center for Health Statistics. 1983c.
Health, United States, 1983, p. 13. Public Health Ser-
vice, DHHS Publ. No. (PHS) 84-1232. Washington,
D.C.: U.S. Government Printing Office.
ASSESSING MEDICAL TECHNOLOGY
National Commission for the Protection of Human
Subjects. 1978. Report and Recommendations. Wash-
ington, D.C.: U.S. Government Printing Office.
National Institutes of Health. 1983. Liver trans-
plantation consensus development. Conference Sum-
mary. Volume 4, Number 7.
National Institutes of Health. 1984. Diagnostic ul-
trasound imaging in pregnancy. Consensus Develop-
ment Conference. Volume 5, Number 1.
NHLBI Angioplasty, Kent et al. 1982. Percuta-
neous transluminal coronary angioplasty: Report
from the registry of the National Heart, Lung, and
Blood Institute. Am. J. Cardiol. 49:2011-2018.
Needleman, H. L, S. K. Geiger, and R. Frank.
1985. Lead and IQ scores: a reanalysis. Science
227:701-702.
Needleman, H. L., C. Gunnoe, A. Leviton, et al.
1979. Deficits in psychologic and classroom perfor-
mance of children with elevated dentine lead levels.
N. Engl. J. Med. 300:689-695.
Neustadt, R., and H. V. Fineberg. 1983. The Epi-
demic That Never Was: Decision Making in the Swine
Flu Scare. 1983. New York: Vintage Books.
Neutra, R. R., S. E. Fienberg, S. Greenland, and
E. A. Friedman. 1978. Effect of fetal monitoring on
neonatal death rates. N. Engl. J. Med. 299:324-326.
Office of Health Research Statistics and Technol-
ogy. 1981. U.S. Department of Health and Human
Services. Transsexual surgery. Assessment Report Se-
ries 1(4). Washington, D.C.: U.S. Department of
Health and Human Services.
Office of Technology Assessment, U.S. Congress.
1978a. Assessing the efficacy and safety of medical
technologies. Washington, D.C.: U.S. Government
Printing Office.
Office of Technology Assessment, U.S. Congress.
1978b. Policy implications of the computed tomo-
graphy (CT) scanner. GPO Stock No. 052-003-00565-
4. Washington, D.C.: U.S. Government Printing Of-
fice.
Office of Technology Assessment, U.S. Congress.
1980a. The implications of cost-effectiveness analysis
of medical technology. Stock No. 051-003-00765-7.
Washington, D.C.: U.S. Government Printing Of-
National (~enter for Health Statistics. lusted. Pro-
cedures and questionnaires of the National Medical
Care Utilization and Expenditure Survey. Prepared
by G. S. Bonham. In Series A, Methodological Report
No. 1. Public Health Service, DHHS Publ. No. 83-
20001. Washington, D.C.: U.S. Government Printing
Office.
Notional Center for Health Statistics. 1983e. Utili- fice.
zation of shortstay hospitals: United States, 1981 an- Office of Technology Assessment, U.S. Congress.
nual summary. In Vital and Health Statistics, Series 1980b. The implications of cost-effectiveness analysis
13, No. 72. Public Health Service, DHHS Publ. No. of medical technology/background paper #1: Method-
(PHS) 83-1733. Washington D.C.: U.S. Government ological issues and literature review. Washington,
Printing Office. D.C.: U.S. Government Printing Office.
National Center for Health Statistics. 1983f. Varia- Office of Technology Assessment, U.S. Congress.
tion in use of obstetric technology. Prepared by J. C. 1980c. The implications of cost-effectiveness analysis
Kleinman, M. Cooke, S. Machlin, and S. S. Kessel. of medical technology/background paper #3: The effi-
Pp. 63-75 in Health, United States, 1983. Public cacy and cost effectiveness of psychotherapy. Wash-
Health Service, DHHS Publ. No. (PHS) 84-1232. ington, D.C.: U.S. Government Printing Office.
Washington, D.C.: U.S. Government Printing Of- Office of Technology Assessment, U.S. Congress.
rice. 1981a. Cost effectiveness of influenza vaccination.
OCR for page 173
METHODS OF TECHNOLOGY ASSESSMENT
Washington, D.C.: U.S. Government Printing Of-
fice.
Office of Technology Assessment, U.S. Congress.
1981b. The implications of cost-effectiveness analysis
of medical technology/background paper #2: Case
studies of medical technologies/care study #2: The
feasibility of economic evaluation of diagnostic proce-
dures: The case of CT scanning. Washington, D.C.:
U.S. Government Printing Office.
Office of Technology Assessment, U.S. Congress.
1981c. The implications of cost-effectiveness analysis
of medical technology/background paper #2. Case
studies of medical technologies/case study #1: Formal
analysis, policy formulation, and end-stage renal dis-
ease. Washington, D.C.: U.S. Government Printing
Office.
Office of Technology Assessment, U.S. Congress.
1981d. The implications of cost-effectiveness analysis
of medical technology/background paper #2: Case
studies of medical technologies/case study #3: Screen-
ing for colon cancer: A technology assessment. Wash-
ington, D.C.: U.S. Governmment Printing Office.
Office of Technology Assessment, U.S. Congress.
1982a. Postmarketing Surveillance of Prescription
Drugs. Washington, D.C.: U.S. Government Print-
ing Office.
Office of Technology Assessment, U.S. Congress.
1982b. Strategies for Medical Technology Assessment.
Washington, D.C.: U.S. Government Printing Of-
fice.
Office of Technology Assessment, U.S. Congress.
1982c. Technology and Handicapped People, p. 22.
OTA-H-179. Washington, D.C.: U.S. Government
Printing Office.
Office of Technology Assessment, U.S. Congress.
1983a. Abstracts of Case Studies in the Health Tech-
nology Case Study Series. OTA-P-225. Washington,
D.C.: U.S. Government Printing Office.
Office of Technology Assessment, U. S. Congress.
1983b. Variations in Hospital Length of Stay: Their
Relationship to Health Outcomes. Health Technology
Case Study 24. OTA-HCS-24. Washington, D.C.:
U.S. Government Printing Office.
Olsen, S. A. 1982. Group Planning and Problem
Solving Methods in Engineering Management. New
York: John Wiley & Sons.
Patrick, K. M., and R. Woolley. 1981. A cost-ben-
efit analysis of immunization for pneumococcal pneu-
monia. J. Am. Med. Assoc. 245:473-477.
Pauker, S., and J. Kassirer. 1975. Therapeutic de-
cision making: A cost-benefit analysis. N. Engl. J.
Med. 293:229-234.
Pearson, E. S. 1938. The probability integral trans-
formation for testing goodness of fit and combining
independent tests of significance. Biometrika 30: 134-
148.
Pearson, E. S. 1950. On questions raised by the
173
combination of tests based on discontinuous distribu-
tions. Biometrika 37:383-398.
Perry, S., and J. T. Kalberer. 1980. The NIH Con-
sensus-Development Program and the assessment of
health-care technologies: The first two years. N.
Engl. J. Med. 303:169-172.
Pillemer, D. B., and R. J. Light. 1980. Synthekiz-
ing outcomes: How to use research evidence from
many studies. Harvard Educ. Rev. 50:176-195.
Pocock, S. J. 1976. The combination of random-
ized and historical controls in clinical trials. J.
Chronic Dis. 29:175-188.
Policy Research Incorporated. 1975a. National
Drug Abuse Policy Delphi Study: Questionnaires and
Summaries of Results. Baltimore.
Policy Research Incorporated. 1975b. National
Drug Abuse Policy Report: Final Report. Baltimore.
Policy Research Incorporated. 1977. A Compre-
hensive Study of the Ethical, Legal, and Social Impli-
cations of Advances in Biomedical and Behavioral Re-
search and Technology: Summary of the Final
Report. Baltimore.
Policy Research Incorporated. 1979a. Medical
Practice Information Demonstration Project: Depres-
sion Project: First Series of Instruments. Baltimore.
Policy Research Incorporated. 1979b. Medical
Practice Information Demonstration Project: Bipolar
Disorder: A State-of-the-Science Report. Baltimore.
Pollack, E. S. 1982a. Monitoring cancer incidence
and patient survival in the United States. Proceedings,
Social Statistics Section. Washington, D.C.: Ameri-
can Statistical Association.
Pollack, E. S. 1982b. SEER cost study. Printed for
private distribution, Natonal Cancer Institute.
President's Commission for the Study of Ethical
Problems in Medicine and Biomedical and Behavioral
Research. 1983. Summing up: Final Report on Studies
of the Ethical and Legal Problems in Medicine and
Biomedical and Behavioral Research. Washington,
D.C.: U.S. Government Printing Office.
Principal investigators of CASS and their associ-
ates. 1981. National Heart, Lung, and Blood Institute
Coronary Artery Surgery Study. Circulation
63(suppl. I):1-81.
Queram, C. J. 1977. Cancer Registries and Report-
ing Systems in the United States. Madisdn, Wis.: De-
partment of Health and Social Services, Bureau of
Health Statistics, Division of Health.
Rand Corporation. 1983. Submission to the Office
of Management and Budget of Supporting Statement
and Data Collection Instruments for Assessing the Ef-
fectiveness of the NIH Consensus Development Pro-
gram.
Ransahoff, D. F., and A. R. Feinstein. 1978. Prob-
lems of spectrum and bias in evaluating the efficacy of
diagnostic tests. N. Engl. J. Med. 299:926-930.
Re, R., R. Novelline7 M. T. Escourrou, et al. 1978.
OCR for page 174
174
Inhibition of angiotensin-converting enzyme for diag-
nosis of renal artery stenosis. N. Engl. J. Med.
298:582-586.
Remington, R. D. 1976. Recommendations. Pp.
141-150 in Assessing Drug Reactions—Adverse and
Beneficial, Vol. 7, Philosophy and Technology of
Drug Assessment, F. N. Allen, ed. Washington, D.C.:
The Interdisciplinary Communications Associates.
Rennie, D. 1981. Consensus Statements. N. Engl.
J. Med. 304:665-666.
Richart, R. H. 1974. Evaluation of a hospital com-
puter system, in Hospital Computer Systems, M. Col-
len, ed. New York: John Wiley & Sons, Inc.
Richart, R. M. 1981. Discussion of session III:
Screening of cervical neoplasia. Gynecol. Oncol.
12:S212.
Robertson, A., and M. H. Zweig. 1981. Use of re-
ceiver operating characteristic curves to evaluate the
clinical performance of analytical systems. Clin.
Chem. 77:1568-1574.
Romm, F. J., and B. S. Hulka. 1979. Developing
criteria for quality of care assessment: Effect of the
Delphi technique. Health Services Res. 14:309-312.
Boos, L. L., Jr. 1979. Alternative designs to study
outcomes. Med. Care 17:1069-1087.
Rosati, R. A., A. G. Wallace, and E. A. Stead.
1973. The way of the future. Arch. Intern. Med.
131:285.
Rosenthal, R. 1978a. Combining results of inde-
pendent studies. Psychol. Bull. 85:185-193.
Rosenthal, R., and D. B. Rubin. 1978b. Interper-
sonal expectancy effects: The first 345 studies. Behav-
ioral and Brain Sciences 3:377-415.
Russell, L. B. 1979. Technology in Hospitals: Med-
ical Advances in Their Diffusion. Washington, D.C.:
The Brookings Institution.
Sackman, H. 1975. Delphi Critique. Lexington,
Mass.: Lexington Books.
Sanders, C. A. 1979. Medical technology and the
health care system: A study of the diffusion of equip-
ment-embodied technology, by the Committee on
Technology and Health Care. Washington, D.C.:
National Academy of Sciences.
Schaffarzick, R., cited by Wennberg and Git-
telsohn, 1973.
Schneiderman, M. A. 1966. Looking backward: Is
it worth the crick in the neck? Or: pitfalls in using ret-
rospective data. Am. J. Roentgenol. Radium Ther.
Nucl. Med. 96:230-235.
Schoenbaum, S., J. N. Hyde, L. Bartoshesky, and
K. Crampton. 1976a. Benefit-cost analysis of rubella
vaccination policy. N. Engl. J. Med. 294:306-310.
Schoenbaum, S. C., B. J. McNeil, and J. Kavet.
1976b. The swine-influenza decision. N. Engl. J.
Med. 295:759-765.
Schwartz, J. S., P. J. Weinbaum, C. Nesler, et al.
1983. Assessment of tests to identify infants at high
ASSESSING MEDICAL TECHNOLOGY
risk of respiratory distress syndrome (RDS) using re-
ceiver operating characteristic (ROC) curve analysis.
Med. Decision Making 3:365.
Scitovsky, A. A. 1979. Changes in the use of ancil-
lary services for ~common' illness. Pp. 39-56 in Medi-
cal Technology: The Culprit Behind Health Costs?
Proceedings of the 1977 Sun Valley Forum on Na-
tional Health Insurance, Stuart H. Altman and
Robert Blendon, eds. DHEW Publ. No. (PHS) 79-
3216. New York: John Wiley.
Shapiro, M., et al. 1983a. Benefit-cost analysis of
antimicrobial prophylaxis in abdominal and vaginal
hysterectomy. J. Am. Med. Assoc. 249:1290-1294.
Shapiro, S. H., and T. A. Louis. 1983. Clinical
Trials: Issues and Approaches, pp. ix, 209. New York:
Marcel Dekker.
Sherins, R. J., C. L. Olivery, and J. L. Ziegler.
1978. Gynecomastia and gonadal dysfunction in ado-
lescent boys treated with combination chemotherapy
for Hodgkin's disease. N. Engl. J. Med. 299:12-16.
Showstack, J., et al. 1981. Evaluating the costs and
benefits of a diagnostic technology: The case of upper
gastrointestinal endoscopy. Med. Care 19:498-509.
Shur, R. D. 1981. Estimating the natural history of
cancer from cross-sectional data. Ph.D. Dissertation,
Engineering-Economic Systems, Stanford University.
Shwartz, M. 1978. An analysis of the benefits of se-
rial screening for breast cancer based upon a mathe-
matical model of the disease. Cancer 51:1550.
Siegel, E. R., 1980. Use of computer conferencing
to validate and update NLM's hepatitis knowledge
base, in The Future of Electronic Communications,
M. M. Henderson, and M. J. MacNaughton, eds.
Washington, D.C.: American Association for the Ad-
vancement of Science.
Snow, J. 1847. On the Inhalation of the Vapor of
Ether in Surgical Operations. (Reproduced by Lea &
Febiger, Philadelphia, 1959.)
Stange, P., and A. T. Summer. 1978. Predicting
treatment costs and life expectancy for end-stage renal
disease. N. Engl. J. Med. 298:372-378.
Stason, W,. and M. Weinstein. 1977. Allocation of
resources to manage hypertension. N. Engl. J. Med.
296:732.
Steinberg, D. 1985. Heart panel's conclusions (let-
ter). Science 227:582.
Swain, R. W. 1981. Health Systems Analysis. Co-
lumbus, Ohio: Grid Publishing.
Swets, J. A. 1979. ROC analysis applied to the
evaluation of medical imaging techniques. Invest. Ra-
diology 14:109-121.
Swets, J. D., and R. M. Pickett. 1982. Evaluation
of Diagnostic Systems. New York: Academic Press.
Tatro, D. S., T. N. Moore, and S. N. Cohen. 1979.
Computer-based system for adverse drug detection
and prevention. Am. J. Hosp. Pharm. 36:198-201.
OCR for page 175
METHODS OF TECHNOLOGY ASSESSMENT
Teasdale, G. 1978. Prediction in action. Scott.
Med. J. 23:111.
The Lancet Commission on Anesthesia. 1893. Lan-
cet 1:629-638, 693-708, 761-776, 899-914, 971-978,
1236-1240, 1479-1498.
Thornell, C. A. 1981. Comparison of Strategies for
the Development of Process Measures in Emergency
Medical Services, NCHSR Research Summary Series.
Hyattsville, Md.: National Center for Health Services
Research.
Turoff, M., and S. R. Hiltz. 1982. Computer sup-
port for group versus individual decisions. IEEE
Trans. Comm. COM-30(1~:82-91.
U.S. Consumer Product Safety Commission. 1982.
National Electronic Injury Surveillance System.
NEISS Data Highlights 6:1-4.
U.S. Public Health Service. 1964. Smoking and
Health. Report of the Advisory Committee to the Sur-
geon General of the Public Health Service. Washing-
ton, D.C.: U.S. Department of Health, Education,
and Welfare.
Warner, K. E., and R. C. Hutton. 1981. Cost-
benefit and cost-effectiveness analysis in health care.
Med. Care 19:498-509.
Warner, K. E., and B. R. Luce. 1982. Cost-Benefit
and Cost-Effectiveness Analysis in Health Care. Ann
Arbor, Mich.: Health Administration Press.
Weinstein M., and W. Stason. 1976. Hyperten-
sion: A Policy Perspective. Cambridge, Mass.: Har-
vard University Press.
Weinstein, M. C., and H. V. Fineberg. 1980. Clin-
ical Decision Analysis. Philadelphia: W. B. Saunders.
Weinstein, M. C., and B. Stason. 1977. Founda-
tions of cost-effectiveness analysis for health and med-
ical practices. N. Engl. J. Med. 296:716-721.
Wennberg, J. 1984. Dealing with medical practice
variations: A proposal for action. Health Affairs 3:6-
32.
Wennberg, J., and A. Gittlesohn. 1973. Small area
variations in health care delivery. Science 182:1102-
1108.
Wennberg, J. E. 1981. A strategy for the post-
marketing surveillance and evaluation of health care
technology. Department of Community and Family
Medicine, Dartmouth Medical School.
Wennberg, J. E., J. P. Bunker, and B. Barnes.
1980. The need for assessing the outcome of common
medical practices. Ann. Rev. Public Health 1:277-
295.
175
White, C., and J. C. Bailar. 1956. Retrospective
and prospective methods of striding association in
medicine. Am. J. Public Health 46:35-44.
White, J. J., and N. W. Axnick. 1975. The benefits
from 10 years of measles immunization in the United
States. Public Health Rep. 90:205-207.
Willems, J. S., R. Sanders, A. Riddiough, and C.
Bell. 1980. Cost-effectiveness of vaccination against
pneumococcal pneumonia. N. Engl. J. Med. 303:553-
559.
Williamson, J. W. 1978. Formulating priorities for
quality assurance activity. J. Am. Med. Assoc.
239:631-637.
Wilson, D. H. 1979a. The acute abdomen in the
accident and emergency department. Practitioner
222:480-485.
Wilson, S. 1979b. Explorations of the usefulness of
case study evaluations. Evaluation Q. 3:446-59.
World Health Organization. 1976a. WHO Hand-
book for Standardized Cancer Registrations. Geneva.
World Health Organization. 1976b. ICD-O, Inter-
national Classification of Diseases for Oncology, 1st
ed. Geneva.
Wortman, P. M. 1983. Evaluation research: A
methodological perspective. Ann. Rev. Psychol.
34:223-260.
Wortman, P. M., and L. Saxe. 1982a. Assessment
of medical technology: Methodological considerations
(Appendix C). In Strategies for Medical Technology
Assessment, prepared by the Office of Technology As-
sessment, U.S. Congress. Washington, D.C.: U.S.
Government Printing Office.
Wortman, P. M., and A. Vinokur. 1982b. Evalua-
tion of NIH Consensus Development Process. Phase I:
Final Report. Ann Arbor, Mich.: Center for Research
on Utilization of Scientific Knowledge, University of
Michigan.
Yin, R. K. 1984. Cast Study Research, Deisgn and
Methods. Beverly Hills: Sage Publications.
Young, J. L., Jr., A. Asire, and E. Pollock. 1976.
SEER Program: Cancer Incidence and Mortality in
the United Stats 1973-1976. DHEW Pub. No. (NIH)
78-1837. Bethesda, Md: National Cancer Institute.
Young, J. L., C. L. Percy, A. J. Asire, eds. 1981.
SEER Program: Incidence and Mortality Data, 1973-
77. National Cancer Institute Monograph 57, USG
80, NIH Pub. No. 81-2330. Washington, D.C.: Na-
tional Cancer Institute.
Representative terms from entire chapter:
diagnostic tests
