Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 34
Measurement of Outcomes
Before an evaluation can be designed, the goals of He intervention
program must be set. Indeed, goal sewing should precede the design and
implementation of a program. This chapter provides an overview of goal
sewing, the strengths and weaknesses of various outcome objectives, and
the measurement of outcomes for AIDS prevention programs.
PROGRAM OBJECTIVES
CDC's overall mission has been "to prevent the spread of REV infection"
(CDC, 1989), an objective shared by the three programs of interest
here. For the media program, campaign efforts were to be focused on the
general population "for an improved understanding of the risk factors [for
REV diffusion, and] . . . for affecting a change in attitude, understanding
and behaviour" to result in "curbing the further spread of AIDS and
HIV" (Ogilvy & Mather, 19871. The stated national program goals of
the new direct grants program that win fund CBOs and He goals of the
testing and counseling project network were: (1) to institute surveillance
programs; (2) "to reduce the risk of ERV infection and to effect, maintain,
measure, and evaluate the significance of behavioral change" among the
general population and high-risk groups; and (3) "to inform and educate
the general public in order to gain broad support" for HIV prevention
programs (CDC, 19881.
The overall goal of reducing the spread of HIV is appropriate from
a "big picture" perspective, but as stated it is inadequate as an evaluation
objective. First, it is too general: it does not specify how the spread of
HIV will be reduced by specific interventions among specific populations
and in specific contexts. Furthermore, it is too distant: reducing the rate
34
OCR for page 35
MEASUREMENT OF OUTCOMES ~ 35
of H[V transmission is a long-term goal that will have to be measured
and remeasured over a long period of time. Finally, focusing on HIV
alone can be misleading In some contexts: for example, In a community
that begins with zero REV prevalence, an outcome of zero prevalence
following an intervention cannot be taken as evidence of the ~ntervention's
effectiveness.
In addition to the overarching goal of eliminating HIV trans-
mission, the pane! recommends that explicit objectives be
written for each of the major intervention programs and
that these objectives be framed as measurable biological,
psychological, and behavioral outcomes.
Objectives should be framed as proximate outcomes of behavior—
that is, as near-te~ or intermediate outcomes that are linked to long-term
goals. In fact, when an intermediate goal is known to lead to a long-term
outcome, the need to evaluate results beyond the first stage is obviated
(Weiss, 1972:38-39~. The long-terTn goal, of course, is the elimination
of the transmission of ~V, or at least its reduction to a reproductive
rate of less than 1.0.i As for proximate outcomes, the pane] believes
that the appropriate, linked, intermediate objectives include (~) accurate
knowledge about EUV risks, (2) the reduction of risk behaviors, and (3)
the adoption of protective behaviors. Biological outcomes are not on this
list because their interpretation is much more problematic than these three
(see discussion below). Of these three intermediate objectives, the pane}
believes that valid and reliable indicators of the frequency of both risk-
associated and protective behaviors (the second and Bird objectives) are
He most appropriate proximate outcomes for AIDS prevention efforts.
The hope of an intervention program is that any changes that occur as a
result of its efforts win show up rapidly in these indicators and that those
changes, if maintained, will lead to He achievement of He program's
long-term goal of reducing REV transmission.
Suchman (1967:3941) has suggested guidelines for goal setting
Hat He panel believes would be helpful to CDC In specifying outcome
objectives, including:
· the content of the objective (e.g., whether a goal is to change
or to eliminate behaviors);
1 Studies of the dynamics of epidemics of measles and smallpox have demonstrated that the reproduc-
tive rate of an infectious disease must exceed 1.0 to sustain an epidemic. When the rate drops below
1.0, the epidemic will dwindle, although small, infected segments of the population may remain. One
unplication of this fact is that, as a first step toward eliminating AIDS, it is not necessary to prevent
all new HIV infections to halt the epidemic spread of HIV.
OCR for page 36
36 ~ EVALUATING AIDS PREVENTION PROGRAMS
· the target audience;
· when the change is to take place (i.e., short-term versus
{ong-term changes);
whether the objectives are unitary or multiple and whether
they have potential side effects;
the desired magnitude of effect; and
· how the objective is to be attained (i.e., the instrumentation
of the program).
One of the guidelines Suchman suggests for setting objectives ad-
uresses the side effects or unwanted (and sometimes unanticipated) con-
sequences that may attend efforts to achieve a particular goal. Side effects
are not a tnvial concern, and the pane] believes that the negative effects
of AIDS interventions should be assessed along with the positive out-
comes. Negative side effects such as excessive fear or psychological
distress—can often be anticipated at the time of program design from
data derived from comparable studies or from anecdote evidence.
A side effect Hat has received attention recently is the impact of
AIDS-related stressors on individuals' mental health. In the case of
testing and counseling, In particular, there is increasing evidence that
for some individuals, learning about one's seropositivity is associated
with certain types of psychological distress symptoms (e.g., anxiety,
anger, depression, sexual dysfunction) and other untoward effects (e.g.,
disruption of one's partner relationship, increased illicit drug use); see,
for example, Coates, Morin, and McKusick (1987), Grant and Anns
(1988), Ostrow et al. (1988), Martin et al. (1989~. It is generally not
the test itself that is He source of the distress; rather, it is He message
an individual receives that he or she has tested positive for HIV and is
susceptible to developing a fatal disease.2 How to deliver this message
and provide the requisite emotional support and medical referral deserve
serious attention.
The pane] recommends that all evaluation protocols pronde
for the assessment of potential harmful effects as well as the
assessment of desired effects.
In He following section the parcel discusses a variety of project
objectives that are suitable outcomes for HIV/AIDS prevention protons,
along win the strengths and weaknesses of each.
2The test itself may be a culprit in certain cases in which it is taken involuntarily, such as in a drug
treatment center, hospital, or prison. More attention to negative effects is thus warranted in the case of
involuntary testing.
OCR for page 37
MEASUREMENT OF OUTCOMES ~ 37
OUTCOMES FOR EVALUATIONS OF
HIV PREVENTION PROGRAMS
There are a wide range of possible evaluation outcome variables, the
choice of which will vary among projects. For example, the cognitive
or behavioral outcomes that are appropriate to a counseling and testing
session at a gay men's health citric will be different from those asso-
ciated with an information program offered in a drug treatment center.
The objective for the former project may be abstinence from anal ~nter-
course; for the latter project, it may be the adoption of using bleach with
Hug paraphernalia. The specific objective or objectives selected by a
community-based organization or a testing and counseling center should
be consonant with the overarching goal of reduced REV transmission, but
this does not mean that they must or should be identical from project to
project. Indeed, the pane! believes Hat CDC should seek advice from
project staffs and communities conceding potential outcome variables.
We discuss potential outcome vanables in three categories: biolog-
ical, behavioral, and psychological. Before discussing He strengths and
weaknesses of each type of outcome, He variables In each category are
described below. Table 2-l summarizes the evaluation outcome measures
discussed In the following pages.
Biological Outcomes
New HIV infection (in the form of seroconversion rates in specific pop-
ulations) is the most informative biological outcome variable to indicate
the spread of the disease among adults or adolescents. This variable is
quite appealing as an outcome measure because it relates Erectly to the
overarching goal of CDC's prevention programs. If reliable data from
representative samples of the target populations could be collected, HIV
incidence rates might prove to be useful indicators of program effec-
tiveness. However, incidence data on HIV infection must be used with
caution because so many factors other than risk reduction efforts such
as the saturation of the virus In a given population influence these rates.
While incidence rates for AIDS might also be used, they could lead
to faulty inferences because they reflect HIV infections contracted several
years prior to the onset of the disease. The most recent estimated mean
incubation period for AIDS is 9.8 years (Longing et al., 1989), and even
this estimate may be revised upward as new cases and longer latencies
are recorded. Rates of sexually transmitted diseases (STDs) may also
be useful, as they should respond (other Dings being equal) to changes
In sexual behaviors that risk HIV transmission. Moreover, data on STD
rates may be useful for validating self-reported behavior changes. Like
OCR for page 38
38 ~ EVALUATING AIDS PREVENTION PROGRAMS
TABLE 2-1 Summary of Possible Evaluation Outcome Measures
I. Biological Outcomes
A. Incidence of HIV infection
B. Fertility rate
C. Incidence of sexually transmitted diseases
H. Behavioral Outcomes
A. Primary prevention behaviors
1. Elimination of risk behaviors
a. Abstinence from all sexual contact
b. Abstinence from all IV drug use
c. Avoidance of anal and vaginal intercourse
d. Avoidance of unsterilized IV dog injection equipment
e. Avoidance of pregnancy by V-positive women
2. Reduction of risk behaviors
a. Monogamy
b. Avoidance of anonymous and extradomestic sex
c. Avoidance of'~shooting gallenes"
3. Protective behaviors
a. Use of condoms
b. Use of anti-HIV spe~micides
c. Use of bleach for cleaning IV drug paraphernalia
d. Participation in needle exchange program
B. Complementary prevention behaviors
1. HIV antibody colmseling
2. HIV antibody testing
3. Enrolling in drug treatment protons
4. DeteImining HIV status of sexual or drug partners
5. Providing names of contacts to public heals agents
6. Using family planning services
7. Personal involvement in HIV prevention program
III. Psychological outcomes
A. Awareness of AIDS and lIIV
B. Knowledge of AIDS and HIV transmission modes
C. Stigmatization of persons with AIDS and HIV infection
OCR for page 39
MEASUREMENT OF OUTCOMES ~ 39
seroconversion data, however, STD rates must be viewed with some
caution, since many other factors influence these rates.
Fertility rates may also be reasonable biological measures for eval-
uating AIDS prevention efforts aimed at certain samples of women, e.g.
those who are seropositive or at high risk of infection such as intravenous
Hug users or the sexual partners of intravenous drug users.
Strengths and Weaknesses of Biological Outcomes
Using serological data on HIV infection as the primary outcome of an
evaluation effort seems intuitively appealing. After all, HIV infection
is exactly what one wants to prevent. In addition, the approach to
evaluation using HIV infections as the outcome variable of interest seems
straightforward: to the extent that there are fewer new cases of infection
among people involved In an intervention program, compared with an
equivalent group of people who were not involved, one would conclude
that the prevention campaign was effective; to the extent that the incidence
rates of HIV infection were similar in both groups, one would conclude
that the interventions had no effect.
But He use of these data is often problematic. There must be a
certain level of occurrence of a particular event (i.e., seroconversions, or
the change from HIV negative to HIV positive) before statistical tests can
determine whether a particular program is effective. HIV seroconversion
is a relatively rare event, even in the highest risk populations in the
United States. Seroconversion among urban gay men was in the range
of 0.5-2.0 percent per year as of 1987. Among East Coast {V drug users,
the rate was probably as high as 3.0-7.0 percent per year, as of 1987.3
such low rates of occurrence, evaluation requires extremely large
samples and long intervals of time before there are enough occurrences
of He event to conduct statistical tests of program effectiveness. Such
evaluation studies must use a longitudinal cohort study design to establish
a fixed denominator of susceptible individuals for all eeaunent and con-
tro} groups. This type of design may require the screening of thousands
of individuals to assemble a sufficiently large HIV-negative cohoIt, a
process that requires great time and expense. In addition, once screening
and study enrollment have been accomplished, reevaluation and tracing
procedures must be implemented. Like screening, these tasks are highly
labor intensive, and they will be very expensive and time-consuming if
sample sizes must be very large.
3 these rates contrast sharply with those in over countries such as Thailand, where seroconversion
rates of more Man 2 percent per month have been observed recently among IV drug users.
OCR for page 40
40 ~ EVALUATING AIDS PREVENTION PROGRAMS
Another factor that argues for limiting the number of evaluation stud-
ies that rely on HIV seroconversion as an outcome is that such studies
tend to focus primarily on those who are uninfected. Although the goal
of [IIV prevention efforts is to keep people uninfected, a significant de-
terminant of a person's remaining HIV negative involves the behavior of
those who are infected that is, HIV-positive individuals. To maximize
prevention of HIV infection, intervention efforts should not be limited
only to the uninfected but should include all members of a population.
Hence, the use of seroconversion as an outcome measure excludes a part
of the population that should be included in prevention programs.
Despite these substantial reservations, however, there may be ~n-
stances in which HIV seroconversion is a major variable of interest in an
evaluation.4 One example would be a study of transmission rates among
couples In which one partner was seropositive. In those cases, it is ad-
visable to include a range of sound behavioral measures in the study
protocols. This approach will not only allow the detection of biological
program effects but will also help in interpreting and understanding the
reasons for the presence or absence of such effects.
Sexually transmitted and blood-borne diseases that occur with higher
incidence rates than MV can sometimes provide a biological outcome of
practical value In the evaluation of AIDS prevention programs. The logic
for the use of other STDs as outcome measures is that the behavioral
changes that protect individuals against HIV will also protect against
gonorrhea, syphilis, chIamydia, hepatitis, and similar diseases. But for
these outcomes, too, low incidence rates may require use of relatively
large samples or long periods of follow-up. However, the Incidence
rates for these diseases are generally much higher than for HIV, and in
some populations the frequency of such infection can be quite high; see,
for example, Strobino's (1987:94-105) review of STD infections among
adolescents.
In considering the use of over STDs and blood-borne infections
as outcomes, two considerations are important. First, evaluations using
such outcome measures will ordinanly require scheduling coccal exam-
~nations or serological testing of the individuals taking part In the study.
4 The panel notes a number of large cohort studies are currently monitoring HIV seroconversion rates
among high-nsk groups in the United States. Many of these studies include components Hat as-
sess knowledge, beliefs, coping, social skills, psychological factors, and social supports, as well as
other variables of interest. However, most of these studies are funded as scientific projects by specific
biomedical agencies (such as the National Institute of Allergy and Infectious Diseases) rather than as
efforts lo protect the public health. Thus, seroconversion rates are being carefully monitored, as are
their potential determinants. These studies are valuable, and the panel notes that the aging cohorts
might well be supplemented with new recruits to enhance the usefulness of their data
OCR for page 41
MEASUREMENT OF OUTCOMES ~ 41
Official STD statistics are not appropriate for use in program evaluations
since they do not identify the individuals who have been exposed to par-
ticular AIDS prevention programs. (These statistics also vary In quality
from locale to locale.) Second, interpreting changes In STD rates can be
difficult without reliable data on the behaviors that place individuals at
risk of infection and data on trends over time in the prevalence of infec-
tion In the local community. It is possible, for example, for an increase
In STD rates to occur In a group under study despite declines in their
frequency of risky behavior. This seemingly paradoxical outcome can
result from rising STD rates in the population at large, which masks the
protective effect of behavioral change in a target group. In this situation,
fewer episodes of risky behavior enacted with parmers who are more
likely to be infected may result in an increased overall risk of infection.
Behavioral Outcomes
The list of possible behavioral outcomes for evaluating HIV prevention
programs is quite extensive and can be subdivided In a number of ways.
The first main distinction is between those behaviors that have a direct
influence on the risk of acquiring HIV infection and those that have an
indirect influence. The former can be called primary prevention behav-
lore because adopting or not adopting them has a direct bearing on an
~ndividual's chances of infection. The latter can be called complementary
prevention behaviors because they increase the likelihood of engaging in
primary prevention behaviors but do not themselves alter the risk of
infection.
A further useful distinction can be made among primary prevention
behaviors between those behaviors that elf nate or reduce the risk of
infection and those that increase the ability to protect against infection.
Eliminating or reducing risk generally involves the elimination of a
behavior from the behavioral repertoire; increasing protection involves
awing a new behavior. Each of these processes is psychologically distinct
in terms of the skills required, and each of them may require different
types of intervention or prevention programs. Even if both the acquisition
and elimination of behaviors are included In a prevention program, the
standards used to evaluate the effectiveness of the program (e.g., what
size of effect will be considered important) may vary, depending on
whether the focus is on risk reduction or increased protection.
Primary Prevention Behaviors
Risk Reduction. The first group of behaviors that have a direct influence
On HIV transmission includes anal and vaginal intercourse; the use of
OCR for page 42
42 ~ EVALUATING AIDS PREVENTION PROGRAMS
nonsterile drug injection equipment; and pregnancy for women who are
HIV seropositive.5 These behaviors account for more than 96 percent of
all new cases of HIV infection occurring worldwide, and they are the
central means by which the infection is passed from person to person. Any
infected individual who avoids these three behaviors virtually eliminates
the possibility that he or she will transmit HIV to an uninfected person;
any uninfected individual who avoids these behaviors virtually eliminates
the possibility that he or she will acquire HIV from an infected person.
A second group of direct behaviors that may be of interest in evalua-
tion research on HIV prevention programs involve the partial elimination
of risk. These behaviors include monogamy; avoidance of extradomes-
tic sexual activity (e.g., sex in bathhouses, backwood bars, and so on);
and avoidance of drug "shooting galleries." However, these behavioral
outcomes are not as useful or informative as those involving intercourse,
shared drug injection equipment, and pregnancy because the extent to
which they actually contribute to a person's risk reduction depends on
the characteristics of his or her sexual or drug shooting partners and on
the particular behaviors in which all of these inclividuals engage.
Protective Behaviors. A third set of behaviors that bear directly
on HIV transmission emphasizes increaser! protection against risk rather
than the elimination or reduction of risk-taking activity. These behaviors
include the use of battier contraception (concloms) dunng anal or vaginal
intercourse; the use of anti-HIV agents (contained In some spermicides);
the cleaning of injection equipment with bleach and water prior to use;
and participation in a neeclle exchange program. The adoption of these
behaviors are only important among those individuals who continue to
engage in the primary HIV risk behaviors anal or vaginal intercourse
· . .
Or c rug 1nJecuon.
Complementary Prevention Behaviors
As defined above, complementary prevention behaviors have an indirect
beating on HIV transmission risk. By increasing an individual's chances
of either practicing risk elimination or risk reduction behavior or of
engaging in protective behavior; they do not in themselves reduce an in-
dividual's risk of acquiring or transmitting the virus. Nevertheless, these
behaviors may be important as evaluation outcomes of HIV prevention
programs. Complementary prevention behaviors include HIV antibody
counseling; HIV antibody testing; enrollment in a drug treatment pro-
gram; informing sexual and drug partners of one's HIV status; providing
l
5Pregnancy, of course, is a condition, not a behavior. It is used here as shorthand for the behaviors that
constitute the risk of transmission of HIV from an infected mother to an unborn child.
OCR for page 43
MEASUREMENT OF OUTCOMES ~ 43
names of drug and sexual contacts to public health officials; the use of
family planrung services; and personal involvement in or establishment
of an HIV prevention program.
Some of these complementary prevention behaviors are themselves
part of a prevention or intervention program that is in need of evaluation.
For example, there is currently great interest In evaluating whether HIV
counseling and testing programs help people to reduce risk or to practice
protection. There have been similar calls for assessing whether partner
notification (of sexual or drug partners) helps to prevent HIV infection.
Despite the lack of empirical data on the effectiveness of these comple-
mentary behaviors in preventing HIV infection, they remain defensible
evaluation outcomes if one is willing to assume that Hey promote risk
reduction or protection.
Strengths and Weaknesses of Behavioral Outcomes
The past eight years of research on AIDS and HIV transmission have
confirmed the specific behaviors that are overwhelmingly responsible
for HIV infection. Consequently, it is not necessary to rely on EUV
seroconversion as the primary outcome of evaluation studies of HIV
prevention programs because those specific behaviors can be targeted for
assessment.
In contrast to studies that rely on seroconversion as the primary
outcome, studies that rely on behavioral outcomes can be conducted
more quickly and less expensively because they need not calTy out He
extensive screening required in seroconversion studies. (This is not to
say, however, that behavioral studies are inexpensive.) Because both
primary and complementary risk behaviors occur at rates that far exceed
the rate of seroconversions, it is possible to make reliable estimates of
specific behaviors using smaller samples thereby making the research
less costly than for seroconversion studies.
In addition, a reliance on behavioral outcomes rather than serocon-
version circumvents the problem of the reluctance of some people to be
tested. REV antibody testing is not routine among He general popula-
tion, although the picture is mixed among high-nsk Individuals. There is
evidence from a number of cohort studies of He willingness of gay men
to be tested, although what proportion of people in those communities is
wining to be tested is unknown. But there is evidence that many indi-
viduals in drug treatment programs or STD clinics are not willing to be
tested (see, e.g., Hull et al., 1988; Fleming et al., 1989~. Thus, if AIDS
prevention programs and their companion evaluation studies can avoid
the need to conduct HIV testing, they may be likely to reach a wider and
more representative portion of their target populations.
OCR for page 44
44 ~ EVALUATING AIDS PREVENTION PROGRAMS
For those evaluation protocols that must include HIV testing, the
use of behavioral outcomes is nonetheless important for adequate eval-
uation. For an individual who is REV positive, the primary behaviors
of interest involve behaviors that impose risk on others; e.g., allowing
another person to use his or her unsterilized IV drug injection equipment,
becoming pregnant. For those who are HIV negative, the primary behav-
iors of interest involve risk taking; e.g., using unsterilized Hug injection
equipment.
The main weaknesses of behavioral data involve problems of validity
and reliability (see Appendix C). Problems also arise in interpreting
data about similar outcomes when wordings of questions differ among
studies. Moreover, even the interpretation of the same question may
be problematic over time because respondents' willingness to report
behaviors may change.
Another problem In research based on self-reports involves respon-
dents' ability to remember and accurately report specific events and
behaviors (there is a fairly large literature on this topic). The problem
of recall often may be decreased by shortening the time frame over
which respondents must recall the occurrence and frequency of behav-
iors. However, a shorter time frame may change He meaning of a
particular outcome: celibacy or monogamy for a week, for example, is
quite different from celibacy or monogamy for a year.
Another major issue surrounding behavioral outcomes involves which
outcomes to emphasize In prevention programs and which to focus on
in evaluation studies. While risk reduction and risk elimination are most
desirable outcomes Tom the standpoint of halting the spread of HIV,
protective behaviors may be more practical to teach and maintain.
Psychological Outcomes
There are a handful of psychological outcomes that are also of interest
from the standpoint of AIDS prevention. These include an awareness of
AIDS and HIV and of the gravity of the problems they pose; knowledge
of AIDS and HIV transmission; and stigmatization of individuals who
have AIDS or who are infected with HIV. These psychological outcomes
do not have a direct bearing on HIV infection in the way that intercourse,
drug injection, and pregnancy (for seropositive women) do. Never~e-
less, differences in the degree to which Individuals are aware of AIDS,
understand which behaviors transmit HIV, and disparage or devalue those
who are id or infected may be important detenn~nants of whether they
adopt risk reduction or protective behaviors.
OCR for page 45
MEASUREMENT OF OUTCOMES ~ 45
Of all the possible goals for the educational campaigns that have
been mounted In this country, the greatest progress has been made in
the area of increased knowledge and awareness of AIDS (Turner, Miller,
and Barker, 19881. This must be seen as an important accomplishment
because behavioral change is unlikely to occur without knowledge and
awareness of the AIDS problem and the means of HIV transmission.
How much of the credit for this increased knowledge belongs to Public
Health Service efforts, as compared with efforts ong~nating from other
educational institutions, personal experience, news stories, and so form,
can probably never be accurately assessed. From a public health stand-
po~nt, it is not crucial to make such a determination. Instead, one of the
most important public health goals now should be to maintain the current
high level of knowledge and awareness that exists In the population about
the risk of REV transmission by sexual contact and the sharing of infected
equipment and dispelling misconceptions about the risk of transmission
through casual contact.
Strengths and Weaknesses of Psychological Outcomes
Psychological outcomes that Involve awareness of AIDS and HIV, knowI-
edge about AIDS and EUV transmission, and attitudes toward those who
are infected and iB may be the easiest outcomes to measure and study, In
the sense that these measures are easily incorporated In a survey ques-
tionna~re. Nonetheless, the development of specific questions must be
carefully done. Questions involving knowledge and attitudes must be
sensitively crafted as well as clearly worded In the vernacular of the
target population to avoid introducing substantial response bias and error.
Similarly, Here is a need to test the sensitivity of the measurements to al-
ternative question wordings or interview formats (see Turner and Martin,
1984: Volume I, Ch. 4-5~.
EVALUATION MEASURES
There are several methodological issues involved In developing reliable
indicators of changes in knowledge, attitudes, and risky behaviors. Two
issues in particular challenge He measurement of program effectiveness:
(~) detellIiining the appropriate time intervals between program imple-
mentation and the measurement of change and (2) assessing the quality of
survey instruments (questionnaires) and the resulting data. Some of the
issues addressed in the sections below are generic to program evaluation;
others are more specifically related to the measurement of unobserved
phenomena, such as sexual behavior or attitudes.
OCR for page 46
46
EVALUATING AIDS PREVENTION PROGRAMS
Timing of Measurement
People who adopt new behaviors in the face of the epidemic may dis-
continue the change, relapse from time to time, or, conversely, enhance
the change over time. Measurements taken soon after the intervention
has been delivered may show substantial change but that change may
decay over time. Because immediate post~ntervention measurements and
delayed measurements may yield different results, periodic measurement
is desirable.
One other measurement timing issue warrants mention here. The
national media campaign (see Chapter 3) changed its goal and even its
target audience before measuring any outcomes. Thus, four separate
phases of the campaign were launched without measuring what occurred
during previous phases. Such haste is dysfunctional. It does not permit
evidence to be gathered on what it is about the intervention that works
or does not work or which elements should be continued or discontinued
in subsequent phases. The panel cautions that over CDC programs,
such as the forthcoming program of direct grants to community-based
organizations (CBOs) (see Chapter 4) arid other projects with multiple
objectives are vulnerable to shifting program goals.
The consequences of interventions should be measured and assessed
before deciding on new or alternative stages of intervention programs,
a policy that will require periodic measurement of program outcomes.
In addition, even if a well-planned, well-unplemented project has been
successful In meeting its specified goal or goals, periodic evaluation
efforts should not cease. For example, media messages may lose their
impact with repeated use, the audience may change as the population
ages, Individuals may feel secure and become careless In their behaviors,
or the program itself may drift In its purposes.
The pane' recommends that once goals are met, projects be
reevaluated periodically to monitor their continued eiTective-
ness.
Quality of Measures
There are a number of methodological topics that are related to the qual-
ity of survey data: sampling hard-to-reach groups, appropriate designs
for intervention research and evaluation, the validity of self-reported in-
formation, and the construction of reliable questionnaire items. Because
Of the importance of data quality when trying to determine whedler pro-
gram objectives have been reached, Appendix C includes a chapter from
the recent report of the parent committee (Miller, Turner, and Moses,
OCR for page 47
MEASUREMENT OF OUTCOMES ~ 47
1990) reviewing methodological research on the accuracy of surveys
measurements of sexual and drug using behaviors.
There are two general methodological issues that should be con-
sidered in an assessment of evaluation measures: validity and reliability.
Valid measures reflect without bias the presence or intensity of the concept
that Hey intend to quantify. (Bias refers to systematic misrepresentation
or other systematic error which may be due to faulty measurement, sam-
pling, or other factors.) Reliable measures will yield the same data if
applied repeatedly. In considering the difficulties associated with AIDS
evaluation measures, the pane] believes that measurement validity will
pose Me most serious and difficult problem. With regard to reliability,
the pane} notes a small but important set of studies that have begun to
exaniine the reliability of sexual behavior data, collected mainly from gay
men, and a limited number of small studies that have examined the reli-
ability of measures of behavioral change made In response to the AIDS
epidemic: see, for example, Saltzman and colleagues (19871; Catania
and colleagues (19901; Martin and colleagues (19891; and Martin and
Dean (1989~.
Differential validity, that is, nonequivalent measurement biases in
treatment and control groups, is a particularly worrisome concern for
outcome evaluations. It is understandable that some individuals would
be reluctant to report risky sexual or drug use behaviors. In an outcome
evaluation experiment that randomly assigns participants to different
treatments, this reporting bias can be expected to be equivalent across
groups at the beginning of the experiment. However, the social pressures
created by an intervention program can differentially affect the report-
~ng bias. Individuals enrolled in Intervention programs should fee! an
increase in psychological and social pressure to refrain from engaging in
risk-associated behavior. Consequently, they may also feel pressure to
conceal conduct that is at odds with program objectives. In such a cir-
cumstance, differential reporting biases may cause the group that receives
the intervention to appear to have adopted safer behavioral patterns than
He control group, when in fact the observed effect is due to the effect of
the intervention on the reporting bias.
The quality of measures will also depend on the level of detail of
information elicited by the survey instrument. For example, to evaluate
He effectiveness of an intervention program, it is not be sufficient to
know whether respondents have ever used condoms or whether they
have begun to use them. It is also important to determine the frequency
of use, and it will be helpful to ascertain the conditions that foster use.
This last point underscores the need to collect data that are not only
OCR for page 48
48 | EVALUATING AIDS PREVENTION PROGRAMS
valid and reliable but that are also meaningful. Outcome measures should
include comprehensive and up-to-date observations of the attainment of a
program's explicitly stated objectives as well as anticipated side effects,
if any. Careful measurement in the context of good research design and
implementation will then allow thoughtful inferences to be made about
what the results of evaluation mean.
REFERENCES
Catania, J. A., Gibson, D. R., Marin, B., Coates, T. J., and Greenblatt, R. M.
(1990) Response bias In assessing sexual behaviors relevant to HIV transmission.
Evaluation and Program Planning 13: 19-29.
Centers for Disease Control (CDC) (1988) Announcement No. 901. September 20.
Federal Register 53 (182~:36492-36493.
Centers for Disease Control (CDC) (1989) A Comprehensive Program to Prevent
HIV Transmission. Fiscal Year 1989 Operating Plan. Washington, D.C.: U.S.
Department of Health and Human Services.
Coates, T. J., Morin, S.F., and McKusick, L. (1987) Behavioral consequences of AIDS
antibody testing among gay men. Journal of the American Medical Association
258:1889.
Fleming, D., Bennett, D., Klockner, R., Gould, J., Cassidy, D., and Foster, L. (1989)
HIV Infected STD Clients Who Decline HIV Counseling and Testing. Paper
presented at the Fifth International AIDS Conference. Montreal, June 4-9.
Grant, D., and Anns, M. (1988) Counseling AIDS antibody-positive clients: Reactions
and treatment. American Psychologist 43:72-74.
Hull, H. F., Bettinger, C. J., Gallaher, M. M., Keller, N. M., Wilson, J., and Mertz,
G. J. (1988) Companson of HIV-antibody prevalence in patients consenting to
and declining HIV-antibody testing in an STD clinic. Journal of the American
Medical Association 260:935-938.
Longini, I. M., Jr., Clark, W. S., Horsburgh, C. R., Lemp, G. F., Byers, R. H., Darrow,
W. W., and others (1989) Statistical analysis of the stages of HIV infection using
a Markov model. Paper presented at the Fifth International AIDS Conference.
Montreal, June 4-9.
Martin, J. L., and Dean, L. (1989) Risk factors for AIDS-related bereavement in a
cohort of homosexual men in New York City. In B. Cooper and T. Helgason,
eds., Epidemiology and the Prevention of Mental Disorders. United Kingdom:
Routledge.
Martin, J. L., Dean, L., Garcia, M., and Hall, W. (1989) The impact of AIDS on a
gay community: Changes in sexual behavior, substance use, and mental health.
American Journal of Community Psychology. 17~3~:269-293
Miller, H. G., Turner, C. F., and Moses, L. E., eds. (1990) AIDS: The Second Decade.
Report of the NBC Committee on AIDS Research and the Behavioral, Social and
Statistical Sciences. Washington, D.C.: National Academy Press.
Ogilvy & Mather (1987) Contract for REP No. 200-87-0525. Section m (Methodology
and Approach). June.
Ostrow, D. G., Joseph, J., Soucey, J., Eller, M., Kessler, R., Phair, J., and Chmiel,
J. (1988) Mental health and behavioral correlates of HIV antibody testing in a
cohort of gay men. Paper presented at the Fourth International AIDS Conference.
Stockholm, June 12-16.
OCR for page 49
MEASUREMENT OF OUTCOMES ~ 49
Saltzman, S. P., Stoddard, A. M., McCusher, J., Moon, M. W., and Mayer, K. H.
(1987) Reliability of self-reported sexual behavior risk factors for HIV infection
in homosexual men. Public Health Reports 102:692-697.
Strobino, D. M. (1987) The health and medical consequences of adolescent sexuality
and pregnancy: A review of the literature. In S. Hofferth and C. Hayes, eds.,
Risking the Future: Adolescent Sexuality, Pregnancy, and Childbearing. Volume
2, Working Papers and Statistical Appendixes. Report of the NRC Panel on
Adolescent Pregnancy and Childbearing. Washington D.C.: National Academy
Press.
Suchman, E. A. (1967) Evaluative Research. New York: Russell Sage Foundation.
Turner, C. F., and Martin E., eds. (1984) Surveying Subjective Phenomena, 2 vols.
New York: Russell Sage.
Turner, C. F., Miller, H. G., and Barker, L. (1988) AIDS research and the behavioral
and social sciences. In R. Kulstad, ea., AIDS, 1988. Washington, D.C.: American
Association for the Advancement of Science.
Turner, C. F., Miller, H. G., and Moses, L. E., eds. (1989) AIDS, Sexual Behavior,
and Intravenous Drug Use. Report of the NRC Committee on AIDS Research
and He Behavioral, Social, and Statistical Sciences. Washington, D.C.: National
Academy Press.
Weiss, C. H. (1972) Evaluation Research. Englewood Cliffs, N.J.: Prentice-Hall, Inc.
Representative terms from entire chapter:
hiv infection
