Click for next page ( 35


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 34
Measurement of Outcomes Before an evaluation can be designed, the goals of He intervention program must be set. Indeed, goal sewing should precede the design and implementation of a program. This chapter provides an overview of goal sewing, the strengths and weaknesses of various outcome objectives, and the measurement of outcomes for AIDS prevention programs. PROGRAM OBJECTIVES CDC's overall mission has been "to prevent the spread of REV infection" (CDC, 1989), an objective shared by the three programs of interest here. For the media program, campaign efforts were to be focused on the general population "for an improved understanding of the risk factors [for REV diffusion, and] . . . for affecting a change in attitude, understanding and behaviour" to result in "curbing the further spread of AIDS and HIV" (Ogilvy & Mather, 19871. The stated national program goals of the new direct grants program that win fund CBOs and He goals of the testing and counseling project network were: (1) to institute surveillance programs; (2) "to reduce the risk of ERV infection and to effect, maintain, measure, and evaluate the significance of behavioral change" among the general population and high-risk groups; and (3) "to inform and educate the general public in order to gain broad support" for HIV prevention programs (CDC, 19881. The overall goal of reducing the spread of HIV is appropriate from a "big picture" perspective, but as stated it is inadequate as an evaluation objective. First, it is too general: it does not specify how the spread of HIV will be reduced by specific interventions among specific populations and in specific contexts. Furthermore, it is too distant: reducing the rate 34

OCR for page 34
MEASUREMENT OF OUTCOMES ~ 35 of H[V transmission is a long-term goal that will have to be measured and remeasured over a long period of time. Finally, focusing on HIV alone can be misleading In some contexts: for example, In a community that begins with zero REV prevalence, an outcome of zero prevalence following an intervention cannot be taken as evidence of the ~ntervention's effectiveness. In addition to the overarching goal of eliminating HIV trans- mission, the pane! recommends that explicit objectives be written for each of the major intervention programs and that these objectives be framed as measurable biological, psychological, and behavioral outcomes. Objectives should be framed as proximate outcomes of behavior that is, as near-te~ or intermediate outcomes that are linked to long-term goals. In fact, when an intermediate goal is known to lead to a long-term outcome, the need to evaluate results beyond the first stage is obviated (Weiss, 1972:38-39~. The long-terTn goal, of course, is the elimination of the transmission of ~V, or at least its reduction to a reproductive rate of less than 1.0.i As for proximate outcomes, the pane] believes that the appropriate, linked, intermediate objectives include (~) accurate knowledge about EUV risks, (2) the reduction of risk behaviors, and (3) the adoption of protective behaviors. Biological outcomes are not on this list because their interpretation is much more problematic than these three (see discussion below). Of these three intermediate objectives, the pane} believes that valid and reliable indicators of the frequency of both risk- associated and protective behaviors (the second and Bird objectives) are He most appropriate proximate outcomes for AIDS prevention efforts. The hope of an intervention program is that any changes that occur as a result of its efforts win show up rapidly in these indicators and that those changes, if maintained, will lead to He achievement of He program's long-term goal of reducing REV transmission. Suchman (1967:3941) has suggested guidelines for goal setting Hat He panel believes would be helpful to CDC In specifying outcome objectives, including: the content of the objective (e.g., whether a goal is to change or to eliminate behaviors); 1 Studies of the dynamics of epidemics of measles and smallpox have demonstrated that the reproduc- tive rate of an infectious disease must exceed 1.0 to sustain an epidemic. When the rate drops below 1.0, the epidemic will dwindle, although small, infected segments of the population may remain. One unplication of this fact is that, as a first step toward eliminating AIDS, it is not necessary to prevent all new HIV infections to halt the epidemic spread of HIV.

OCR for page 34
36 ~ EVALUATING AIDS PREVENTION PROGRAMS the target audience; when the change is to take place (i.e., short-term versus {ong-term changes); whether the objectives are unitary or multiple and whether they have potential side effects; the desired magnitude of effect; and how the objective is to be attained (i.e., the instrumentation of the program). One of the guidelines Suchman suggests for setting objectives ad- uresses the side effects or unwanted (and sometimes unanticipated) con- sequences that may attend efforts to achieve a particular goal. Side effects are not a tnvial concern, and the pane] believes that the negative effects of AIDS interventions should be assessed along with the positive out- comes. Negative side effects such as excessive fear or psychological distresscan often be anticipated at the time of program design from data derived from comparable studies or from anecdote evidence. A side effect Hat has received attention recently is the impact of AIDS-related stressors on individuals' mental health. In the case of testing and counseling, In particular, there is increasing evidence that for some individuals, learning about one's seropositivity is associated with certain types of psychological distress symptoms (e.g., anxiety, anger, depression, sexual dysfunction) and other untoward effects (e.g., disruption of one's partner relationship, increased illicit drug use); see, for example, Coates, Morin, and McKusick (1987), Grant and Anns (1988), Ostrow et al. (1988), Martin et al. (1989~. It is generally not the test itself that is He source of the distress; rather, it is He message an individual receives that he or she has tested positive for HIV and is susceptible to developing a fatal disease.2 How to deliver this message and provide the requisite emotional support and medical referral deserve serious attention. The pane] recommends that all evaluation protocols pronde for the assessment of potential harmful effects as well as the assessment of desired effects. In He following section the parcel discusses a variety of project objectives that are suitable outcomes for HIV/AIDS prevention protons, along win the strengths and weaknesses of each. 2The test itself may be a culprit in certain cases in which it is taken involuntarily, such as in a drug treatment center, hospital, or prison. More attention to negative effects is thus warranted in the case of involuntary testing.

OCR for page 34
MEASUREMENT OF OUTCOMES ~ 37 OUTCOMES FOR EVALUATIONS OF HIV PREVENTION PROGRAMS There are a wide range of possible evaluation outcome variables, the choice of which will vary among projects. For example, the cognitive or behavioral outcomes that are appropriate to a counseling and testing session at a gay men's health citric will be different from those asso- ciated with an information program offered in a drug treatment center. The objective for the former project may be abstinence from anal ~nter- course; for the latter project, it may be the adoption of using bleach with Hug paraphernalia. The specific objective or objectives selected by a community-based organization or a testing and counseling center should be consonant with the overarching goal of reduced REV transmission, but this does not mean that they must or should be identical from project to project. Indeed, the pane! believes Hat CDC should seek advice from project staffs and communities conceding potential outcome variables. We discuss potential outcome vanables in three categories: biolog- ical, behavioral, and psychological. Before discussing He strengths and weaknesses of each type of outcome, He variables In each category are described below. Table 2-l summarizes the evaluation outcome measures discussed In the following pages. Biological Outcomes New HIV infection (in the form of seroconversion rates in specific pop- ulations) is the most informative biological outcome variable to indicate the spread of the disease among adults or adolescents. This variable is quite appealing as an outcome measure because it relates Erectly to the overarching goal of CDC's prevention programs. If reliable data from representative samples of the target populations could be collected, HIV incidence rates might prove to be useful indicators of program effec- tiveness. However, incidence data on HIV infection must be used with caution because so many factors other than risk reduction efforts such as the saturation of the virus In a given population influence these rates. While incidence rates for AIDS might also be used, they could lead to faulty inferences because they reflect HIV infections contracted several years prior to the onset of the disease. The most recent estimated mean incubation period for AIDS is 9.8 years (Longing et al., 1989), and even this estimate may be revised upward as new cases and longer latencies are recorded. Rates of sexually transmitted diseases (STDs) may also be useful, as they should respond (other Dings being equal) to changes In sexual behaviors that risk HIV transmission. Moreover, data on STD rates may be useful for validating self-reported behavior changes. Like

OCR for page 34
38 ~ EVALUATING AIDS PREVENTION PROGRAMS TABLE 2-1 Summary of Possible Evaluation Outcome Measures I. Biological Outcomes A. Incidence of HIV infection B. Fertility rate C. Incidence of sexually transmitted diseases H. Behavioral Outcomes A. Primary prevention behaviors 1. Elimination of risk behaviors a. Abstinence from all sexual contact b. Abstinence from all IV drug use c. Avoidance of anal and vaginal intercourse d. Avoidance of unsterilized IV dog injection equipment e. Avoidance of pregnancy by V-positive women 2. Reduction of risk behaviors a. Monogamy b. Avoidance of anonymous and extradomestic sex c. Avoidance of'~shooting gallenes" 3. Protective behaviors a. Use of condoms b. Use of anti-HIV spe~micides c. Use of bleach for cleaning IV drug paraphernalia d. Participation in needle exchange program B. Complementary prevention behaviors 1. HIV antibody colmseling 2. HIV antibody testing 3. Enrolling in drug treatment protons 4. DeteImining HIV status of sexual or drug partners 5. Providing names of contacts to public heals agents 6. Using family planning services 7. Personal involvement in HIV prevention program III. Psychological outcomes A. Awareness of AIDS and lIIV B. Knowledge of AIDS and HIV transmission modes C. Stigmatization of persons with AIDS and HIV infection

OCR for page 34
MEASUREMENT OF OUTCOMES ~ 39 seroconversion data, however, STD rates must be viewed with some caution, since many other factors influence these rates. Fertility rates may also be reasonable biological measures for eval- uating AIDS prevention efforts aimed at certain samples of women, e.g. those who are seropositive or at high risk of infection such as intravenous Hug users or the sexual partners of intravenous drug users. Strengths and Weaknesses of Biological Outcomes Using serological data on HIV infection as the primary outcome of an evaluation effort seems intuitively appealing. After all, HIV infection is exactly what one wants to prevent. In addition, the approach to evaluation using HIV infections as the outcome variable of interest seems straightforward: to the extent that there are fewer new cases of infection among people involved In an intervention program, compared with an equivalent group of people who were not involved, one would conclude that the prevention campaign was effective; to the extent that the incidence rates of HIV infection were similar in both groups, one would conclude that the interventions had no effect. But He use of these data is often problematic. There must be a certain level of occurrence of a particular event (i.e., seroconversions, or the change from HIV negative to HIV positive) before statistical tests can determine whether a particular program is effective. HIV seroconversion is a relatively rare event, even in the highest risk populations in the United States. Seroconversion among urban gay men was in the range of 0.5-2.0 percent per year as of 1987. Among East Coast {V drug users, the rate was probably as high as 3.0-7.0 percent per year, as of 1987.3 such low rates of occurrence, evaluation requires extremely large samples and long intervals of time before there are enough occurrences of He event to conduct statistical tests of program effectiveness. Such evaluation studies must use a longitudinal cohort study design to establish a fixed denominator of susceptible individuals for all eeaunent and con- tro} groups. This type of design may require the screening of thousands of individuals to assemble a sufficiently large HIV-negative cohoIt, a process that requires great time and expense. In addition, once screening and study enrollment have been accomplished, reevaluation and tracing procedures must be implemented. Like screening, these tasks are highly labor intensive, and they will be very expensive and time-consuming if sample sizes must be very large. 3 these rates contrast sharply with those in over countries such as Thailand, where seroconversion rates of more Man 2 percent per month have been observed recently among IV drug users.

OCR for page 34
40 ~ EVALUATING AIDS PREVENTION PROGRAMS Another factor that argues for limiting the number of evaluation stud- ies that rely on HIV seroconversion as an outcome is that such studies tend to focus primarily on those who are uninfected. Although the goal of [IIV prevention efforts is to keep people uninfected, a significant de- terminant of a person's remaining HIV negative involves the behavior of those who are infected that is, HIV-positive individuals. To maximize prevention of HIV infection, intervention efforts should not be limited only to the uninfected but should include all members of a population. Hence, the use of seroconversion as an outcome measure excludes a part of the population that should be included in prevention programs. Despite these substantial reservations, however, there may be ~n- stances in which HIV seroconversion is a major variable of interest in an evaluation.4 One example would be a study of transmission rates among couples In which one partner was seropositive. In those cases, it is ad- visable to include a range of sound behavioral measures in the study protocols. This approach will not only allow the detection of biological program effects but will also help in interpreting and understanding the reasons for the presence or absence of such effects. Sexually transmitted and blood-borne diseases that occur with higher incidence rates than MV can sometimes provide a biological outcome of practical value In the evaluation of AIDS prevention programs. The logic for the use of other STDs as outcome measures is that the behavioral changes that protect individuals against HIV will also protect against gonorrhea, syphilis, chIamydia, hepatitis, and similar diseases. But for these outcomes, too, low incidence rates may require use of relatively large samples or long periods of follow-up. However, the Incidence rates for these diseases are generally much higher than for HIV, and in some populations the frequency of such infection can be quite high; see, for example, Strobino's (1987:94-105) review of STD infections among adolescents. In considering the use of over STDs and blood-borne infections as outcomes, two considerations are important. First, evaluations using such outcome measures will ordinanly require scheduling coccal exam- ~nations or serological testing of the individuals taking part In the study. 4 The panel notes a number of large cohort studies are currently monitoring HIV seroconversion rates among high-nsk groups in the United States. Many of these studies include components Hat as- sess knowledge, beliefs, coping, social skills, psychological factors, and social supports, as well as other variables of interest. However, most of these studies are funded as scientific projects by specific biomedical agencies (such as the National Institute of Allergy and Infectious Diseases) rather than as efforts lo protect the public health. Thus, seroconversion rates are being carefully monitored, as are their potential determinants. These studies are valuable, and the panel notes that the aging cohorts might well be supplemented with new recruits to enhance the usefulness of their data

OCR for page 34
MEASUREMENT OF OUTCOMES ~ 41 Official STD statistics are not appropriate for use in program evaluations since they do not identify the individuals who have been exposed to par- ticular AIDS prevention programs. (These statistics also vary In quality from locale to locale.) Second, interpreting changes In STD rates can be difficult without reliable data on the behaviors that place individuals at risk of infection and data on trends over time in the prevalence of infec- tion In the local community. It is possible, for example, for an increase In STD rates to occur In a group under study despite declines in their frequency of risky behavior. This seemingly paradoxical outcome can result from rising STD rates in the population at large, which masks the protective effect of behavioral change in a target group. In this situation, fewer episodes of risky behavior enacted with parmers who are more likely to be infected may result in an increased overall risk of infection. Behavioral Outcomes The list of possible behavioral outcomes for evaluating HIV prevention programs is quite extensive and can be subdivided In a number of ways. The first main distinction is between those behaviors that have a direct influence on the risk of acquiring HIV infection and those that have an indirect influence. The former can be called primary prevention behav- lore because adopting or not adopting them has a direct bearing on an ~ndividual's chances of infection. The latter can be called complementary prevention behaviors because they increase the likelihood of engaging in primary prevention behaviors but do not themselves alter the risk of infection. A further useful distinction can be made among primary prevention behaviors between those behaviors that elf nate or reduce the risk of infection and those that increase the ability to protect against infection. Eliminating or reducing risk generally involves the elimination of a behavior from the behavioral repertoire; increasing protection involves awing a new behavior. Each of these processes is psychologically distinct in terms of the skills required, and each of them may require different types of intervention or prevention programs. Even if both the acquisition and elimination of behaviors are included In a prevention program, the standards used to evaluate the effectiveness of the program (e.g., what size of effect will be considered important) may vary, depending on whether the focus is on risk reduction or increased protection. Primary Prevention Behaviors Risk Reduction. The first group of behaviors that have a direct influence On HIV transmission includes anal and vaginal intercourse; the use of

OCR for page 34
42 ~ EVALUATING AIDS PREVENTION PROGRAMS nonsterile drug injection equipment; and pregnancy for women who are HIV seropositive.5 These behaviors account for more than 96 percent of all new cases of HIV infection occurring worldwide, and they are the central means by which the infection is passed from person to person. Any infected individual who avoids these three behaviors virtually eliminates the possibility that he or she will transmit HIV to an uninfected person; any uninfected individual who avoids these behaviors virtually eliminates the possibility that he or she will acquire HIV from an infected person. A second group of direct behaviors that may be of interest in evalua- tion research on HIV prevention programs involve the partial elimination of risk. These behaviors include monogamy; avoidance of extradomes- tic sexual activity (e.g., sex in bathhouses, backwood bars, and so on); and avoidance of drug "shooting galleries." However, these behavioral outcomes are not as useful or informative as those involving intercourse, shared drug injection equipment, and pregnancy because the extent to which they actually contribute to a person's risk reduction depends on the characteristics of his or her sexual or drug shooting partners and on the particular behaviors in which all of these inclividuals engage. Protective Behaviors. A third set of behaviors that bear directly on HIV transmission emphasizes increaser! protection against risk rather than the elimination or reduction of risk-taking activity. These behaviors include the use of battier contraception (concloms) dunng anal or vaginal intercourse; the use of anti-HIV agents (contained In some spermicides); the cleaning of injection equipment with bleach and water prior to use; and participation in a neeclle exchange program. The adoption of these behaviors are only important among those individuals who continue to engage in the primary HIV risk behaviors anal or vaginal intercourse . . Or c rug 1nJecuon. Complementary Prevention Behaviors As defined above, complementary prevention behaviors have an indirect beating on HIV transmission risk. By increasing an individual's chances of either practicing risk elimination or risk reduction behavior or of engaging in protective behavior; they do not in themselves reduce an in- dividual's risk of acquiring or transmitting the virus. Nevertheless, these behaviors may be important as evaluation outcomes of HIV prevention programs. Complementary prevention behaviors include HIV antibody counseling; HIV antibody testing; enrollment in a drug treatment pro- gram; informing sexual and drug partners of one's HIV status; providing l 5Pregnancy, of course, is a condition, not a behavior. It is used here as shorthand for the behaviors that constitute the risk of transmission of HIV from an infected mother to an unborn child.

OCR for page 34
MEASUREMENT OF OUTCOMES ~ 43 names of drug and sexual contacts to public health officials; the use of family planrung services; and personal involvement in or establishment of an HIV prevention program. Some of these complementary prevention behaviors are themselves part of a prevention or intervention program that is in need of evaluation. For example, there is currently great interest In evaluating whether HIV counseling and testing programs help people to reduce risk or to practice protection. There have been similar calls for assessing whether partner notification (of sexual or drug partners) helps to prevent HIV infection. Despite the lack of empirical data on the effectiveness of these comple- mentary behaviors in preventing HIV infection, they remain defensible evaluation outcomes if one is willing to assume that Hey promote risk reduction or protection. Strengths and Weaknesses of Behavioral Outcomes The past eight years of research on AIDS and HIV transmission have confirmed the specific behaviors that are overwhelmingly responsible for HIV infection. Consequently, it is not necessary to rely on EUV seroconversion as the primary outcome of evaluation studies of HIV prevention programs because those specific behaviors can be targeted for assessment. In contrast to studies that rely on seroconversion as the primary outcome, studies that rely on behavioral outcomes can be conducted more quickly and less expensively because they need not calTy out He extensive screening required in seroconversion studies. (This is not to say, however, that behavioral studies are inexpensive.) Because both primary and complementary risk behaviors occur at rates that far exceed the rate of seroconversions, it is possible to make reliable estimates of specific behaviors using smaller samples thereby making the research less costly than for seroconversion studies. In addition, a reliance on behavioral outcomes rather than serocon- version circumvents the problem of the reluctance of some people to be tested. REV antibody testing is not routine among He general popula- tion, although the picture is mixed among high-nsk Individuals. There is evidence from a number of cohort studies of He willingness of gay men to be tested, although what proportion of people in those communities is wining to be tested is unknown. But there is evidence that many indi- viduals in drug treatment programs or STD clinics are not willing to be tested (see, e.g., Hull et al., 1988; Fleming et al., 1989~. Thus, if AIDS prevention programs and their companion evaluation studies can avoid the need to conduct HIV testing, they may be likely to reach a wider and more representative portion of their target populations.

OCR for page 34
44 ~ EVALUATING AIDS PREVENTION PROGRAMS For those evaluation protocols that must include HIV testing, the use of behavioral outcomes is nonetheless important for adequate eval- uation. For an individual who is REV positive, the primary behaviors of interest involve behaviors that impose risk on others; e.g., allowing another person to use his or her unsterilized IV drug injection equipment, becoming pregnant. For those who are HIV negative, the primary behav- iors of interest involve risk taking; e.g., using unsterilized Hug injection equipment. The main weaknesses of behavioral data involve problems of validity and reliability (see Appendix C). Problems also arise in interpreting data about similar outcomes when wordings of questions differ among studies. Moreover, even the interpretation of the same question may be problematic over time because respondents' willingness to report behaviors may change. Another problem In research based on self-reports involves respon- dents' ability to remember and accurately report specific events and behaviors (there is a fairly large literature on this topic). The problem of recall often may be decreased by shortening the time frame over which respondents must recall the occurrence and frequency of behav- iors. However, a shorter time frame may change He meaning of a particular outcome: celibacy or monogamy for a week, for example, is quite different from celibacy or monogamy for a year. Another major issue surrounding behavioral outcomes involves which outcomes to emphasize In prevention programs and which to focus on in evaluation studies. While risk reduction and risk elimination are most desirable outcomes Tom the standpoint of halting the spread of HIV, protective behaviors may be more practical to teach and maintain. Psychological Outcomes There are a handful of psychological outcomes that are also of interest from the standpoint of AIDS prevention. These include an awareness of AIDS and HIV and of the gravity of the problems they pose; knowledge of AIDS and HIV transmission; and stigmatization of individuals who have AIDS or who are infected with HIV. These psychological outcomes do not have a direct bearing on HIV infection in the way that intercourse, drug injection, and pregnancy (for seropositive women) do. Never~e- less, differences in the degree to which Individuals are aware of AIDS, understand which behaviors transmit HIV, and disparage or devalue those who are id or infected may be important detenn~nants of whether they adopt risk reduction or protective behaviors.

OCR for page 34
MEASUREMENT OF OUTCOMES ~ 45 Of all the possible goals for the educational campaigns that have been mounted In this country, the greatest progress has been made in the area of increased knowledge and awareness of AIDS (Turner, Miller, and Barker, 19881. This must be seen as an important accomplishment because behavioral change is unlikely to occur without knowledge and awareness of the AIDS problem and the means of HIV transmission. How much of the credit for this increased knowledge belongs to Public Health Service efforts, as compared with efforts ong~nating from other educational institutions, personal experience, news stories, and so form, can probably never be accurately assessed. From a public health stand- po~nt, it is not crucial to make such a determination. Instead, one of the most important public health goals now should be to maintain the current high level of knowledge and awareness that exists In the population about the risk of REV transmission by sexual contact and the sharing of infected equipment and dispelling misconceptions about the risk of transmission through casual contact. Strengths and Weaknesses of Psychological Outcomes Psychological outcomes that Involve awareness of AIDS and HIV, knowI- edge about AIDS and EUV transmission, and attitudes toward those who are infected and iB may be the easiest outcomes to measure and study, In the sense that these measures are easily incorporated In a survey ques- tionna~re. Nonetheless, the development of specific questions must be carefully done. Questions involving knowledge and attitudes must be sensitively crafted as well as clearly worded In the vernacular of the target population to avoid introducing substantial response bias and error. Similarly, Here is a need to test the sensitivity of the measurements to al- ternative question wordings or interview formats (see Turner and Martin, 1984: Volume I, Ch. 4-5~. EVALUATION MEASURES There are several methodological issues involved In developing reliable indicators of changes in knowledge, attitudes, and risky behaviors. Two issues in particular challenge He measurement of program effectiveness: (~) detellIiining the appropriate time intervals between program imple- mentation and the measurement of change and (2) assessing the quality of survey instruments (questionnaires) and the resulting data. Some of the issues addressed in the sections below are generic to program evaluation; others are more specifically related to the measurement of unobserved phenomena, such as sexual behavior or attitudes.

OCR for page 34
46 EVALUATING AIDS PREVENTION PROGRAMS Timing of Measurement People who adopt new behaviors in the face of the epidemic may dis- continue the change, relapse from time to time, or, conversely, enhance the change over time. Measurements taken soon after the intervention has been delivered may show substantial change but that change may decay over time. Because immediate post~ntervention measurements and delayed measurements may yield different results, periodic measurement is desirable. One other measurement timing issue warrants mention here. The national media campaign (see Chapter 3) changed its goal and even its target audience before measuring any outcomes. Thus, four separate phases of the campaign were launched without measuring what occurred during previous phases. Such haste is dysfunctional. It does not permit evidence to be gathered on what it is about the intervention that works or does not work or which elements should be continued or discontinued in subsequent phases. The panel cautions that over CDC programs, such as the forthcoming program of direct grants to community-based organizations (CBOs) (see Chapter 4) arid other projects with multiple objectives are vulnerable to shifting program goals. The consequences of interventions should be measured and assessed before deciding on new or alternative stages of intervention programs, a policy that will require periodic measurement of program outcomes. In addition, even if a well-planned, well-unplemented project has been successful In meeting its specified goal or goals, periodic evaluation efforts should not cease. For example, media messages may lose their impact with repeated use, the audience may change as the population ages, Individuals may feel secure and become careless In their behaviors, or the program itself may drift In its purposes. The pane' recommends that once goals are met, projects be reevaluated periodically to monitor their continued eiTective- ness. Quality of Measures There are a number of methodological topics that are related to the qual- ity of survey data: sampling hard-to-reach groups, appropriate designs for intervention research and evaluation, the validity of self-reported in- formation, and the construction of reliable questionnaire items. Because Of the importance of data quality when trying to determine whedler pro- gram objectives have been reached, Appendix C includes a chapter from the recent report of the parent committee (Miller, Turner, and Moses,

OCR for page 34
MEASUREMENT OF OUTCOMES ~ 47 1990) reviewing methodological research on the accuracy of surveys measurements of sexual and drug using behaviors. There are two general methodological issues that should be con- sidered in an assessment of evaluation measures: validity and reliability. Valid measures reflect without bias the presence or intensity of the concept that Hey intend to quantify. (Bias refers to systematic misrepresentation or other systematic error which may be due to faulty measurement, sam- pling, or other factors.) Reliable measures will yield the same data if applied repeatedly. In considering the difficulties associated with AIDS evaluation measures, the pane] believes that measurement validity will pose Me most serious and difficult problem. With regard to reliability, the pane} notes a small but important set of studies that have begun to exaniine the reliability of sexual behavior data, collected mainly from gay men, and a limited number of small studies that have examined the reli- ability of measures of behavioral change made In response to the AIDS epidemic: see, for example, Saltzman and colleagues (19871; Catania and colleagues (19901; Martin and colleagues (19891; and Martin and Dean (1989~. Differential validity, that is, nonequivalent measurement biases in treatment and control groups, is a particularly worrisome concern for outcome evaluations. It is understandable that some individuals would be reluctant to report risky sexual or drug use behaviors. In an outcome evaluation experiment that randomly assigns participants to different treatments, this reporting bias can be expected to be equivalent across groups at the beginning of the experiment. However, the social pressures created by an intervention program can differentially affect the report- ~ng bias. Individuals enrolled in Intervention programs should fee! an increase in psychological and social pressure to refrain from engaging in risk-associated behavior. Consequently, they may also feel pressure to conceal conduct that is at odds with program objectives. In such a cir- cumstance, differential reporting biases may cause the group that receives the intervention to appear to have adopted safer behavioral patterns than He control group, when in fact the observed effect is due to the effect of the intervention on the reporting bias. The quality of measures will also depend on the level of detail of information elicited by the survey instrument. For example, to evaluate He effectiveness of an intervention program, it is not be sufficient to know whether respondents have ever used condoms or whether they have begun to use them. It is also important to determine the frequency of use, and it will be helpful to ascertain the conditions that foster use. This last point underscores the need to collect data that are not only

OCR for page 34
48 | EVALUATING AIDS PREVENTION PROGRAMS valid and reliable but that are also meaningful. Outcome measures should include comprehensive and up-to-date observations of the attainment of a program's explicitly stated objectives as well as anticipated side effects, if any. Careful measurement in the context of good research design and implementation will then allow thoughtful inferences to be made about what the results of evaluation mean. REFERENCES Catania, J. A., Gibson, D. R., Marin, B., Coates, T. J., and Greenblatt, R. M. (1990) Response bias In assessing sexual behaviors relevant to HIV transmission. Evaluation and Program Planning 13: 19-29. Centers for Disease Control (CDC) (1988) Announcement No. 901. September 20. Federal Register 53 (182~:36492-36493. Centers for Disease Control (CDC) (1989) A Comprehensive Program to Prevent HIV Transmission. Fiscal Year 1989 Operating Plan. Washington, D.C.: U.S. Department of Health and Human Services. Coates, T. J., Morin, S.F., and McKusick, L. (1987) Behavioral consequences of AIDS antibody testing among gay men. Journal of the American Medical Association 258:1889. Fleming, D., Bennett, D., Klockner, R., Gould, J., Cassidy, D., and Foster, L. (1989) HIV Infected STD Clients Who Decline HIV Counseling and Testing. Paper presented at the Fifth International AIDS Conference. Montreal, June 4-9. Grant, D., and Anns, M. (1988) Counseling AIDS antibody-positive clients: Reactions and treatment. American Psychologist 43:72-74. Hull, H. F., Bettinger, C. J., Gallaher, M. M., Keller, N. M., Wilson, J., and Mertz, G. J. (1988) Companson of HIV-antibody prevalence in patients consenting to and declining HIV-antibody testing in an STD clinic. Journal of the American Medical Association 260:935-938. Longini, I. M., Jr., Clark, W. S., Horsburgh, C. R., Lemp, G. F., Byers, R. H., Darrow, W. W., and others (1989) Statistical analysis of the stages of HIV infection using a Markov model. Paper presented at the Fifth International AIDS Conference. Montreal, June 4-9. Martin, J. L., and Dean, L. (1989) Risk factors for AIDS-related bereavement in a cohort of homosexual men in New York City. In B. Cooper and T. Helgason, eds., Epidemiology and the Prevention of Mental Disorders. United Kingdom: Routledge. Martin, J. L., Dean, L., Garcia, M., and Hall, W. (1989) The impact of AIDS on a gay community: Changes in sexual behavior, substance use, and mental health. American Journal of Community Psychology. 17~3~:269-293 Miller, H. G., Turner, C. F., and Moses, L. E., eds. (1990) AIDS: The Second Decade. Report of the NBC Committee on AIDS Research and the Behavioral, Social and Statistical Sciences. Washington, D.C.: National Academy Press. Ogilvy & Mather (1987) Contract for REP No. 200-87-0525. Section m (Methodology and Approach). June. Ostrow, D. G., Joseph, J., Soucey, J., Eller, M., Kessler, R., Phair, J., and Chmiel, J. (1988) Mental health and behavioral correlates of HIV antibody testing in a cohort of gay men. Paper presented at the Fourth International AIDS Conference. Stockholm, June 12-16.

OCR for page 34
MEASUREMENT OF OUTCOMES ~ 49 Saltzman, S. P., Stoddard, A. M., McCusher, J., Moon, M. W., and Mayer, K. H. (1987) Reliability of self-reported sexual behavior risk factors for HIV infection in homosexual men. Public Health Reports 102:692-697. Strobino, D. M. (1987) The health and medical consequences of adolescent sexuality and pregnancy: A review of the literature. In S. Hofferth and C. Hayes, eds., Risking the Future: Adolescent Sexuality, Pregnancy, and Childbearing. Volume 2, Working Papers and Statistical Appendixes. Report of the NRC Panel on Adolescent Pregnancy and Childbearing. Washington D.C.: National Academy Press. Suchman, E. A. (1967) Evaluative Research. New York: Russell Sage Foundation. Turner, C. F., and Martin E., eds. (1984) Surveying Subjective Phenomena, 2 vols. New York: Russell Sage. Turner, C. F., Miller, H. G., and Barker, L. (1988) AIDS research and the behavioral and social sciences. In R. Kulstad, ea., AIDS, 1988. Washington, D.C.: American Association for the Advancement of Science. Turner, C. F., Miller, H. G., and Moses, L. E., eds. (1989) AIDS, Sexual Behavior, and Intravenous Drug Use. Report of the NRC Committee on AIDS Research and He Behavioral, Social, and Statistical Sciences. Washington, D.C.: National Academy Press. Weiss, C. H. (1972) Evaluation Research. Englewood Cliffs, N.J.: Prentice-Hall, Inc.