EVALUATION METHODS



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs EVALUATION METHODS

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs This page in the original is blank.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs ASSESSING THE EFFICACY OF NEEDLE EXCHANGE PROGRAMS: AN EPIDEMIOLOGICAL PERSPECTIVE NOREEN V. HARRIS and JAMES P. MCGOUGH, King County Department of Public Health, Seattle, Washington; and NOEL S. WEISS, Department of Epidemiology, University of Washington INTRODUCTION In order to evaluate the efficacy of needle exchange programs, we must examine what they attempt to accomplish, and how they go about accomplishing it. Needle exchange programs have as their goal reducing the transmission of HIV (and other pathogens) among injection drug users. Such programs typically attempt to achieve their aims through three different kinds of interventions: first, the provision of materials such as sterile needles, bleach for syringe cleaning, and condoms; second, by providing education and information; and third, by referring clients to other interventions, such as drug treatment programs, social services, and medical care. The success of all of these interventions may be achieved through behavior change. Freer access to sterile needles, bleach, and condoms may offer participants the materials necessary to adopt new, less-risky injection and sexual practices, and education may provide them with the motivation and skills necessary to make those changes. Drug treatment and access to social and medical services may decrease a drug-user's need to inject drugs, and so the number of times needles are shared. It is also possible, however, that needle exchanges may be successful at reducing the transmission of HIV independently of changes in sharing patterns, if they reduce the likelihood that the needles that are shared are contaminated with HIV. In the absence of behavior change, such a reduction would take place if exchange programs either replaced a supply of contaminated needles with sterile ones or reduced "the number of people each needle shared" by shortening the circulation time of the average needle before it is replaced by a sterile one (Kaplan 1993). If a needle exchange program succeeded only in supplanting an existing supply of sterile needles, then there would be no reduction in the likelihood of their contamination with HIV. CAUSAL INFERENCE The evaluation of needle exchange programs involves a quest for evidence that needle exchanges caused a particular outcome of interest. The development of causal hypotheses relies on an inductive approach which is commonly guided by a set of criteria (Hill 1965): Presence of an association, especially one that is consistent across studies.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Presence of a strong association (i.e., a large relative risk). Weak associations may be causal, but are more likely than strong ones to be wholly the result of non-causal factors. A temporal relationship in which the putative cause precedes the effect. A gradient in the strength of the association that accords with predictions based on our biological, social, or psychological understandings of the issue (Weiss 1981). The plausibility of a causal relationship on other, non-epidemiological, grounds. After the development of causal hypotheses, a deductive approach is followed in which predictions arising from those hypotheses are subjected to empirical tests. While we cannot "prove" causal hypotheses, we can (in principle) refute them. Hypotheses not falsified by the data at hand are "confirmed" in that they remain reasonably good explanations until they are falsified by new data, and are replaced by other hypotheses that better explain the observations (Popper 1965; Rothman 1986). OUTCOMES OF INTEREST Outcomes Related to Reducing Risk of HIV Transmission In order to determine if needle exchange programs succeed in their aim of reducing the parenteral and sexual risk of HIV transmission among injection drug users, one could measure the prevalence or incidence of HIV, the behavioral predictors of HIV transmission, surrogate measures of those behaviors, or the number of different persons sharing individual needles. HIV There are two means of determining whether or not an individual has acquired HIV or any other parenterally or sexually transmitted infection during a specific time period. One could establish whether or not seroconversion occurred between baseline and follow-up testing or, when only one test is done, the presence of short-lived antibodies or antigens can in principle serve as markers of recent infection (Hart et al. 1989; van den Hoek et al. 1989; Nelson et al. 1991; Kaplan et al. 1991; van Ameijden et al. 1992).

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Behavior Some of the parenteral behaviors associated with HIV infection which could be measured include: "sharing" of injection equipment (the word here meaning injecting with needles after their use by others); sharing of injection equipment without effective cleaning; sharing with high-risk individuals; and the frequency of injection with uncleaned shared equipment (Donoghoe et al. 1989; van Ameijden et al. 1992; Guydish 1993). Some of the sexual behaviors related to HIV infection which could be measured include: sex with multiple partners; sex without condoms; and sex with high-risk partners (Donoghoe et al. 1989; Stimson et al. 1989; Hart et al. 1989). Surrogate Measures of HIV Serological or other clinical evidence of recent acquisition of parenterally-transmitted or sexually-transmitted diseases other than HIV may serve to indicate that individuals were involved in high-risk activities that could also transmit HIV (van Haastrecht et al. 1991; Brettle 1991). Number of Persons Sharing Each Syringe Even in the absence of significant change in the propensity to share needles, a needle exchange might reduce the number of people who share the average needle. The fewer people sharing a needle, the smaller the chances of that needle becoming infected with HIV, and so the lower the risks of disease transmission. One could, in principle, measure the number of sharers directly, measure the average time that needles spend in circulation in a community (Donoghoe et al. 1989; Kaplan 1993), or measure syringe barrel wear as a surrogate of syringe use (Smith et al. 1981). Outcomes Related to Injection Drug Use It is possible that needle exchange programs promote drug use by "condoning" it, by making drug injection easier, or by fostering the initiation of new drug injectors. On the other hand, needle exchange programs might lead to a reduction in drug use by facilitating entry to drug treatment, or by supporting users in stopping or reducing injection. The mere presence of needle exchange programs might convince injectors of the seriousness of the HIV epidemic, and of the need to reduce or stop drug use to avoid AIDS. Possible measures of drug use include initiation or cessation of injection, changes in the frequency of injection, and a shift to non-injection modes of drug use (U.S. General Accounting Office 1993).

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs POTENTIAL RELATIONSHIPS OF EXPOSURE TO OUTCOME The impact of needle exchange might be related to the duration, recency, or directness of an individual's exposure to the exchange. Choice of a particular model or models of the possible relationship between exposure and outcome will have an important influence on the selection of appropriate measures of exposures and outcomes. Duration Can Be Modelled in a Number of Different Ways The One-Hit Model Patrons of the exchange might achieve benefit after a single exposure. This model reflects what might happen if the very existence of the exchange alerted clients to the risk of AIDS, and convinced them to stop sharing needles or to stop using drugs. To assess such an impact, it would be necessary to classify participants in terms of past exchange use (ever versus never) and to measure the relevant outcomes after exposure to needle exchange (or in the cases of non-exchangers, during a comparable time period). The Threshold Model The effect of needle exchange might manifest itself only after a sufficient minimum number of repeated exposures. For example, it may be the case that outreach workers staffing the exchange table develop a rapport with clients, and earn their trust, only after a certain number of exchange encounters. It also might well be the case that the success of various risk reduction educational and other interventions would be dependent on the development of that rapport and trust, and so require a certain number of needle exchange visits. The Dose-Response Model An effect of needle exchange might be related to the total amount of exposure; the effect might increase with increasing frequency, duration, or intensity of exposure, with no threshold level of exposure.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Recency of Exchange Might Also Be Modelled in Different Ways Latency of Effects In some cases the effects might be immediate, while in others a specified latent period might be required before benefit or harm accrues (Schlesselman 1982). If effects were immediate, a study that compared repeat exchangers to non-exchangers would not be able to detect any differences between them if the benefit from the exchange manifested itself right after the first visit. Transience of Effects The effects of needle exchange might be either permanent or transient. A change to lower-risk behavior might, for example, not persist as the novelty of the needle exchange intervention wanes, leading to "relapses" to higher-risk activity. Directness of Effects It is possible, particularly in communities with large-volume needle exchanges, that indirect effects might be of importance. Non-exchangers might derive "herd immunity" from needles and education provided them by those who are exchangers. In such circumstances, observational studies of individuals may not be able to tease out the effects of needle exchange without collecting data on sources of needles and information among both needle exchangers and non-exchangers. In this case it may also be more difficult to demonstrate an association between needle exchange and HIV risk. Direct and indirect effects might be in operation at the same time, of course, and they may have different impacts on different outcomes. As far as we know, all studies of individuals have looked only at direct effects. STUDY DESIGN There have been numerous attempts to glean information on the effect of needle exchange programs from surveys that do not include comparable, concurrently sampled, control groups (Donoghoe et al. 1989; Kaplan et al. 1991; Klee et al. 1991; Nelson et al. 1991; Guydish et al. 1993). Unfortunately, it is not possible on the basis of such surveys to assess whether or not the presence of a needle exchange is associated with the occurrence of HIV or the behaviors associated with HIV, let alone whether any presumed associations are causal in nature. Observations provided by these surveys can be suggestive of causation, but not conclusive.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Controlled epidemiological studies of the efficacy of exchange programs may consist of studies of populations or of individuals, and each type of study can be conduced as either an observational study or an experimental trial. No study design (epidemiological or otherwise) is without its strengths and potential shortcomings. Observational Studies of Populations (''Ecologic Studies") The unit of study in this design is a population or community of injection drug users. For example, the occurrence of HIV in communities with needle exchange programs might be compared with that in communities without needle exchanges. Such studies are relatively inexpensive, and often can be completed quickly when they rely on extant data. Differences observed, however, may not be due to the exposure of interest, but rather to differences in the populations or communities being compared. Because the number of communities under study is usually small, there is limited ability to examine other reasons for any differences in rates that are present in the different populations. For example, Ljungberg et al. (1991) reported an HIV prevalence of 1% among injection drug users in Skäne, a southern province of Sweden which is served by needle exchange programs in the towns of Lund and Mälmo. By contrast, HIV prevalence among a injection drug users sampled in Copenhagen (a 45 minute ferry boat ride away from Lund) was 15%, among amphetamine injectors in Stockholm it was 4-6%, and among heroin injectors in Stockholm it was 45-60%. Neither Copenhagen or Stockholm are served by needle exchange programs. As the investigators acknowledged, there could well be differences other than the presence or absence of needle exchange programs which could account for the observed variation in HIV prevalence. Copenhagen is a large and cosmopolitan city, while Lund is a relatively small rural university town. Within Sweden, Stockholm has a concentration of heroin users (Käll 1992), many among whom are HIV-positive immigrants from high incidence areas abroad (Kerstin Käll, Karolinska Institute, personal communication 1993). The role of needle exchange in causing a reduction in HIV occurrence would be supported by large differences in HIV associated with needle exchange; demonstrated comparability between the exchange and non-exchange populations with respect to other predictors of HIV; and samples of communities large enough to enable statistical stability and the assessment of potential confounding. For example, it would be difficult to interpret a comparison of HIV prevalence among injection drug users in Salt Lake City (which does not have a needle exchange program) with a population of injectors in New York City (which does have a needle exchange) because the two communities differ with respect to the timing of the epidemic and many demographic, sociologic, and cultural factors which may be predictors of HIV occurrence. However, comparisons of injection drug users among comparable New York City and New Jersey communities with and without exchanges, could possibly provide more valid evidence for a possible relationship of needle exchange programs to HIV transmission.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Experimental Trials of Populations (Community Intervention Trials) Community intervention trials involve choosing a large number of communities, and then randomly or systematically assigning needle exchange programs to some number of the communities. No such trial has been attempted. If the number of communities were sufficiently large, random assignment of communities to needle exchange could achieve comparability with respect to the predictors of HIV. However, if a small number of communities are selected, the randomization process may not achieve such comparability. With an intervention as controversial as needle exchange, random assignment of communities may not be possible, and if it were, "contamination" of the comparison communities might be hard to prevent. Observational Studies of Individuals In these studies, the unit of study is the individual, and the sample size is the number of individuals who participated. There are three types of observational studies of individuals; cross-sectional, case-control, and cohort studies. Cross-Sectional Studies Many studies of needle exchange are essentially cross-sectional in nature. In such studies, the exposure (in this case needle exchange use) and the outcomes (e.g., HIV infection) are measured at the same point in time. Cross-sectional studies are usually less expensive than other observational studies, and can often be completed in relatively short time periods. The chief limitation of cross-sectional studies of needle exchange is the inability to determine which came first; the exposure (to needle exchange) or the outcome (for instance, HIV infection or behavior change). For example, in prevalence surveys among injection drug users entering drug treatment programs, we found that HIV prevalence was higher among those who patronized the needle exchange than among those who did not (Harris et al., n.d.). From these data alone, it is not clear whether knowledge of HIV positivity led people to use the exchange, or exchange use led to acquisition of HIV infection. Case-Control Studies In a case-control study, the investigator compares a group with a particular outcome (for example, newly acquired HIV infections) with a comparable group of controls without the outcome (those free of HIV infection), to determine whether or not the groups differ with respect to exposure to needle exchange in the past (prior to acquiring HIV infection, or, in the case of controls, prior to a concurrent time period). Case-

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs control studies, like cross-sectional studies, are less expensive and more quickly completed than are cohort studies. However, if the study requires asking subjects to recall past events, their inaccuracy in doing so may lead to a biased result. Also, in a case-control study it care must be taken to establish that the outcome truly followed exposure to needle exchange, and did not precede it. For example, we conducted a case-control analysis in which we interviewed 2,500 drug injectors in Seattle, systematically recruited in a variety of settings. We defined as cases those who in the 12 months before interview either increased or maintained injecting with non-sterile needles, and we defined as controls those who either decreased or continued not to do so. We then compared the proportion of cases and of controls who had used a needle exchange prior to the 12 month period for which outcomes were measured, to clearly establish a temporal sequence between exposure to the exchange and injection behavior outcomes. Cohort Studies In a cohort study, the investigator classifies individuals with respect to exposure and follows them over time to assess subsequent outcomes. Cohort studies generally afford a greater opportunity to evaluate the temporal sequence between needle exchange use and the outcomes of interest than do other observational designs. Aside from the expense and the length of time it takes to complete them, a major consideration in conducting cohort studies, particularly of cohort studies of injection drug users, is the importance of maximizing follow-up. A cohort study was conducted by Hartgers et al. (1989) in which they interviewed 72 injection drug users in Amsterdam who usually obtained their needles from an exchange, and 73 injection drug users who never or only irregularly used the exchange. A second interview was administered one year later. If those who are successfully followed in a cohort study differ from those not followed, study findings may be distorted because observed differences in outcome between the exposed and the non-exposed might be due to factors connected with follow-up instead of, or in addition to, differences in the exposures of interest. The authors of the Amsterdam cohort study note that only 49% of exchangers, and even fewer non-exchangers (34%), returned for the follow-up interview, and that those successfully followed differed in important ways from those not followed. The differences between those followed and those not followed could conceivably account for the differences in outcome that they observed between exchangers and non-exchangers. Experimental Trials of Individuals In a randomized trial of the impact of a needle-exchange program, participants would be randomly assigned to either needle exchange or to no intervention (or to a different intervention). Subjects would be followed over time, just as in a cohort study,

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs to determine subsequent outcomes. The advantage to such studies is that they avoid the distortion in results that might occur if those who self-select to attend a needle exchange program differ from those who don't with respect to risk behaviors. In fact, however, it may not be ethically, politically, or logistically feasible to assign some individuals to needle exchange and prevent others from attending, and as far as we know, no experimental trials have been carried out. ISSUES OF METHODOLOGICAL BIAS In epidemiologic studies, bias is present when the observed association between exposure and outcome does not accurately portray the (theoretical) true association. Although the lines between them are occasionally hard to draw, it is useful to consider three general categories of bias: confounding; selection; and information bias. Confounding Confounding is the distortion of the true association (or lack thereof) between an exposure and an outcome that is caused by the presence of another exposure or characteristic which also leads to the outcome of interest. In order for a factor to confound, it must be associated both with the outcome and with the exposure. Sexual behavior (particularly male homosexual behavior) is an example of a potentially confounding factor which should be considered in studies of the impact of needle exchange on the risk of HIV. If men who have sex with men are either more or less likely than others to patronize needle exchange programs (that is, if male homosexuality is associated with the exposure), studies which fail to deal with this fact in design, sampling, or analysis could be confounded and so biased in their results. Nonetheless, few studies of needle exchange have controlled for such behavior. An example of a study which did is that of Nelson et al. (1991), in which the investigators considered in the analysis the potential confounding effects of receptive anal intercourse in their study of the relationship of diabetes (who have access to sterile needles) to HIV prevalence among injection drug users. There are a number of ways to deal with the problem of confounding. In design, one can match or stratify on potential confounders. In sampling, one can recruit index and comparison groups from the same underlying target populations in a representative (probabilistic) fashion. In analysis, one can conduct stratified, logistic regression, or other multivariate analyses which incorporate information on potential confounding factors and thus, permit assessment of, and adjustment for, bias resulting from confounding. Such

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Heimer R, Myers S, Cadman E et al. (1992a). Detection by polymerase chain reaction of human immunodeficiency virus Type I proviral DNA sequences in needles of injecting drug users. Journal of Infectious Diseases 165:781782. Heimer R, Kaplan EH and Cadman C (1992b). Prevalence of HIV-infected syringes during a syringe exchange program. New England Journal of Medicine 327:1883-1884. Heimer R, Kaplan EH, Khoshnood K et al. (1993). Needle exchange decreases the prevalence of HIV-1 proviral DNA in returned syringes in New Haven, Connecticut. American Journal of Medicine 95:214-220. Joseph SC and Des Jarlais DC (1989). Needle and syringe exchange as a method of AIDS epidemic control. AIDS Updates 2:1-8. Kaplan EH (1989). What are the risks of risky sex? Modeling the AIDS epidemic. Operations Research 37:198-209. Kaplan EH (1990). Modeling HIV infectivity: Must sex acts be counted? Journal of Acquired Immune Deficiency Syndromes 3:55-61. Kaplan EH (1994). A method for evaluating needle exchange programs. Statistics in Medicine (in press). Kaplan EH and Heimer R (1992). HIV prevalence among intravenous drug users: Model-based estimates from New Haven's legal needle exchange. Journal of Acquired Immune Deficiency Syndromes 5:163-169. Kaplan EH and Heimer R (1993). What happened to HIV transmission among drug injectors in New Haven? Chance 6(2):9-14. Kaplan EH and Heimer R (1994). HIV incidence among needle exchange participants: Estimates from syringe tracking and testing data. Journal of Acquired Immune Deficiency Syndromes 7:182-189. Kaplan EH and O'Keefe E. (1993). Let the needles do the talking! Evaluating the New Haven needle exchange. Interfaces 23(1):7-26. Kaplan EH and Soloshatz D (1993). How many drug injectors are there in New Haven? Answers from AIDS data. Mathematical and Computer Modelling 17:109-115. Larson RC and Kaplan EH (1981). Decision-oriented approaches to program evaluation. New Directions for Program Evaluation 10:49-68. Larson RC and Odoni AR (1981). Urban Operations Research. Prentice-Hall, Englewood Cliffs, New Jersey. Ljungberg B, Christensson B, Tunving K et al. (1991). HIV prevention among injecting drug users: Three years of experience from a syringe exchange program in Sweden. Journal of Acquired Immune Deficiency Syndromes 4:890-895.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Mausner JS and Kramer S (1985). Epidemiology: An introductory text. W.B. Saunders Company, Philadelphia, Pennsylvania. Muñoz A, Wang M-C, Bass S et al. (1989). Acquired immunodeficiency syndrome (AIDS)-free time after human immunodeficiency virus type 1 (HIV-1) seroconversion in homosexual men. American Journal of Epidemiology 130:530539. Myers SS, Heimer R, Liu D and Henrard D (1993). HIV DNA and antibodies in syringes from injecting drug users: A comparison of detection techniques. AIDS 7:925-931. Nahmias S (1989). Production and operations analysis. Richard D. Irwin Incorporated, Boston, Massachusetts. Nelson KE, Vlahov D, Muñoz A et al. (1992). Incident HIV-1 infections in a cohort of intravenous drug users (IVDUs). Presented at the VIII International Conference on AIDS, Amsterdam, The Netherlands (Abstract PoC 4698). Newmeyer JA (1988). The prevalence of drug use in San Francisco in 1987. Journal of Psychoactive Drugs 20:185-189. Page ES (1955). A test for a change in a parameter occurring at an unknown point. Biometrika 42:523-527. Schoenbaum EE, Hartel D, Selwyn PA et al. (1989). Risk factors for human immunodeficiency virus infection in intravenous drug users. New England Journal of Medicine 321:874-879. Shiboski SC and Jewell NP (1992). Statistical analysis of the time dependence of HIV infectivity based on partner study data. Journal of the American Statistical Association 87:360-372. Stimson G (1989). Syringe exchange programmes for injecting drug users. AIDS 3:253-260. Valleroy L, Weinstein B, Groseclose S et al. (1993). Evaluating the impact of a new needle/syringe law: Surveillance of needle/syringe sales at Connecticut pharmacies. Presented at the IXth International Conference on AIDS, Berlin, Germany (Abstract PO-C24-3189). Watters J, Cheng Y-T and the Prevention Point Research Group (1991). Syringe exchange (SE) in San Francisco: Preliminary findings. Presented at the VII International Conference on AIDS, Florence, Italy (Abstract ThC 99). Weinstein B and Hadler JL (1993). Impact of new legislation on needle and syringe purchase and possession-Connecticut, 1992. Morbidity and Mortality Weekly Report 42:145-148. Wiley JA, Herschkorn SJ and Padian NS (1989). Heterogeneity in the probability of HIV transmission per sexual contact: The case of male-to-female transmission in penile-vaginal intercourse. Statistics in Medicine 8:93-102.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Willemain TR and Larson RC (1977). Emergency medical systems analysis: Papers on the planning and evaluation of services. Lexington Books, D.C. Heath and Company, Lexington, Massachusetts. Wodak A, Dolan K, Imrie A et al. (1987). Antibodies to human immunodeficiency virus in needles and syringes used by intravenous drug abusers. Medical Journal of Australia 147:275-276.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Figure 1 Client Participation and Visitation

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Figure 2 Volume of Needle Exchange

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Figure 3 Mean Needle Circulation Time

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Figure 4 Needle Exchange and Removal Rates

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs Figure 5 HIV Infection in Needles over Time

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs DISCUSSION: EVALUATION METHODS ROBERT BOOTH Robert Booth observed that, on one hand, quantitative evaluation methods are especially appealing because of the typically large sample sizes they employ, their use of statistical inference, and their generalizability. On the other hand, they are subject to biases and problems of recall and social desirability. Such methods are also subject to threats to internal validity, particularly one-shot case studies and studies using cross-sectional designs, in which it is difficult to parcel out the influence of historical trends on observed differences. Some statistical procedures have been developed to attend to these difficulties, but the use of experimental designs that strengthen the grounds for inference is preferable. He commented that qualitative methods are also appealing, in that the yield from qualitative methods has a richness and detail often lacking in quantitative approaches. In addition, qualitative methods can uncover information that survey research often misses. However, qualitative research is often threatened by issues of sampling bias and sample size. Booth described mathematical modeling as a particularly strong methodology, especially as operationalized by Edward Kaplan. Kaplan's approach is intuitively appealing; it yields a conservative estimate of effectiveness; its limitations, Booth noted, are its exclusive focus on needles and their exchange and the attribution of HIV reduction solely to needle exchange. Other interventions and other factors may also play a role in how injectors obtain and use needles. In addition, although exposure time can be calculated in the model, because the focus is on needles and not on people, one does not know how many people use a needle; one knows only the length of time the needle was in circulation. Looking at the context of program evaluation, particularly quantitative methods, which have dominated the literature to date, Booth stated that the evidence clearly suggests that HIV risk behaviors, particularly needle risk behaviors, have declined over time. The decline is possibly due to the interventions that have been implemented, the media, word of mouth, or other factors. Evidence of the decline, however, has been found in cross-sectional research designs, one-shot case studies, pre-post-quasi-experimental designs, true experimental designs, time series designs, and other designs. He said that evidence also suggests that risk has not been totally eliminated; addicts sometimes find themselves in situations in which it is difficult to follow risk reduction protocols. These findings are supported in evaluations of drug treatment programs, community-based outreach interventions, educational office-based interventions, pre-and post-HIV testing and counseling, and needle exchanges. Indeed, risk reduction in the absence of intervention has also been reported. This is not to say, he went on, that any and all interventions work or that behaviors will change without interventions, although they may. Nor does it mean that needle exchanges are no better or worse than other interventions. Indeed, data have

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs shown that participants in needle exchange programs differ demographically from participants in drug treatment and other interventions and that participants in drug treatment differ from those involved in street-based outreach programs. As was said a number of times during the workshop, different programs attract different types of people and people at different stages of their addiction, and all the intervention programs may be necessary. What are lacking in the evaluations discussed over the past two days, other than in a few studies, are rigorous tests of program effectiveness, Booth concluded. Nearly 10 years ago, Mark Lipsey wrote an article entitled ''Evaluation: The State of the Art and the Sorry State of the Science" (Lipsey, Crosse, Dunkle, et al., 1985). He noted that most studies conducted under the label of program evaluation embody at least the rudiments of experimental methods. The telltale signs of quantitative measurement are there, some attempts have usually been made at control comparison or use of baseline groups, and cause-and-effect thinking is evident, even when there is little resemblance to classical randomized experiments. However, he said, program evaluation is often poorly done within the experimental paradigms, and he cited two reasons for this. First, there are numerous practical difficulties in matching good research designs to practical program circumstances. Second, social scientists, for the most part, are not very well trained to do methodologically exacting research under field conditions. To this Booth added a third reason: outcome evaluation is more exciting than process evaluation. Hence, in his view, program evaluation has not only neglected theory, but attention to the treatments or interventions provided—that is, the intervention dose—has been neglected. The result is that treatments or interventions have been represented as black boxes. Moreover, the vast majority of studies of risk reduction programs that involve some sort of experimental design represent the treatment or intervention level as a dichotomy. That is, they assume that each member of the treatment or intervention group receives the same treatment and each member of the control group receives nothing. When outcomes show success, they are due to the treatment, and when outcomes show failure, they are due to failure of the treatment. However, the lack of outcome success may be due to the failure of the research, not of the treatment or intervention. Needle exchange programs are particularly difficult to evaluate, he observed, in part because the standards for good research are more stringent than for other interventions, notably community-based outreach interventions, and in part because the research itself may be a deterrent to program participation. Consequently, Booth recommended the use of multiple methods to evaluate needle exchange programs, including quantitative and qualitative studies, mathematical modeling, and use of what have been referred to as focal-local indicators, such as those used in the Tacoma and Portland studies reported on earlier, in which the researchers looked at hepatitis-B prevalence and incidence over time. In addition to hepatitis data, other indicators would include HIV and AIDS cases, drug-related crimes and arrests, and sexually transmitted diseases. Such investigations would require an extended period of time. In addition, he asked for recognition that any single indicator, as well as any single method, has its limitations.

OCR for page 153
Proceedings Workshop on Needle Exchange and Bleach Distribution Programs He summarized by saying that most effectiveness studies have not taken into account the influence of other interventions or temporal trends in subject populations. Participants in needle exchanges or community outreach programs are considered as existing in a vacuum. Sterile needles can be obtained from sources other than needle exchanges, and behavior modifications can occur, in the absence or in the presence of needle exchange, unrelated to needle exchange. Noting that a number of presenters at the workshop have argued for looking at multiple interventions rather than attribute change to single interventions, he concluded that there is a need to focus on single interventions while measuring the landscape of other possible influential factors. Booth ended by mentioning the important issue of retention, which has been stressed repeatedly in evaluations of drug treatment programs. In the context of needle exchange programs, the issue becomes: What are the criteria for success regarding retention in a program? REFERENCE Lipsey, M. W., S. Crosse, J. Dunkle, J. Pollard, and G. Stobart 1985 Evaluation: The state of the art and the sorry state of the science. New Directions for Testing and Measurement 27:7-28.