Click for next page ( 317


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 316
5 Evaluating the Effects of AIDS Interventions Previous chapters of this report have dealt with understanding the behaviors that transmit HIV, monitoring the spread of infection, and designing and implementing intervention programs to oppose the further spread of the disease. The committee has called for the implementation of planned variations of programs to determine how best to facilitate change in those behaviors associated with risk. Making those determinations requires sound evaluations of the different program variations. Yet evaluation is rarely part of a program's activities. In its review of existing intervention programs, the committee was distressed to find a dearth of associated evaluation activity. Committee members were also disappointed to see a lack of data on behavioral variables for those evaluations that had been conducted. The committee believes that the time has come to make a commit- ment to the rational design of intervention strategies and to careful evaluation of the effectiveness of those strategies through controlled experiments that use carefully defined populations. Knowlecige must be gained from current intervention programs to improve future ef- forts. Evaluation is the process that will enable us to learn from experience. The committee recommencis that the Office of the Assistant Secretary for Health take responsibility for an evaluation strategy that will provide timely information on the relative effectiveness of different AIDS intervention programs. The political realities of evaluation point to both positive and negative aspects of the process. On the one hand, good evaluations 316

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 317 can generate support for effective programs. Well-publicizecT findings of evaluation activities can legitimately defend successful programs that may be viewed as politically sensitive or controversial, while gestures that were merely symbolic can be shown to be ineffective. On the other hand, evaluation efforts are likely to show that programs are less effective than might be hoped. Perfect studies and absolute, permanent change in behavior are standards that are rarely, if ever, met. Every effort should be made to ensure that evidence of imperfect improvement is not used to overturn programs that may be viewed as politically undesirable. There is every reason to believe that the nation can and will do better in determining which interventions change behavior and which do not. Discussions with many people who have been on the front lines of AIDS prevention activities since the early days of the epidemic reveal their clesire for evaluation of their work. To ciate, however, most of these individuals have not conducted evaluationsnot be- cause they were unwilling but because they lacked the capability. The links between those who provide services ant! manage programs, on the one hand, and those who conduct research and evaluation, on the other hand, have not been strong in the past. This chapter discusses basic approaches to and problems in- herent in conducting controlled experiments and evaluations. The committee recognizes that it will not meet the needs of all readers: for some, it will be too basic. For those who are not yet familiar with the techniques of experimental (resign and evaluation, how- ever, we hope it will be a useful introduction. Yet it must be noted that any document such as this cannot take the place of individual, program-specific consultation. It is therefore imperative to establish supportive, productive linkages among program and evaluation pro- fessionals so that future prevention efforts can result in sound, useful information. DIMENSIONS OF EVALUATION It is not always easy to learn from experience, but it is certainly possible. To increase the likelihood of such learning requires the advance planning of evaluations as well as the precise, controlled execution of programs. Good evaluation does not just happen; it must be planned for and arranged. Evaluation is a systematic process that produces a trustworthy account of what was attempted and why; through the examination of results the outcomes of intervention programs it answers the questions, "What was ~lone?" "To whom,

OCR for page 316
318 ~ LIMITING THE SPREAD OF HIV and how?" and "What outcomes were observed?" Well-clesigned evaluation permits us to ciraw inferences from the data and addresses the difficult question: "What do the outcomes mean?" Well-executed evaluations provide credible information about program effectiveness. Such information is critical to developing rational policies, allocating limited resources, and serving needs in a targeted, productive, ant! economical fashion. Ensuring complete, high-quality evaluation requires advance specification of the program. Thus, in preparing for an evaluation, the design of a program can sometimes be improved by increasing its specificity and establishing standards of performance at the outset. At its best, a process in which program innovations are informed by feedback from careful, prompt evaluations can lead to the timely elimination of ineffective concepts and designs and the selection and adoption of effective ones. A successful evaluation of an intervention program must provide answers to several key questions: 1. What were the objectives of the intervention? 2. How was the intervention designed to be conducted? 3. How was the intervention actually conducted? Who participated? Were there any unexpected problems? What parts of the program were easier to conduct than was anticipated? What parts were harder? 4. What outcomes were observed, and how were they mea- sured? 5. What were the results of the intervention? These questions are presented in a logical order of progression; they are also ranked according to the ease with which they can be answered those at the end are harder to answer than those at the beginning. It is not uncommon to find reports of programs that use the term evaluation to refer to activities that would answer only the first three questions. The committee does not follow that usage. Evaluation refers to the whole set of questions, with particular emphasis on the last two. It is important to distinguish between the outcomes of the pro- gram and the results of the intervention: an outcome (denotes what occurred; an evaluation seeks to determine whether the outcome resulted from the intervention or from some other external factor.

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 319 Program Objectives An intervention program may start with a protocola document that includes a description of the intervention, the activities to be undertaken, the groups targeted to receive the intervention, and the roles and responsibilities of the individuals or groups undertaking the tasks. (Ffequently, funding agencies require such a document from organizations or individuals who are applying for a grant or seeking a contract to support the program.) This protocol is an integral part of an intervention: it should spell out unambiguously the objectives of the program and how they will be measured, as well as the operational content of the work to be performed. The program objectives are the clesired outcomes. They do not, for example, specify an intent to spend a certain sum on an activity or to deliver a certain number of advertisements or pamphlets; rather, they relate to the reasons that motivate the program. Objectives state the outcomes the program seeks to achieve. Program Design Two key elements in program design are defining the measures of the outcome or effect of the program and selecting the treatment unit. A variety of outcome measures can be chosen for study. It may help to conceptualize them along two dimensions: (1) their relevance as indicators of program achievement and (2) the feasibility of actually measuring them. For example, both the behavior and knowledge of subjects may be affected by a program; yet the former, although harder to measure, may be more relevant to the program's objectives. Many AIDS prevention programs have chosen knowledge as the outcome measure; it is easier to gauge than behavior but less relevant to the process of preventing the spread of HIV infection. It is not unusual for programs to include multiple outcome measures that vary in importance. The treatment unit refers to the body to which the intervention will be applied. Variations in this construct and in outcome mea- sures are illustrated in the following four examples of intervention programs. . Two different pamphlets on the same subject are pre- pared. They are sent to inclividuals calling an AIDS hot line and are distribute`] in an alternating sequence. The outcome to be measured is whether recipients return a card asking for more information.

OCR for page 316
320 ~ LIMITING THE SPREAD OF HIV . Two cTiscussion/instruction curricula about AIDS and HTV infection are prepared for use in high school health education classes. The outcome to be measured is the score on a test of knowledge. A subset of all STD clinics in a large standard metropoli- tan statistical area is ranclomly chosen to introduce a change in fee schedules. The outcome to be measured is the change in patient load. . A community-level prevention program establishes a co- ordinated set of interventions involving community lead- ership, social service agencies, the media, community associations, and other groups. The two outcomes to be measured are knowledge (as assessed by testing) ancT condom sales in the community's ret ail outlets. These fictitious interventions are appliec3 to very different treat- ment units. In the first example, the treatment unit is an individual person who receives either pamphlet A or pamphlet B. If either "treatment" were to be applied again, it woul(1 be applied to a per- son. In the second example, the high school health education class is the treatment unit; everyone in a specific class is given either cur- ricuTum A or curriculum B. If either treatment were to be applied again, it would be applied to a class. The treatment unit in the third example is the clinic. In the fourth example, the treatment unit is the whole community. To make inferences from the results of an evaluation, the treat- ment units that are analyzed must correspond to those that are sampled. For example, when organizations or groups are randomly chosen to receive a given intervention, the sample size is the number of organizations or groups. Because in~livi~luals within each broader unit do not yield independent observations, they cannot be used as statistical units. The consistency of the outcome measures or effects of a par- ticular intervention during repetitions of it is critical in appraising the intervention. It is important to remember that repetitions of a treatment or intervention are counted as the number of treatment units to which the intervention is applied. Planning the intervention program and planning its evaluation should go hand in hand. The evaluation plan is an appropriate part of the protocol, which should state how the measurement and analysis of intendecl outcomes will be conducted. The comparisons to be used to assess the program's results are also part of an evaluation plan.

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 321 Program Implementation Whatever results are judged to have occurred because of an inter- vention, they must relate to the program as it was actually carried . out rather than to the program as it was planned or intended. Thus, it is necessary to determine what services were in fact delivered, to whom, and how. Evaluations that address the actual execution of the program are usually called process evaluations or implementation analyses that is, if the evaluator's role is passive. When the evaluator takes a more active role, these efforts are referred to as trouble shooting or forma- tive evaluations. Process evaluation has three important purposes: (1) to verify that planned services are actually offered and received, (2) to determine how the quality or extent of a service varied, and (3) to develop ideas about how to improve the organization or delivery , ~ ~ o. : services. The best of such evaluations recognize that no policy is ever fully Implemented and no service ever delivered exactly as planned. Eval- uation that discovers, for instance, that a fraction of the condoms received by individuals remained unused, that many free needles were stolen or not received by the target group, or that "counseling" in some cases was based on inaccurate and naive information is im- portant. There are many good examples of formative evaluations. For example, in a study of whether a special community-based pro- gram for the severely and persistently mentally ill worked better than conventional approaches, Brekke and Test (1987) undertook an ap- proach that monitored a sample of clients regularly over three years to determine where treatment took place, the amount and nature of treatment, who received it, and how continuity of care was achieved. Similarly, in a multisite study of which of three regimens worked best to ameliorate the problems of patients afflicted with mental de- pression, Waskow (1984) and Elkin and colleagues (in press) ensured that therapists used the treatment variation that they had agreed to use and determined how they did so. Their work included the de- velopment of manuals that stipulated guidelines, the preparation of training regimens for therapists, and studies of interactions between clients and therapists; their evaluation data also included more con- ventional measures of the number, frequency, and length of therapy sessions. Process evaluations can be designed in different ways to focus more squarely on the treatment target. Recent work conducted by the Center for AIDS Prevention Studies in San Fiancisco investigated three stages in the prevention process: (1) ensuring that individuals

OCR for page 316
322 ~ LIMITING THE SPREAD OF HIV who engage in high-risk behavior know they are (loin" so; (2) under- standing how, why, and when commitments are made to reduce such behavior; and (3) understanding the factors that influence the way people seek out and act on alternative approaches to risk behavior reduction (Catania et al., 1988~. The best of such process evaluations also direct attention to the families and partners of individuals at high risk in an attempt to understand the frequency and character of support from those sources and how that support may discourage or foster high-risk behavior. Well-executed process evaluations also make plain the stan- dards or criteria against which an organization's performance will be judged. Standardized criteria for assessing programs have been established in some substantive areas by different groups and orga- nizations. The American Public Health Association has developed criteria for health education programs that have been used in the design of a university-based AIDS program that seeks to reduce risk-associated behavior (VaTcliserri et al., 1987~. Defining ant} Measuring Outcomes As noted earlier, there are two dimensions to outcome measures: relevance and feasibility. Often, there are many possible outcome measures inherent in the design of an intervention, and their relevance is obvious. For example, consider an instructional program aimed at persuading people to use condoms. Each of the following questions corresponds to at least one outcome measure: Did the subjects attend the program? Did they pay attention? Did they understand the message correctly? Did they believe and accept it? Did they thereafter use condoms? Did they benefit from that behavior? The questions become progressively harder to answer; in addi- tion, the answer to each succeeding question is more important than the one before it. The choice of an outcome measure or measures is largely a matter of balancing importance against feasibility. In the case of AIDS, the issue of time must also be considered. The extent and character of current condom use are important pieces of information, but the need to use condoms goes beyond the present. Moreover, the lengthy incubation period of HIV may mean that an- swers to the later questions on the list may only be acquired some

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 323 years in the future, when the effect of condom use on HTV prevalence can be seen. Obstacles to the feasibility of outcome measurement, the second of the two dimensions, span a wide range of difficulty. An outcome may simply be impossible to measure. For example, the incidence of HIV infection in the general population is not known because such knowledge wouIc! require repeated serologic testing of a very large probability sample of the population, which is currently impossible. For an outcome like sexual behavior, which is effectively impossible to observe, one can use surrogates or verbal reports about the behav- ior. Other obstacles to measurement are illustrated in the following examples: the failure of a respondent to understand what is asked- in some STD clinics, there are concerns that clients may not understand the term vaginal intercourse; . nonresponse, perhaps by withholding cooperation Hull and colleagues (1988) reported that the 18 per- cent of their sample of patients from an STD clinic who refused antibody testing contained more seropositives than the 82 percent who were tested; . the sheer difficulty of collating information from many sources for each subjectthis has been a problem in measuring the costs associated with health care utiliza- tion; . fearful and inaccurate responses from people who are worried about their vulnerable status or from those who are concerned about legal strictures that may threaten them; . perplexity about how to measure complex concepts, such as perceived self-efficacy; and . cultural and linguistic barriers that may lead to nonco- operation and misunderstanding on the part of respon- dents, investigator, or both. The extent and severity of such obstacles will necessarily in- fluence which outcome measures are chosen for study. Although they may not be widely known, there are many instruments to mea- sure such constructs as the quality of life, depression, perceived selefficacy, and satisfaction with care, as well as instruments for measuring knowledge and comprehension (Mitchell, 1985~. Once the outcome measure or measures have been chosen (per- haps following a pilot test, a procedure the committee endorses),

OCR for page 316
324 ~ LIMITING THE SPREAD OF HIV there remains the task of actually doing the measuring. Often, a difficult measurement problem can be solved by using a surrogate. For example, the incidence of STDs can serve as a prompt, sensitive indicator of behavioral change to prevent sexually acquired HIV in- fection because STDs and HIV can be spread by the same behaviors. If a particular outcome is simply too difficult to measure accurately, a major component of that outcome may serve as a reasonable al- ternative. Thus, ascertaining the total costs of medical care for each of many AIDS patients can be daunting, bud accurate figures for the number of days spent in the hospital, in the intensive care unit, and in nursing facilities may be readily available. Although these estimates ignore unit costs and omit the cost of such items as drugs and respiratory therapy, they may be better outcome measures than the conceptually complete but inaccurately measured total cost. Finding a good surrogate for outcomes that are difficult to mea- sure and devising ways to cross-check those measures call for ingenu- ity and imagination. Inferring Results: The Value of Controller! Experiments Good evaluations can provide accurate descriptions of the interven- tion process and measure the outcomes. Yet there remains the prob- lem of inferring the effects of the intervention. Describing the process and stating the results are not sufficient. A good evaluation should say something about the relationship between the intervention and the outcome. A patient who receives a worthless treatment for the common cold is still likely to get better within a week because that is what usually happens with a cold, with or without the benefit of treatment. To infer that a given intervention has produced a partic- ular effect involves comparing what did happen with the intervention to what would have happened without it. Because it is not possible to make this comparison directly, inference strategies rely on proxies for what would have happened. Such proxies include the patient's past history and comparison groups of various sorts. In some circumstances, extrapolating a trend from a patient's pretreatment history as a proxy for what would have happened is the best that can be done. Yet extrapolation is always problematic: past records may be incomplete, and it is impossible to control for all intervening factors. This type of approach should probably be used as a near-to-last resort. Uncler certain conditions, comparison groups can be quite use- fuT for inferring results, although defining and recruiting a suitably

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 325 similar control group can be difficult. For example, after selecting ethnically similar control and treatment groups, an investigator may find that one group is, on average, older, sicker, or more educated. There are evaluation strategies that attempt to adjust for the dif- ferences between the two groups, but making those adjustments is seldom easy. Three types of information or knowledge are required: (1) knowledge of intervening variables that also affect the outcome of the intervention and that consequently need adjustment to make the groups comparable; (2) measurements on all intervening variables for all subjects; and (3) knowledge of how to make the adjustments properly, which in turn requires an understanding of the functional relationship between the intervening variables and the outcome vari- ables. Satisfying each of these information requirements is likely to be more difficult than attaining the primary goal of the activity, which, simply stated, answers the question, "Does this intervention produce beneficial effects?" With differently constituted groups, inferences about results are hostage to uncertainty about the extent to which the observed out- come actually results from the intervention and is not an artifact of intergroup differences. Fortunately, there is a remedy: establish one, singly constituted group in which to assess treatment effects. To be included in the group, inclividuals must satisfy the inclusion ant! exclusion criteria for the study. A subset of this group is then randomly chosen to receive the intervention, thus forming two com- parable groups. They are not identical, but because they are two random samples drawn from the same population, they are as simi- lar as is possible. Moreover, they are not systematically different in any respect, which is important for all variables those known and those as yet unidentified that can influence the outcome. Dividing a singly constituted group into random and therefore comparable subgroups cuts through the tangle of causation and establishes a basis for the valid comparison of treated and untreated subjects. After establishing two or more comparable subgroups, a good evaluation must ensure that outcome measurement is performed symmetrically for all subjects. For example, if treated subjects are examined and tested at hospital A and untreated subjects are exam- ined and tested at hospital B. it is impossible to determine whether observed differences are due to treatment or are merely artifacts of noncomparable outcome measurement. The foregoing ideas are cen- tral to randomized clinical trials and randomized field experiments, which are discussed in the next section. Although highly desirable,

OCR for page 316
326 ~ LIMITING THE SPREAD OF HIV partitioning strategies of this type are not practical for every ques- tion. The nonparticipation of some individuals or high attrition rates among participants may cause an investigator to use methods that are less satisfactory for inferring results. This, unfortunately, has been the case with randomized trials of some drug treatment strate- gies: for example, many {V drug users prefer methadone maintenance over detoxification, making it difficult to recruit subjects for random assignment treatment studies. Constraints on Evaluation The above sections have described the basic characteristics of eval- uation. Three additional considerations are discussed here. First, the size of the study, or the number of treatment units, is a function of several factors. Budget constraints may influence the size of the study, in all likelihood, by setting the upper limits. Moreover, time limits may affect the capacity to coordinate a study ant! thus set limits on the number of units. important statistical issues. The major one is determining how large a sample must be to reliably detect the impact of an intervention strategy with a states! (albeit hypothetical) degree of effectiveness. Analyses of statistical power help to avoid study designs that are not sufficiently sensitive to detect an intervention's effects. Analyses of power can dictate an increase in the size of the study to achieve the necessary sensitivity, or, occasionally, suggest a reduction in size, thus saving time and other resources. Finally, in planning programs and evaluations, consideration should be given to pilot tests. The committee strongly believes that every intervention program and every evaluation program should be tested in advance, on a small scale and in a realistic way, to identify problems before more substantial resources are expen(le(l. It is possible to avoid using funds or other resources on programs or evaluations that, with a small pilot test, can easily be seen to be infeasible. A large number of AIDS intervention programs are currently being implemented. Goof! evaluations of these programs may be difficult to perform; they will almost certainly be expensive. To improve the likelihood that high-quaTity evidence of program effectiveness will be obtained, it would be justified to focus what are necessarily finite resources on the best-designecl, best-implemente<1 intervention programs. In addition to pragmatic considerations, there are

OCR for page 316
346 ~ LIMITING THE SPREAD OF HIV would be surprising if memory-related causes of distortion were any less influential in the recalling of drug-use or sexual activity. Finally, self-reportec3 data may be subject to nonsystematic variation that can be regarded as random. Interviewers, for instance, may vary subtly in the way they elicit information from drug dealers; the dealers in turn may vary in the accuracy of their reports on the sale of clean neecIles. The research literature on the accuracy of self-reported data on sensitive behavior is sparse because such research is extremely difficult. It is especially important to understand the relationship between self-reports and actual behavior, but it is rare to see re- search projects that can correlate self-reported data with direct, in- dependent observation or with observations that are arguably more accurate than retrospective self-reports. This pattern argues for methodological research that: . correlates self-reported data with direct observations of IV drug use and the shared use of needles; . correlates retrospective self-reports of sexual behavior among prostitutes, {V drug users, and others at risk for HIV infection with monitoring (e.g., more frequent interviews or the use of diaries) that is more proximate to the time of condom use or the use of other protections against infection; and . investigates the cognitive processes that individuals use to answer sensitive questions (e.g., memory flaws and distortion that is not deliberate). Although to date such research has not been extensive, the or- ganizational mechanisms to support it are in place. The National Laboratory for Collaborative Research in Cognition and Survey Mea- surement at the National Center for Health Statistics, ant! the grant mechanisms of the National Center for Health Services Research and the National Institute of Mental Health, are important resources for such activities, which are likely to take considerable time and con- sume other resources. Yet the problems engendered by inadequate knowledge of the quality of self-reported behavior will persist un- less the matter is given serious attention. For example, self-reportecl data collected in surveys on drug use are generally thought to be reliable; that is, the reports remain constant for an individual from one time to the next. However, less is known about the validity of these ciata the extent to which self-reports reflect actual behavior. Clearly, memory decay can occur, but the impact of the passage of

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 347 time on reporter! data depends on the salience of the events and the manner in which questions are asked, among other factors. The litany of difficulties presented in this chapter is not intended to discourage evaluation activity. On the contrary: it is hoped that, by discussing the various methodological, legal, and ethical hurdles, each can, in some degree, be dealt with and made more tractable as experience is acquired. Indeed, each difficulty stands to benefit from the systematic approach and cumulation of knowledge that typify good programs of evaluation. IMPI`EMENTING GOOD EVAI,UATIONS Evaluation tennis to evoke sentiments on the part of those being evaluated that are similar to the responses inspired by a visit from an auditor or by interaction with a customs inspector. The feel- ings and fears are not surprising; nevertheless, they have impeded productive collaborative arrangements and severely diminished the returns that are otherwise attainable from good evaluation. Some of the impediments arise from program practitioners' lack of access to individuals with expertise in the area of evaluation. The com- mittee recommencis that evaluation support be provided to ensure collaboration between practitioners and evaluation researchers. The challenge to leadership and management is to remove im- pediments to evaluation. An effective strategy should, inasmuch as possible, remove all basis for the fear and trepidation that has ex- isted in the past. In essence, this means that no person or unit should be punished as a result of an evaluation. In making this strong statement, the committee recognizes that occasionally justi- fie(1 punishment will be withheld. Yet that may be a small cost to pay for fostering program evaluation, which is a valuable part of an organization's process of self-criticism. Oversights and problems are inevitable. The occurrence of some mistakes, some errors, some imperfections is not generally cause for punitive action; properly viewed, they present opportunities. If something goes wrong, evaluation may allow an un(lerstanding of how and why the problem occurred. This understanding in turn can enable a change or adjustment that may forestall similar error in the future. An organization that adopts this ethos and is recognized by staff to have done so, can learn much more effectively from its own experience.

OCR for page 316
348 ~ LIMITING THE SPREAD OF HIV People can learn from both successes and failures. Unfortu- nately, however, (lescriptions of failed interventions are less likely to be published than reports of successful ones, and they accrue little prestige for those who conduct them. Yet the publication of nega- tive research results can forestall further unfruitful efforts. Because available resources are limited, it is important to try to repeat suc- cessfuT interventions and not to repeat the clear failures. In reviewing AIDS intervention programs, the committee found that the descries tive information typically published about an actual intervention does not provi(le sufficient de-tai! to permit its replication. There- fore, the committee recommencis to the research community that the results of well-conductec} evaluations be published, regardless of the intervention's effectiveness. The commit- tee further recommencis that all evaluations publish cletailed descriptive information on the nature and methods of in- tervention programs, along with evaluation data to support claims of relative effectiveness. The resources required to perform evaluations include money, personnel with relevant expertise, and the time and attention of managers. All are essential components of the process, and some are not always readily available. :[t is evident that choices of emphasis and allocation must be male. Indeed, not every program should receive a full-blown evaluation, although every intervention should probably receive at least some minimal assessment, if only to know what was done and what actually occurred over the course of the program. Setting priorities for the use of evaluation resources ap- pears to rest on several factors: the importance of the intervention, the extent of existing knowledge concerning it, the perceived value of additional information, and the estimated feasibility of the assess- ment. In order to use available evaluation resources most efficiently, the committee recommencis that only the best-clesigned and best-implementec! intervention programs be selectee! to re- ceive those special resources that will be needed} to conduct scientific evaluations. In this chapter and those that precede it, we urge that interven- tions to prevent the spread of HIV be conducted in accordance with two principles: (1) plannecI variants of new interventions should be systematically used and should replace the "one-best-shot" approach; and (2) evaluations of new initiatives should be planned in advance and carefully executed. The committee believes that following these two principles will result in more effective programs of education and

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 349 behavioral change in a shorter time frame. This belief rests on the following propositions: 1. More information is cleveloped and conserved when the evaluation of planner! variants is carried out. 2. Good program ideas are more promptly recognize<] and accepted. 3. Less effective ideas are more promptly recognized ant! eliminated. 4. Agreement on the relative merits of alternatives is eas- ier to reach and can be effected with more confidence when there are systematically acquired data concerning plausible alternatives. When possible, for at least each major type of intervention and each major target population, a minimum of two intervention pro- grams should be subjected to rigorous evaluations that are designed to produce research evidence of the highest possible quality. Vari- ants of intervention programs should be developed for and tested in different populations ant] in different geographic areas using random assignment strategy accompanied by careful evIauation. When ethi- cally possible, one of the variants should be a nontreatment control. The committee recognizes that difficulties will attend the effort to adopt this strategy, difficulties that include not only the challenges of unfamiliarity and the extra work required to prepare several vari- ants of a brochure, curriculum, radio message, or other intervention tool, but also the problems of actually performing evaluations. All such endeavors call for skills and additional resources that may be in short supply for the agencies that are aIreacly heavily committed to coping with AIDS. Despite the difficulties, we believe the achievable benefits are too important to pass by. The first steps to implement the ideas discussed above shouIc3 be taken promptly. The com- mittee recommencis that CDC substantially increase efforts, with links to extramural scientific resources, to assist health departments and others in mounting evaluations. State and local health departments as well as education departments will likely require additional resources as they mount evaluation efforts. The committee sees two steps as sufficient to initiate this process. First, CDC land any other agency that undertakes AIDS prevention programs) should assign to some administrative unit the responsibil- ity for ensuring the use of planned variants of intervention programs and for overseeing a system of evaluating such programs. Second, there should be easy access to extramural resources to help with the

OCR for page 316
350 ~ LIMITING THE SPREAD OF HIV task of evaluation. These resources might be consultants, commer- cial research organizations, committees of outside experts, or some combination of these individuals and bodies. BIBLIOGRAPHY The entries below inclucle references cited in the chapter and other publications of interest. Citations with an asterisk are material rele- vant to guidelines and standards of evidence for evaluation. Ackerman, A. M., Froman, D., and Becker, D. (1987) The multiple risk factor intervention trial: Implications for nurses. Progress in Cardiovascular Nursing 2:92-99. Alexander, J. F., and Parsons, B. V. (1973) Short-term behavioral intervention with delinquent families: Impact on family practices and recidivism. Journal of Abnormal Psychology 81:219-225. American Psychological Association, Committee for the Protection of Human Partic- ipants in Research (1982~. Ethical Principles in the Conduct of Research with Human Participants. Washington, D.C.: American Psychological Association. Barcikowski, R. S. (1981) Statistical power with group mean as the unit of analysis. Journal of Educational Statistics 6:267-285. Barnes, B. A. (1977) Discarded operations: Surgical innovation in trial and error. In J. P. Bunker, B. A. Barnes, and F. Mosteller, eds., Costs, Risks, and Benefits of Surgery. New York: Oxford University Press. Becker, M. H. (1985) Patient adherence to prescribed therapies. Medical Care 25:539- 555. Berk, R. A. (1986) Anticipating the Social Consequences of a Catastrophic AIDS Epidemic. Department of Sociology, University of California at Santa Barbara. Berk, R. A., and Rauma, D. (1983) Capitalizing on nonrandom assignment to treatments: A regression discontinuity evaluation of a crime control program. Journal of the American Statistical Association 78:21-27. Berk, R. A., et al. (1985) Social policy experimentation: A position paper. Evaluation Review 9:387-429. *Bernstein, I. N., and Freeman, H. E. (1975) Academic and Entrepreneurial Research. New York: Russell Sage Foundation. Betsey, C. L., Hollister, R. G., and Papageorgiou, M. R., eds. (1985) Youth Employ- ment and Training Programs: The YEDPA Years. Washington, D.C.: National Academy Press. Blythe, B. J., Gilchrist, L. D., and Schinke, S. (1981) Pregnancy prevention groups for adolescents. Social Work 26:503-504. Boeckmann, M. (1981) Rethinking the results of a negative income tax experiment. In R. F. Boruch, P. M. Wortman, and D. S. Dordray, eds., Reanalyzing Program Evaluations. San Francisco: Jossey-Bass. Boruch, R. F., and Cecil, J. S. (1979) Assuring Confidentiality of Social Research Data. Philadelphia: University of Pennsylvania Press. Boruch, R. F., and Dennis, M. (1986) Understanding respondent cooperation: Field experiments versus surveys. Pp. 296-318 in Proceedings: Second AnnualResearch Conference. Washington, D.C.: U.S. Department of Commerce. Boruch, R. F., Dennis, M., and Carter-Greer, K. (1988) Lessons from the Rocke- feller Foundation's Experiments on the Minority Female Single Parent Program. Evaluation Review 12:396-426.

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 351 Boruch, R. F., McSweeny, A. J., and Soderstrom, J. (1978) Bibliography: Illustrative randomized experiments for program planning, development, and evaluation. Evaluation Quarterly 4:655-696. *Boruch, R. F., and Pearson, R. (1988) Comparative evaluation of longitudinal surveys. Evaluation Review 12:3-58. Boruch, R. F., Reiss, A., Larntz, K., Address, A., and Friedman, L. (1988) Report of the Program Review Team: Spouse Assault Replication Project. National Institute of Justice, Washington, D.C. Breger, M. J. (1983) Randomized social experiments and the law. Pp. 97-144 in R. F. Boruch and J. S. Cecil, eds., Solutions to Ethical and Legal Problems in Social Research. New York: Academic Press. Brekke, J. S., and Test, M. A. (1987) An empirical analysis of services delivered in a model community treatment program. Psychosocial Rehabilitation Journal 10:51-61. Brownell, K. D., Marlatt, G. A., Lichtenstein, E., and Wilson, G. T. (1986) Under- standing and preventing relapse. American Psychologist 41:762-782. Bunker, J. P., Barnes, B. A., and Mosteller, F. (1977) Costs, Risks, and Benefits of Surgery. New York: Oxford University Press. Campbell, D. T., and Stanley, J. S. (1966) Experimental and Quasi-Experimental Designs for Research. Chicago: Rand-McNally. Catania, J. A., Kegeles, S. M., and Coates, T. J. (1988) Towards an Understanding of Risk Behavior: The CAPS' AIDS Risk Reduction Model (ARRM). Center for AIDS Prevention Studies, Department of Psychiatry and Department of Medicine, University of California at San Francisco. January. *Chalmers, T. C. (1981) A method for assessing quality of a randomized control trial. Controlled Clinical Deals 2:31-49. Chelimsky, E. (1988) Educating People at Risk of AIDS. Testimony before the U.S. Congress, Committee on Governmental Affairs. U.S. General Accounting Office, Program Evaluation and Methodology Division, Washington, D.C. June 8. Clausen, J. A., Seidentefeld, M. A., and Deasy, L. C. (1954) Parent attitudes toward participation of their children in polio vaccine trials. American Journal of Public Health 44:1526-1536. Coates, T. J., McKusick, L., Kuno, R., and Sites, D. P. (1987) Stress Management Training Reduces Number of Sexual Partners but Does Not Enhance Immune Function in Men Infected with Human Immunodeficiency Virus (HIV). University of California at San Francisco. Condiotte, M. M., and Lichtenstein, E. (1981) Self-e~cacy and relapse in smoking cessation programs. Journal of Consulting and Clinical Psychology 49:648-658. Conner, R. F. (1982) Random assignment of clients in social experimentation. In J. E. Sieber, ea., The Ethics of Social Research: Surveys and Experiments. New York: Springer-Verlag. Cook, T., and Campbell, D. T. (1979) Quasi-Experimentation. Boston: Houghton- Mifflin. *Cordray, D. S. (1982) An assessment of the utility of the ERS standards. New Directions for Program Evaluation 15:67-82. Darrow, W. W. (1988) Behavioral research and AIDS prevention. Science 239:1477. *Davis, H. R., Windle, C., and Sharfstein, S. S. (1977) Developing guidelines for program evaluation capability in community mental health centers. Evaluation 4:25-34. Deniston, O. L., and Rosenstock, I. M. (1972) The Validity of Designs for Evaluating Health Services. School of Public Health, University of Michigan.

OCR for page 316
352 ~ LIMITING THE SPREAD OF HIV Dennis, M. (1988) Implementing Randomized Field Experiments: An Analysis of Civil and Criminal Justice Research. Ph.D. dissertation, Department of Psychology, Northwestern University. Des Jarlais, D. C. (1987) Effectiveness of AIDS Educational Programs for Intravenous Drug Users. Background paper prepared for the Health Program, Office of Technology Assessment, U.S. Congress, Washington, D.C. Dondero, T. J., Pappaioanou, M., and Curran, J. W. (1988) Monitoring the levels and trends of HIV infection: The Public Health Service's HIV surveillance program. Public Health Reports 103:213-220. Elkin, I., Pilkonis, P. A., Docherty, J. P., and Sotsky, S. (In press) Conceptual and methodological issues in comparative studies of psychotherapy and pharmacother- apy. American Journal of Psychiatry. Farquhar, J. W., Maccoby, N., and Wood, P. D. (1985) Education and community studies. Chapter 12 in W. W. Holland, R. Detels, and G. Knox, eds., Oxford Textbook of Public Health. London: Oxford University Press. Farquhar, J. W., Wood, P. D., and Breitose, I. T. (1977) Community education for cardiovascular health. Lancet 1:1191-1195. Federal Judicial Center. (1981) Social Experimentation and the Law. Washington, D.C.: Federal Judicial Center. Ferber, R., Sheatsley, P., Turner, A., and Waksberg, J. (1980) What Is a Survey? Washington, D.C.: American Statistical Association. Fienberg, S. E., Martin, M. E., and Straf, M. L., eds. (1985) Sharing Research Data. Washington, D.C.: National Academy Press. Flay, B. R. (1986) Efficacy and effectiveness trials and other phases of research in the development of health promotion programs. Preventive Medicine 15:451-474. Fraker, T., and Maynard, R. (1985) The Use of Comparison Group Designs in Evaluations of Employment Related Programs. Princeton, N.J.: Mathematics Policy Research. Fraker, T., and Maynard, R. (1987) Evaluating comparison group designs with employment related programs. Journal of Human Resources 22:195-227. Freedman, D., Pisani, R., and Purves, R. (1978) Statistics. New York: W. W. Norton. Freedman, R., Takeshita, J. Y., et al. (1969) Family Planning in Taiwan: An Experiment in Social Change. Princeton, N.J.: Princeton University Press. Freeman, H. E., and Rossi, P. H. (1981) Social experiments. Milbank Memorial Fund Quarterly Health and Society 59:340-374. Friedman, L. M., Furberg, C. D., and DeMets, D. L. (1981) Fundamentals of Clinical l~als. Boston: John Wright. Friedman, L. M., Furberg, C. D., and DeMets, D. L. (1985) Fundamentals of Clinical Trials, 2d ed. Littleton, Mass.: PSG Publishing, Inc. Friedman, S. R., De Jong, W. M., and Des Jarlais, D. C. (1988) Problems and dynamics of organizing intravenous drug users for AIDS prevention. Health Education Research: Theory and Practice 3:49-58. Glaser, E. M., Coffey, H. S., et al. (1967) Utilization of Applicable Research and Demonstration Results. Los Angeles: Human Interaction Research Institute. *Gordon, G., and Morse, E. V. (1975) Evaluation research. In A. Inkeles, ea., Annual Review of Sociology, vol. 10. Palo Alto, Calif.: Annual Reviews, Inc. Gray, J. N., and Melton, G. B. (1985) The law and ethics of psychosocial research on AIDS. University of Nebraska Law Review 64:637-688. Harkin, A. M., and Hurley, M. (1988) National survey on public knowledge of AIDS in Ireland. Health Education Research 3:25-29. Hausman, J. A., and Wise, D. A. (1979) Attrition bias in experimental and panel data: The Gary (Indiana) income maintenancy experiment. Econometrica 47:455-473.

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 353 Haynes, R. B., Taylor, D. W., Snow, J. C., Sackett, D. L., et al. (1979) Appendix 1: Annotated and indexed bibliography on compliance with therapeutic and preventive regimens. In R. B. Haynes, D. W. Taylor, and D. L. Sackett, eds., Compliance in Health Care. Baltimore, Md.: Johns Hopkins University Press. Hill, R., Stycos, J. M., and Back, K. W. (1959) The Family and Population Control: A Puerto Rican Experiment in Social Change. Chapel Hill: University of North Carolina Press. Holland, P. W., and Rubin, D. B. (1986) Research designs and causal inferences: On Lord's paradox. In R. Pearson and R. Boruch, eds., Survey Research Designs. Lecture Notes in Statistics No. 38. New York: Springer-Verlag. Hull, H. F., Bettinger, C. J., Galiaher, M. M., Keller, N. M., Wilson, J., and Mertz, G. J. (1988) Comparison of HIV-antibody prevalence in patients consenting to and declining HIV-antibody testing in an STD clinic. Journal of the American Medical Assocation 260:935-938. Institute of Medicine. (1985) Assessing Medical Technologies. Washington, D.C.: National Academy Press. Jaffe, H. W., Rice, D. T., Voight, R., Fowler, J., and St. John, R. (1979) Selective mass treatment in a venereal disease control program. American Journal of Public Health 69:1181-1182. Janz, N. K., and Becker, M. H. (1984) The health belief model: A decade later. Health Education Quarterly 11:1-47. Job, R. F. (1988) Effective and ineffective use of fear in health promotion campaigns. American Journal of Public Health 78:163-167. *Joint Committee on Standards. (1981) Standards for Evaluations of Programs, Projects and Materials. New York: McGraw-Hill. Kadane, J. (1986) Progress toward a more ethical method for clinical trials. Journal of Medicine and Philosophy 11:285-404. Klein, N. C., Alexander, J. F., and Parsons, B. V. (1977) Impact of family systems intervention on recidivism and sibling delinquency. Journal of Consulting and Clinical Psychology 45:469-474. LaLonde, R. J. (1986) Evaluating the econometric evaluations of training programs with experimental data. American Economic Review 76:604-619. Lashley, K. S., and Watson, J. B. (1922) A Psychological Study of Motion Pictures in Relation to Venereal Disease Campaigns. Washington, D.C.: U.S. Interdepart- mental Social Science Board. Leamer, E. E. (1978) Specification Searches: Ad Hoc Inference with Nonexperimental Data. New York: John Wiley & Sons. Levine, R. (1987) Community Consulting. Presented at the annual meetings of the American Psychological Association, Symposium on Clinical Trials in AIDS, New York, August 30. Leviton, L., Valdiserri, R. O., Lyter, D. W., Callahan, C. M., Kingsley, L. A., and Rinaldo, C. R. (1988) AIDS Prevention in Gay and Bisexual Men: Experimental Evaluation of Attitude Change from Two Risk Reduction Interventions. Presented at the annual meeting of the American Evaluation Assocation, New Orleans, La., October 19. Light, R., and Pillemer, D. (1984) Summing Up: The Science of Reviewing Research. Cambridge, Mass.: Harvard University Press. Magidson, J. (1988a) CHAID, LOGIT and log linear modeling. In Marketing Infor- mation Systems. New York: McGraw-Hill and Data Pro Research. Magidson, J. (1988b) Progression beyond regression. DMA (Direct Marketing Associ- ation) Research Council Newsletter 4:6-7.

OCR for page 316
354 ~ LIMITING THE SPREAD OF HIV Mathiowetz, N. (1987) Response error: Correlation between estimation and episodic recall tasks. Pp. 43~435 in Proceedings of the Survey Research Methods Sec- tion: American Statistical Association. Washington, D.C.: American Statistical Association. Mathiowetz, N., and Duncan, G. J. (1984) Temporal patterns of response errors on retrospective reports of unemployment and occupation. Pp. 652-657 in Proceed- ings of the Survey Research Methods Section: American Statistical Association. Washington, D.C.: American Statistical Association. McAuliffe, W. E., Doering, S., Breer, P., Silverman, H., Branson, B., and Williams, K. (1987) An Evaluation of Using Ex-Addict Outreach Workers to Educate Intra- venous Drug Users About AIDS Prevention. Presented at the Third International AIDS Conference, Washington, D.C., June 1-5. *McTavish, D. G., Cleary, J. D., Brent, E. C., Perman, L., and Knudsen, K. R. (1977) Assessing research methodology: The structure of professional assessments of methodology. Sociological Methods and Research 6:3-44. Meier, P. (1972) The biggest public health experiment ever: The 1954 field trial of the Salk poliomyelitis vaccine. In J. M. Tanur, F. Mosteller, W. Kruskal, R. F. Link, R. S. Peters, and G. Rising, eds., Statistics: A Guide to the Unknown. San Francisco: Holden-Day. Miller, W. R. (1986) Inpatient alcoholism treatment. American Psychologist 41:794- 805. Mitchell, J. V., Jr., ed. (1985) The Ninth Mental Measurements Yearbook, 2 vols. Lincoln: Buros Institute of Mental Measurements, University of Nebraska. Morisky, D. E., DeMuth, N. M., Field-Fass, M., Green, L. W., and Levine, D. M. (1985) Evaluation of family health education to build social support for long term control of high blood pressure. Health Education Quarterly 12:35-50. Moskowitz, J., Malvin, J. H., Schaeffer, G. A., and Schaps, E. (1983) Evaluation of a cooperative learning strategy. American Educational Research Journal 20:687-696. Moskowitz, J., Malvin, J. H., Schaeffer, G., and Schaps, E. (1984) An experimental evaluation of a drug education course. Journal of Drug Education 14:9-22. Moskowitz, J., Schaps, E., Malvin, J., Schaeffer, G., and Condon, J. (1981) An Evaluation of an Innovative Drug Education Program: Follow-Up Results. Report to the National Institute on Drug Abuse. Pacific Institute for Research and Evaluation, Napa, Calif. Moskowitz, J. M., Schaps, E., Malvin, J. H., and Schaeffer, G. A. (1984) The effects of drug education at follow-up. Journal of Alcohol and Drug Education 30:45~9. *Mosteller, F. M., Gilbert, S. P., and McPeek, B. (1980) Reporting standards and research strategies: Agenda for an editor. Controlled Clinical Trials 1:37-58. National Cancer Institute. (1988) Summary: Community Intervention Trial for Smoking Cessation. Bethesda, Md.: National Institutes of Health. National Research Council. (1975) Protecting Individual Privacy in Evaluation Re- search. Report of the Committee on Federal Agency Evaluation Research. Washington, D.C.: National Academy Press. National Research Council. (1982) An Analysis of Marijuana Policy. Report of the Committee on Substance Abuse and Habitual Behavior. Washington, D.C.: National Academy Press. Ostrom, T. M., Steel, C. M., Rosenblood, L. K., and Mirels, H. L. (1971) Modification of delinquent behavior. Journal of Applied Social Psychology 1:118-136. *Pear, R. (1984) Taking the measure, or mismeasure of it al1. New York Times, August 28.

OCR for page 316
EVALUATING AIDS INTERVENTIONS ~ 355 Public Health Service. (1988) Understanding AIDS: A Message from the Surgeon General. Rockville, Md.: Public Health Service. Reisner, L., David, J., and Turnbull, B. (1988) Evaluation of the Chapter I Tech- nical Assistance Centers (TAC). Report to the U.S. Department of Education. Washington, D.C.: Policy Studies Associates. Riecken, H. W., Boruch, R. F., Campbell, D. T., Caplan, N., Glennau, T. K., Pratt, J. W., Rees, A., and Williams, W. W. (1974) Social Experimentation: A Method for Planning and Evaluating Social Programs. New York: Academic Press. Rivlin, A. M., and Timpane, M.,- eds. (1971) Ethical and Legal Issues of Social Experimentation. Washington, D.C.: Brookings Institution. Robertson, L. S., Kelley, A. B., O'Neill, B., Wixom, L. W., Eiswirth, R. S., and Haddon, W. (1974) A controlled study of the effect of television messages on safety belt use. American Journal of Public Health 64:1071-1080. Rosenbaum, P. (1987) A nontechnical introduction to statistical power and control of bias. Pp. 174-185 in J. Steinberg and M. Silverman, eds., Preventing Mental Disorders. Rockville, Md.: National Institute of Mental Health. Rossi, P. (1987) Estimating the number of homeless in Chicago. Pp. 1-7 in Proceedings of the Section on Survey Research Methods: American Statistical Association. Washington, D.C.: American Statistical Association. Rossi, P. (1988) Homelessness in America: Social Research and Policy. New York: Twentieth Century Fund. Rubin, D. (1974) Estimating causal effects of treatments in randomized and nonran- domized studies. Journal of Educational Psychology 66:688-701. Rubin, D. (1977) Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics 2:1-26. Ruffin, J. N., Grizzle, J. E., Hightower, N. C., McHardy, G., Shull, H., and Krisher, J. B. (1969) A cooperative double blind evaluation of gastric freezing in the treatment of duodenal ulcer. New England Journal of Medicine 281:16-19. Rutstein, D. C. (1969) The ethical design of human experiments. Daedalus 98:523-541. Schaps, E., et al. (1981) A review of 127 drug education prevention evaluations. Journal of Drug Issues 11:17-43. Schaps, E., Moskowitz, J., Condon, J., and Malvin, J. (1982) A process and outcome evaluation of a drug education course. Journal of Drug Education 12:353-364. Schinke, S., Blythe, B. J., and Gilchrist, L. (1981) Cognitive-behavioral prevention of adolescent pregnancy. Journal of Counseling Psychology 28:451-454. Schneider, A. L. (1980) Effects of status offender deinstitutionalization in Clark County, Washington. Presented at the annual convention of the American Society of Criminology, San Francisco, Calif., November 5-8. Severy, L. J., and Whitaker, J. M. (1982) Juvenile diversion: An experimental analysis of effectiveness. Evaluation Review 6:753-774. Silverstein, A. M. (1981) Pure Politics and Impure Science: The Swine Flu Affair. Baltimore, Md.: Johns Hopkins University Press. Smith, T. W. (1983) The hidden 25 percent: An analysis of nonresponse on the 1980 General Social Survey. Public Opinion Quarterly 47:386-404. Solomon, M. Z., and DeJong, W. (1986a) The impact of clinic based educational videotapes on male gonorrhea patients' knowledge and treatment behavior. Ap- pendix A in Final Report to the Center for Prevention Services, Centers for Disease Control: October 1, 1983-September 30, 1986. Educational Development Center, Newton, Mass. Solomon, M. Z., and DeJong, W. (1986b) Recent sexually transmitted disease preven- tion efforts and their implications for AIDS health education. Health Education Quarterly 13:301-316.

OCR for page 316
356 ~ LIMITING THE SPREAD OF HIV Solomon, M. Z., and DeJong, W. (1986c) STD prevention through condom use: Changing patients' knowledge, attitudes, and behavior. Appendix A in Final Report to the Center for Prevention Services, Centers for Disease Control: October 1, 1983-September 30, 1986. Educational Development Center, Newton, Mass. Solomon, M. Z., DeJong, W., and Jodrie, T. A. (1986) Improving drug-regimen adherence among patients with sexually transmitted disease. Appendix A in Final Report to the Center for Prevention Services, Centers for Disease Control: October 1, 1983-September 30, 1986. Educational Development Center, Newton, Mass. Strecher, V. J., DeVellis, B., Becker, M., Rosenstock, I. M. (1986) The role of self efficacy in achieving health behavior change. Health Education Quarterly 13:73-91. Teitlebaum, L. E. (1983) Spurious, tractible, and intractible legal problems: A positivist approach to law and social science. Pp. 11-48 in R. F. Boruch and J. S. Cecil, eds., Solutions to Ethical and Legal Problems in Social Research. New York: Academic Press. Thistlethwaite, D. L., and Campbell, D. T. (1969) Regression-discontinuity analy- sis: An alternative to the ex-post facto experiment. Journal of Experimental Psychology 51:309-317. Tobler, N. (1986) Meta-analysis of 143 adolescent drug prevention programs: Quantita- tive outcome results of program participants compared to a control or comparison group. Journal of Drug Issues 16:537-567. Trochim, W. (1984) Research Design for Program Evaluation: The Regression Dis- continuity Approach. Beverly Hills, Calif.: Sage. *U.S. Department of Education, National Institute of Education. (1977) The Joint Dissemination Review Panel Ideabook. Washington, D.C.: National Institute of Education. *U.S. General Accounting Office. (1975) Evaluation and Analysis to Support Decision- making. Washington, D.C.: U.S. General Accounting Office. *U.S. General Accounting Once. (1978) Assessing Social Program Impact Evaluations: A Checklist Approach. Washington, D.C.: U.S. General Accounting Office. U.S. General Accounting Once. (1986) Teenage Pregnancy: 500,000 Births but Few Tested Programs. PEMD-86-16 BR. Washington, D.C.: U.S. General Accounting Once. Valdiserri, R. O., Leviton, L., et al. (1986) A Randomized Trial Evaluating Two Interventions Promoting AIDS Risk Reduction (proposal submitted to CDC). University of Pittsburgh. Valdiserri, R. O., Lyter, D. W., Leviton, L. C., Stoner, K., and Silvestre, A. (1987) Applying the criteria for the development of health promotion and education to AIDS risk reduction programs for gay men. Journal of Community Health 12:199-212. Vincent, M., Clearie, A. F., and Schlucter, M. D. (1987) Reducing adolescent pregnancy through school and community-based education. Journal of the American Medical Association 257:3382-3386. Waskow, J. E. (1984) Specification of the treatment variable in the NIMH treatment of depression collaborative research program. In J. Williams and R. L. Spitzer, eds., Psychotherapy Research. New York: Guilford Press. Weinstein, N. D. (1987) Unrealistic optimism about susceptibility to health problems: Conclusions from a community-wide sample. Journal of Behavioral Medicine 10:481-500.