Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Evaluating HIV Testing and Counseling Projects In this chapter the panel outlines strategies for evaluating CDC's widely disseminated program of funding HIV testing and counseling services. Goals for this program are similar to those discussed for CBOs: behavior change, HIV surveillance, and public education (CDC, 1988~. Strategies for evaluating the counseling and testing program are somewhat different, however, because the most credible research design for answenng, "Does the program make a difference?" is not appropriate here: that design is the randomized experiment in which some people (controls) receive no services. In the case of counseling and testing for HIV, the panel be- lieves strongly that having a no-treatment control groupthat is, denying access to information.that could have important consequences for peo- ple's personal planning and medical management of infection would be unethical. BACKGROUND AND OBJECTIVES In terms of expenditures $100 million in fiscal year 1989CDC's support of counseling and testing services is its largest AIDS intervention program. At present, the Center for Prevention Services (CPS) channels funds for such programs through 62 cooperative agreements with states, territories, and a handful of major cities to support this widescale program. Grantees provide HIV testing and counseling services free of charge in a variety of health care settings. Through the program, individuals are offered a dual AIDS inter- vention: confidential (frequently anonymous) HIV testing and pre-and 102
HIV TESTING AND COUNSELING ~ 103 pastiest counseling. In addition, in cases of seropositivity, a third service may be available: partner notification and referral. CDC encourages infected individuals to notify their partners of their exposure risk and to refer them to testing and counseling; if clients refuse, CDC recommends that a well-tra~ned site practitioner do the notification. However, partner notification is not mandatory.) From discussions with program staff, the pane] learned that the purpose of the counseling and testing program has evolved and may still be evolving. One of the original motives for the program was to divert individuals from using blood basics to learn Heir antibody status. Accordingly, shortly after the licensure of He ELISA test in March 1985, a program was initiated to deploy a series of what was called alternative test sites around the country. Since then, the demand for counseling and testing services has increased dramatically. In March 1986, CDC recommended that infected but asymptomatic individuals be encouraged to come In for counseling (CDC, 1986), and services were expanded beyond alternative test sites to other health facilities.2 In the next year, demand grew threefold when the Surgeon General recommended testing for heterosexually active individuals and recipients of blood products. There has been widespread support in the public health community for expanded programs of voluntary testing for all those who may have been exposed to HIV (see, for example, IOM/NAS, 1988:74~. ~ estab- lishing new HIV testing and counseling sites, priority has been given to projects that serve those segments of the population Hat are most likely to be infected or that engage In behaviors that risk HIV transmission (CDC, 1987:510~. Halfway through 1989, there were more than 1,600 counseling and testing sites nationwide, and that number is expected to grow to 2,000 by the end of the year. HOW WELL ARE SERVICES DELIVERED? As noted above, the overall objectives for the counseling and testing program are surveillance, promotion of behavior change to reduce the risk of infection, and public education. In addition, one of CDC's internal documents on process performance indicates that "quality" counseling is 1 For example, the 1987 CDC (1987: 513) guidelines for counseling and testing note that persons who are antibody positive should be instructed in how to notify their partners and to refer them for counseling and testing. If they are unwilling to notify their partners or if it cannot be assured that their partners will seek counseling, physicians or health deparanent personnel should use confidential procedures to assure Mat their partners are notified. 2The shift in emphasis from testing to counseling was accompanied by a change in nomenclature to refer to the sites as counseling and testing sites.
104 ~ EVALUATING AIDS PREVENTION PROGRAMS a goal, which introduces a new concept to be evaluated.3 Finally, from He history of CDC's repeated recommendations to encourage various populations to take advantage of counseling and testing, along with the ever-widen~ng deployment of testing sites, the pane} inferred Hat accessibility was also an implicit goal of the program. To abbess these goals, the pane! believes that five aspects of service delivery need to be evaluated. To do so, a number of information gathering me~ods can be used. The pane' recommends that data be gathered from multiple sources including testing sites, clients, groups at increased risk of HIV infection, and independent observer~to eval- uate five aspects of service delivery: the adequacy of the counseling and testing protocol, the adequacy of the coun seling that is actually provided, the proportion of clients that complete the full protocol, the accessibility of services, and the nature of the barriers, if any, to clients seeking and completing counseling and testing. The rate of completion of the program and the identification of banders to participation in the program are subsumed under "adequacy" and "accessibility. " We use "adequacy" to mean correspondence with client needs. In terms of the testing protocol, client needs include confidentiality; rea- sonable waiting periods; secure linkage between counseling and testing; provision of test results; and, possibly, partner notification. In terms of counseling, client needs include support; risk assessment; and accurate and appropriate information about the transmission of the virus, risk factors, risk reduction behaviors and techniques, coping skills, and the meaning of test results. In the case of seropositivity, client needs also include information and counseling about the medical and psychological management of infection and partner notification. We realize that as- sessing adequacy and accessibility involves judgment on He part of an evaluator; nonetheless, we believe a system of cataloguing the fulfillment of needs can be implemented. CDC has in fact developed a prototype "HIV Disease Intervention Skills Inventory" for managers that could be useful in the evaluation of adequacy (CDC, n.d.). There are four sources of information on the venous aspects of ser- vice delivery: the administrators and staff of testing and counseling sites; the clients who use the service; specific population groups who should sin addition to any "calculable results" from He program, counseling efforts are to "be judged by the quality of the process performance" (CDC, n.d.).
HIV TESTING AND COUNSELING ~ 105 TABLE 5-1 Four Sources of Formation about Delivery of Counseling and Testing SOURCES OF INFORMATION SERVICE ASPECT Sites Population Clients Groups Observers Protocol adequacy X X Counseling adequacy X X Client completion X X Accessibility X X X Client barriers X X use services (but may not); and independent observers who visit the sites. Table 5-1 identifies the sources of information that are most likely to be useful for evaluating each feature of HIV testing and counseling service delivery. Each of these sources of information can be translated more or less directly into a method of data collection or study design. In the following sections, the panel suggests several designs for collecting data from each of these information sources. Note that these designs are not mutually exclusive; rather, they are complementary, as each provides a different perspective on the adequacy and accessibility of services. A Site Services Inventory A typical setting for counseling and testing is a local health deparunent, but services are also offered In institutional facilities, health clinics for the treatment of sexually transmitted diseases or drug use, family planning clinics, and other settings. Because clients of these over sites often have non-HIV related reasons for their visits, they may not be motivated to return for test results and pastiest counseling. Thus, the sexing in which testing and counseling services are delivered may be a significant factor to be taken into account in analyzing data collected from project sites. As a first step, the panel believes CDC should prepare an inventory of the venous services that are delivered by HIV testing and counseling sites. Although recipients of CDC funding for testing and counseling are required to provide quarterly summary data about their services, we believe these data are insufficient to descnbe the range of testing and counseling activities now being undertaken at the 1,600 CDC-supported sites across the county. The 62 grantees funded by CDC have no uniform method for reporting data from their counseling and testing sites, and the level of detail provided on any particular service may be inconsistent from
106 ~ EVALUATING AIDS PREVENTION PROGRAMS one grantee to the next. These inconsistencies preclude cross-tabulation or any detailed analysis of how and to whom services are delivered. A promising solution to these problems would be the widespread adoption of an enhanced version of the data management system devel- oped by CPS last year. The central feature of this system is the REV Counseling and Testing Report Form, an electronically scannable record of each visit to a testing and counseling site (see Figure 5-~. The form of- fers the advantages of uniform reporting and relatively easy data analysis. It assigns an identification number to and elicits data on each counseling intervention, recording the type and date of the site visit as well as visitor demographics and risk factors. These data can then be used in four ways: to link test data with type of service site; to identify the demographic distribution of clients and determine trends ~ client utilization of the site; to link test data with risk behavior and demographic data to assess trends in seropositivity; and to link test results with pastiest information to determine trends in return rates for results, counseling, and partner notification and referral. The data gathered with this form would also allow analysis of variations among project sites and geographic regions. The pane! believes that He value of this inventory can be enhanced and that it would be valuable to augment the current form to collect information on other relevant variables (such as counselor characteristics, length of counseling session, whether the session is an initial or repeat visit).4 Furthermore, the pane] believes that the required use of this form by all HIV testing and counseling sites funded by CDC and other government agencies would permit the development of uniform and manipulable data bases that could be used for the evaluation of testing and counseling projects. Some states have developed and are using alternative forms and are already building data bases. To avoid requiring these states to modify their efforts, it may be feasible for CDC to furnish technical assistance to make state data sets compatible with the federal form. Client Surveys The mere presence of clients indicates Hat He testing and counseling intervention has been successful in attracting people to receive services. The gross number of individuals served, however, does not tell us whether 4 The form could also provide an indicator of the socioeconomic status of clients. It has been postulated but not proven that, increasingly, the AIDS epidemic is becoming lodged in the most disadvantaged segments of the American population. Until now, such arguments have been based on trends in the race and ethnicity of new AIDS case~which does not provide a wholly appropriate analysis. To track trends in the socioeconomic status of persons served by counseling and testing projects, the panel suggests that a question on education level be added lo the form.
HIV TESTING AND COUNSELING | 107 ~ U5 GOVERNMEN7 PRIFITING OFFICE:1990734~539 ~ HIV COUNSELING AND TESTING REPORT FORM A IDENTIFICATION NQ _ _ _ PROJ SmE DATE OF INmAL VISIT LOCAL USE ONLY AREA SITE TYPE NUMBER MONTH DAY YR m m rl I I c~ mm rl I I I I I G) (D ~) O (y) ~) C HiV CTS ~) (3D~ C JAN O~) ~C (3D Q) 0(31)~)(0~) G) ~) G) ~) O) O CE) STD O 2 3 O FEB (D ~) ~D CD ~ O CO O (D CD (~:) ~ Q OD ~) ¢) GE) DRUG TRMT O) O) ~) O MAR ¢) ~) 48D (D ~) CD (D (O ~) ~) 3 ~ ~ ~ Cl) GO ~) FAMILY PL ~ GO Gl) O APR (~ (D CED O O (:O ~ CD Gl: O) CI) ~ ~ O ~) G) PRENAT/OB Q (:iD ~) O MAY ~) 4~ O Q 09 0 ~ `33 ~ ~ CD CD ~) ~) O (~) OD TB O CD ~ O JUN 0 ~ C CD ~ (~) CD CD ~ ) 5D ~ G[) O) OTHER HD G) (O ~ O JUL GD (O (D G) CID (30 (O 60 CD CO ~ (O CD (D ~) PRISON l:D O) tI) O AUG CI) O (D (D ~ C ~) 3 ~ O O O G) ~:) COLLEGE (O G) ~ O SEP OD G) (D G) CID CD G) (~ C ~) (E) (D CD~) 6i9PRIV MD O)(ID~) OOCT O) C~ O ~)6D~ G~ OTHER O NOV ~) UNKNOWN O DEC RESIDENCE STATE COUNTY ZIP CODE AGE m rl I I I I I I I I m =~) CO(D~ ~QG)Ci:) =m =~= ~CDC~= ~CD ~ GD ~ CD CD Gi) GO ~ GD ~ C~ GO CD~ ~o mmo~ ~ mm ~CD~ =~CD=~ ~CD ~ ~ ~ 6DCD CD(D =~(D om~a)m mm m CD~ C~ ~<~ ~C30~ ~C~ SEX (3 MALE ~ FEMALE RACE/ETHNICITY O Q WHITE CE) BLACK Q) HISPANIC. F8R~ I tST O ASIAN/PACIFIC ISL COUNSELED C AM INDIAN/AK NATIVE ~) OTHER (D UNDETERMINED · IF HISPANIC, SKCIFY O ~MEXICAN/MEX AMER ~) PUERTO RICAN CO CUBAN G) OTHER CD UNDE I tRMINED REASON FOR VISm (mark all that apply) O REQUESTING HIV TEST O FOLLOW-UP TO STD VISIT O STD EXAM,fTREATMENT O REf BY HIV+ SEX PARTNER O REF BY STD SEX PARTNER O REF 8Y HEALTH DEPT/HIV O REF 8Y HEALTH DEPT/STD O REF BY PMD/BBtHOSP O REQ IMMIGRATION O PRENATAL O TB INFECTION/DISEASE O SYMPT HlV/AtDS DISEASE O ASYMPT, WORRIED A80UT AIDS O OTHER O NOT STATED O UNKNOWN MOhml O JAN O FEB O MAR O APR O MAY O JUN O JUL O AUG O SEP O OCT (3~) NO O NOV O O) YES O DEC O ~)NO CD YES P05 I I tST COUNSELED ~) NO IJ G) YES REFERRED FOR ~ TEST RISK EXPOSURE GROUP (mark all that apply) O MAN WHO HAD SEX W/A MAN O IV DRUG USER O PERSON WITH HEMOPHIL]A O BLOOD RECIPIENT. 1978~85 C HETEROSEXUAL O SEX PARTNER OF HOMOSEXUALfBISEXUAL G SEX PARTNER OF IV DRUG USER O SEX PARTNER Of PWA/+HN O SEX PARTNER OF PERSON WITH HEMOPHILIA C EXCHANGED DRUGStMONEY FOR SEX O NO KNOWN RISK EXPOSURE DAY YR m m ~ DAn- POS-l I EST COUNSELED ~) (3D ~}c oim ~ CD GD ~D PARTNER NOT~CAnON (po~ Only) O PATIENT ALREADY REfERRED PARTNERS CD ~ PATIENT WLL REFER ALL PARTNERS G) GO HD W~L REFER ALL PARTNERS 2) GD PATtENT/HD EACH TO MAKE REFERRALS ALL PARTNERS UNKNOWN/UNLOCATA3LE PATIENT DECLINES PARTICtPAT10N ~) SUBJECT NOT RAISED WITH PATIENT RESERVED 1 2 3 4 5 e 7 8 9 10 REFUSED HIV TEST O O YES O O O O O O O O O O ~) ~ (31) (O ~ C3D O) ~) GD (3D CD ~ 6) Q ~) CD O) G) CD 6) LAB TIEST RESUL~ 2 ~ ~ ~) (D ~) CO CD ¢) CD WESTERN OTHER C :) GO CD CD CD ~ GO CD Cl) t3 ELtiA BLOT CONFIRMATORY Q <g ~ <~~ Q <~> <~~ Q (D (~ (ED O O O ~ ~ ~ ~ ~ ~ ~ ~ ~ im NEGAnVE (D NEGATIVE (3D NEGATNE G) (~ ~ ~) GD (D (O C3~) '~ ~ CD REPEATEDLY ~ POSITIVE ~ POSITIVE ~ C2) O) 0) ~ CD iO) ~ ~ CD REACTNE ~ INDETEF.=NATE ~) G) 6D ~ C~ OD CO ~) G) (O (O ~) (O C~ CD i~ ~ D ( (O FIGURE 5-1 HIV Testing and Counseling Report Form
108 ~ EVALUATING AIDS PREVENTION PROGRAMS IDENTIFICATION NO. - CO CD CD c3 ~ CD CD CD ~ DATE PAST I EST COUNSE' EN MONThI DAY YE ~ mm O JAN CD A) @; O FEB 6) Ci) ~ 5 MAR If) AD APR CD~ GED O MAY A) O JUN CD O JUL ED O AUG CD O SEP (A O OCT (A O NOV O DEC LOCAL USE ONLY (9 ~ G) 2 1 1 ~ 1 1 1 1 1 1 (iD ~ (D ~ ID ~=CDCOG)~ COW CD~COGOCDCOGOCD ID ~CDCED~CE) ~6DGD~ =~a)(D<DCD c~CD2~)~)~ HIV COUNSELING AND TESTING REPORT FORM POSTTEST COUNSELING & PARTNER NOTIFICATION POSTTEST REFERRED COUNSELED FOR TB TEST o (ID NO (GINO ~ YES O G) YES PARTNER NOTIFICATION (Positives Only) O CD PATIENT ALREADY REFERRED PAFFrNERS (D PATIENT WILL REFER Al I PARTNERS C[) HD WILL REFER ALL PARTNERS PATIENT/HD EACH TO MAKE REFERRALS ALL PARTNERS UNKNOWN/UNLOCATABLE PATIENT DECLINES PARTICIPATION C2) SUBJECT NOT RAISED WITH PATIENT PLEASE REMOVE THIS PIECE BEFORE SCANNING FIGURE 5-1 HIV Testing and Counseling Report Form (cont'd)
HIV TESTING AND COUNSELING | 109 the intervention is providing satisfactory services to the groups that need them. A survey of the clients of a testing and counseling site can: · provide specific information on the accessibility of He site and its services to specific populations; · reveal problems with the intervention as it is currently de- livered; · elicit information about users' experiences in obtaining the service, such as how long it takes to get an appointment or to get test results, etc.; · gather data on the scope of the counseling clients receive: the extent of pertinent information provided about HIV infection and AIDS, the time spent with and the emotional support lent by the counselor, the nature of referrals for medical and over support; and gather information about whether the respondent completed the testing and counseling protocol, and if not, why. While the reliability and validity from such surveys must always be con- sidered, information from such surveys might also be helpful in design- ing more inviting and accessible settings for testing and counseling (e.g., "attractive" physical surroundings, "convenient" locations for services), understanding what aspects of the pretest counseling session encourage clients to return to learn tests results, assessing the optimal content and timing of a pastiest counseling session and the provision of referral ser- vices, and specifying He profiles of the more effective counselors for different circumstances. . Population Surveys Another way to evaluate how well services are delivered is to conduct surveys of populations that include potential and actual clients to deter- m~ne whether counseling and testing services are accessible to all who need or want them. This strategy can be used to evaluate barriers to access. The surveys can be directed toward the general population, to- ward particular neighborhoods or communities, or toward high-risk or hard-to-reach groups. They can be used to measure the proportion of the specific population that has had experience with counseling and testing and the proportion of He population that wants services but cannot get Hem (or has chosen not to seek them). Data from such surveys could be analyzed according to demographic and risk factors to identify groups that are not adequately served and the barriers that need to be overcome to make the services more accessible to those groups.
110 ~ EVALUATING AIDS PREVENTION PROGRAMS Such surveys present other opportunities as well. They could be used to identify the testing and counseling services that are desired or expected by a given population and could help project administrators better understand the service needs of these individuals. They might also afford insight into the group's awareness of the availability of services and the perceived cultural relevance of the project to users and nonusers alike. Based on this information, efforts might be made to publicize the availability of services and to modify as necessary such project characteristics as location, hours, site design, esthetics, informational materials, and counselor sensitivity. This kind of survey could also gather users! information on the group's concerns about the confidentiality (or anonymity) of HIV testing and counseling and over concerns or fears about Me test procedure. Case Studies Using Direct Observation The direct observation of interactions at testing and counseling cen- ters is the panel's final suggested study design. This design could be implemented by both nonparticipant observers as well as "professional customer" participants.5 This method mainly provides qualitative data, although quantitative data can be gathered as well. The pane] is aware that CDC already employs regional monitors and public health advisers to conduct oversight and quality assurance activities for the counseling and testing centers.6 Nevertheless, the pane} believes that case studies using direct observation might identify those factors that produce testing and counseling environments that are particularly supportive of clients and that provide effective contexts for educational messages. In selecting the project sites for case studies, choices should be spread across different types of facilities (e.g., public health deponents, clinics for sexually transmitted diseases, drug treatment centers, etc.) and regions of the country, both geographically and by level of HIV prevalence. Methodological Issues As noted above, the four separate options for assessing service delivery are not mutually exclusive. Indeed, the approaches suggested are com- plementary to one another, and all might be undertaken to yield He most 5As noted in chapter 1, the panel believes that project administrators should be given advance notifi- cation that professional customers will be visiting their sites for counseling and testing services, and prior consent should be solicited before this method of data collection is used. fits document on process perfollllance standards covers a series of steps its counselors are to talce in providing counseling and partner notification, and it provides a skills inventory to be used by managers in evaluating counselors performance (see CDC, n.d.).
HIV TESTING AND COUNSELING ~ 1il comprehensive information. Gaps in coverage by an individual study design can be filled in by data from research using another design. Although a services inventory could show who is using testing and counseling services and could monitor trends in this use over time, some caution is warranted in interpreting inventory data. The main method- ological problem will be the accuracy of the project site's reporting. Inaccurate reporting may occur through errors and it is also possible that inaccurate reports may be made purposely to convey a false picture of site activities. But the most plausible threat to accurate reporting will probably be the burden the forms place on the counselors. In the course of a busy day win much "real" and pressing work to do, it is virtually inevitable that filling in forms will not receive high priority. Fur- thermore, where testing is anonymous, repeat testers could be counted multiple times. This problem can be avoided by adding a question to the scannable form that asks whether a client has been previously tested. A major concern about client surveys is Me reliability and validity (or mearungfuiness) of the measurements that are obtained with this memos. Clients who are surveyed, for example, may have little or no experience with counseling. Consequently, they may have unrealistic expectations of what and how services should be delivered, expectations that may color Heir responses.7 Surveys can be conducted to gather data from two levels of society: specific, high-nsk groups and the general population. To conduct surveys of high-risk populations, probability samples should be used whenever possible. When such a sampling frame is not feasible, replicable con- venience samples could be used (see the discussion in Turner, Miller, and Moses, 1989:150-1531. However, this latter method will not provide estimates that will be generalizable to the population of persons in the specific high-nsk groups. As discussed In the section on client surveys, the reliability and validity of the responses obtained in group surveys will always be a matter of concern. The panel notes, for example, that respondents in such surveys may have unrealistic expectations of what and how services should be delivered. The general population is of lesser interest than specific high-risk groups, but a large enough sample can provide important information about certain subpopulations. To this end, He panel suggests that CDC take advantage of the National Health Interview Survey (NEDS) sponsored by the National Center for Heals Statistics. As described in Chapter 3, He NHIS is a weekly household interview survey of a probability sample 7 For discussions of these complexities, see Bradburn and Sudman (1979); Smith (1984); and lower and Martin (1984).
112 ~ EVALUATING AIDS PREVENTION PROGRAMS of the civilian non~nstitutional~zed adult population of He United States. Since August 1987 the NITS has included questions about respondents' knowledge of HIV transmission and their experience with HIV testing; it has also collected limited information on behavioral risk factors. The pane! recommends that the NHIS be periodically aug- mented with several questions about accessibility and barri- ers to HIV testing and counseling services. Given its large size (approximately 50,000 households a year), the NEDS can provide samples of reasonable size, even of relatively rare pop- ulations, as long as they are found in households. For example, the September 1988 NITS estimated that approximately 3.5 percent of He total U.S. population (exclusive of those tested during blood donation) expected to have an HIV test in the next 12 months.9 Given the an- nual sample size, this means that 1,750 respondents intended to seek a voluntary HIV test. To conduct case studies, a site visit team might conduct open-ended interviews with key project staff and 5 to 10 clients. The interviews should focus on counseling and testing activities and materials, opin- ions about activities and materials, and the identification of key content, process, and organizational elements. Alternatively, with prior informed consent from site administrators, "professional customers" can pose as clients of the projects to gather information unobtrusively. A mix of seronegative and seropositive "customers" could be recruited to gather information about the adherence of sites to counseling protocols much In the same way that public health monitors do. For the purposes of case studies, the pane! notes that projects are evolving entities; changes may occur in personnel, organizational structure, project instrumentation and goals, and so on. Because of such developments, case studies cannot be conducted once and considered done. Instead, frequent studies are necessary to ensure good results. The pane} members did not agree about whether such site visits would be well received. Some members believe that site staff would welcome the opportunity to demonstrate their projects; others believe that staff would feel overly scrutinized. If there are strong negative reactions on the part of project staff, it may be necessary to spend considerable time 8 The panel understands from program staff that planning is under way at the National Center for Heals Statistics to add questions of this nature. 9Fith (1989:8) reports that 7 percent of the sample responded "yes" to the question, "Do you expect to have a blood test for the AIDS vines in the next 12 months?" Moreover, 51 percent of those who responded "yes" said the test would be "voluntarily sought" when they were asked, 'GENII it be part of a blood donation, voluntarily sought or part of some other activity that requires a blood sample?" ,1
HIV TESTING AND COUNSELING ~ 113 convincing local staff of the need for case study research. If local staff cannot be satisfied that the research is needed and beneficial, it may be quite difficult or impossible to conduct. If the alternative of using professional customers to gather information is considered, it may be desirable to ask for a site's informed consent before that site is funded for HIV testing and counseling services. Resources and Aspirations An inventory system requires record-keeping by personnel at the coun- seling and testing sites. It also requires a centralized professional staff to ensure the completeness of reporting, conduct data analyses, and dissem- ~nate results. Other costs of the inventory option should be relatively low because He data management system is based on personal computers, and the use of a scannable form minimizes labor costs for data entry. The pane! believes an inventory system could be implemented program-wide within 6 monks. The aspirations for this type of evaluation research are somewhat limited. Although a services inventory would provide data on individuals who avail themselves of services, it cannot identify people who need or want those services but who do not receive them. Another limitation of the system is that it does not provide data on the counseling and testing services provided by private physicians or clinics, blood banks, insurers, and other non-CDC funded sources. This lack is regrettable because such inflation could bear on an evaluation of CDC-funded services, such as whether a client completes a protocol at CDC-funded sites, or seeks services elsewhere. The major advantage of client surveys is that they provide informa- tion on most of the program aspects that must be assessed to determine how well testing and counseling services are being provided. An addi- tional advantage, when compared with other methods of data collection, is that information can be gathered on over intervention activities to which clients have been exposed. Furthermore, the client survey option is one of the least expensive methods of obtaining information about test- ing and counseling services. Nevertheless, it will require the involvement of personnel who are trained in survey research design and faTniliar with its methodological problems. To survey the general population, the NHIS could be expanded at periodic intervals to measure people's needs for H[V testing and counseling services on a national basis. (The NHIS cannot be used for local information.) This option would require some staff time for data tabulation and analysis on He part of the personnel responsible for the
114 ~ EVALUATING AIDS PREVENTION PROGRAMS household survey, but costs should be relatively low. Population surveys are somewhat Innited, however: for example, the THIS does not pennit access to homeless or institutionalized populations, which may include higher proportions of individuals at risk for HIV Han are included in the household population. For a number of reasons, surveys of samples of high-risk populations are more expensive and more difficult Han adding questions to the THIS. These surveys are labor intensive; In addition, they must be repeated at regular intervals In order to monitor changes in the need for counseling and testing services. Furthermore, this kind of research requires highly trained personnel to design and administer the surveys. Conducting case studies requires special skills and knowledge. Ex- pertise in AIDS prevention and in evaluation design is desirable, as is knowledge of the particular risk factors that are accessed In the sewing being studied (e.g., drug treatment centers). A team of two or three site observers is preferable to a single observer because of the range of knowledge desired ---evaluation methodology, counseling expertise, and any other site-specific expertise. Using more than one observer also makes it possible to assess He reliability of several reports. Site visits of 3 or 4 days would be required for each study site. Major costs ~n- clude observer salaries and their travel expenses. Each case study would require about 40 person-days per site. Case studies call provide a rich and ~n-depth look at some aspects of service delivery for a subset of test- ing and counseling sites, which will in turn be important for developing studies Hat evaluate comparative effectiveness. OPTIONS FOR EVALUATING WHETHER HIV TESTING AND COUNSELING SERVICES MAKE A D1FIERENCE The panel weighed several options for addressing the question, "Does He policy of providing free HIV testing and counseling services make a difference?" As noted above, He panel seriously considered the feasibility of randomized tests with a no-~eatment control group. One approach would be to use a randomized experiment at the site level in which individuals who sought services were randomly assigned either to an intervention condition or to a control condition in which they received services from alternative sources or no services at all (e.g., they were put on a waiting list). The individuals in both groups would be measured and compared on the relevant outcome variable to test the effects of treatment. Although conducting such an experiment is the usual desired strategy to evaluate effectiveness, the panel rejected it as unethical alla infeasible. In the context of a deadly epidemic, it is indefensible to
HIV TESTING AND COUNSELING ~ 115 withhold this treatment in the interests of conducting an experiment from any individual who desires it. In addition, follow-up for such a group would be extremely difficult as they would have little incentive to cooperate, and locating a diversely situated group would be difficult. The panel also considered a variation of a randomized experiment that would capitalize on delays in the implementation of projects, but concluded that such an experiment would not usually be feasible. This design variation assumes that lags occur in the deployment of counseling and testing projects because of scarce resources; as funding becomes available for some projects, sites can be randomly assigned to receive the intervention or to continue waiting. However, such a design is not very practical. F~rst, there are already a large number of sites throughout the county, and it is unlikely that a waiting list of homogeneous centers Is available. Second, the recruitment of control sites would be problematic because sites on the waiting list would have to be offered strong incentives to participate in data collection. Altemative strategies for assessing effectiveness (e.g., simple before- and-after designs that establish that a change has occurred or not oc- curred) do not suffice because they do not condom for rival explanations of changes In behavior, knowledge, or serostatus. Over competing ex- planations for such changes may include natural history (the adoption of change regardless of exposure to counseling and testing), the self- selection ot program participants, and the effects of conducting research. The suitability of such research designs for answering the question, "Does it make a difference?" is so low as to invite the investment of evaluation resources In more tractable areas, especially as the value of counseling and testing has been so widely accented. Indeed, the panel ~ . noted some presumptive evidence mat mv testing and counseling do have a positive effect. There is, for example, increasing evidence that testing can result in individual medical benefit among persons infected `',i~h HTV he e.n~hlin~s them to monitor their immune function and Initiate early prophylaxis tor Eneumocyst~s carinii pneumonia. Furthermore, an individual's knowledge about serostatus can be an important factor malting decisions about sexual behavior, needle sharing, and childbearing. For example, one study found that gay and bisexual men who were tested for HIV and received pretest and pastiest counseling were more likely than those who did not to reduce their incidence of unprotected anal intercourse (Coates, Morin, and McKusick, 19871. J ~ ~ ~ ~ . When HIV testing is performed, the panel believes that testing should be accompanied by counseling, both before the test is administered and after the test result is given to the individual. The need for such counsel-
116 ~ EVALUATING AIDS PREVENTION PROGRAMS ing and the ethical and practical motivations for providing it are discussed elsewhere (see, e.g., TOM/NAS, 1986, 1988; Presidential Commission on the Human Immunodeficiency Virus Epidemic, 1988:73-75~. At a m~- mum, pretest counseling is ethically mandated to ensure that individuads give informed consent. Posttest counseling is appropriate to ensure that individuals who are distressed by the results of their tests are comforted and that all individuals are warned about the n~.k of tr~n~mi~c~inn Huh their future behaviors. ~ ~~~ an, ~ ~^v4444~1 ~~ V~~11 Although the pane] found that the question of effectiveness should not be experimentally tested, it did not find the question uninteresting. On the contrary: the pane! discussed current research efforts that study the sequelae of HIV testing.~° One impetus for such studies has been the emerging indications that testing may have negative as well as positive effects (see Chapter 21. Although the evidence is sparse, these studies point to the need to monitor potentially negative as well as positive effects of HIV testing and counseling. WHAT WORKS BETTER? The heart of the question "What works better?" is how to max~n~ze the beneficial] effects of testing arid counseling. The way to learn what these effects are is through well-controlled studies that test two or more approaches to delivering the intervention. As recommended in Chapter I, the panel's preferred strategy for comparative tests is randomized ex- periments. For the "What works better?" question, the control group is assigned not to nontreatment but to an alternative treatment. So, for example, each individual who agrees to participate might be randomly assigned to one of two (or more) programs of counseling and testing that are thought to be effective but whose relative effectiveness is un- known. Because the groups are composed randomly, the comparison of outcomes such as client return rates for clients receiving regimen A or regimen B is Hen a fair one. In some circumstances, REV seroconversion may be a helpful out- come measure for evaluating He effectiveness of different counseling and testing projects. Yet the use of more proximate outcomes is desir- able because seroconversion will not be informative regarding behavioral . 10The particular approach taken in these research efforts is the natural history study. Using longitudinal cohorts, researchers have attempted to estimate the effect of Besting and counseling, compared with ocher factors in a person's life, on behavioral change. Such studies of gay men and IV drug users are currently under way. For example, among the cohorts of gay men, seroconversion rates and behavioral changes in men who have not been tested for HIV are compared with: (1) those who have been tested but do not know their antibody status and (2) those who have been tested and do know their status. Unfortunately, natural history studies do not lead to fully adequate, testable models of behavior, but when there appear to be consistent effects, those effects should be noted.
HIV TESTING AND COUNSELING ~ 117 change among persons whose initial test result is positive. Similarly, In populations In which HIV is not heavily seeded, seroconversion may be a rare event even though the population frequently engages in risky behaviors. Thus, experiments can assess how well different versions of an intervention work to increase a person's willingness to return to a site for test results, to increase his or her knowledge of risks, or to reduce risky behavior. Furthermore, experiments can assess the effectiveness of venous regimens in reducing identified side effects, such as psychological distress. With clear outcomes such as these in mind, alternative approaches to testing and counseling, based on theory and the perceived effectiveness of past approaches, can be evaluated through randomized experiments. After a brief discussion of the unit of assignment and experimental regimens, the next section presents appropriate study designs for answering the question, "What works better?" Randomized Experiments of Alternative Treatments Unit of Assignment As noted in Chapter I, the unit of assignment in a field experiment may be a large organizational unit such as a community, a smaller organizational unit such as a project (i.e., all of the clients of a project), or individual participants. There are several factors involved in He choice of treatment units (see Chapter I). The pane! suggests consideration of three types of assignment: it · random assignment of individual testing and counseling sites to alternative regimens; · random assignment of project staff members at a given site to the use of alternative regimens, and · random assignment of individual clients to alternative regi- mens at a site. In cases of random assignment of sites, we recognize that some facil- ities, such as those whose primary mission is not HIV-related, may not be amenable to being randomly assigned to provide different interventions. However, senice providers In a large city with several counseling and testing sites may be more flexible and should be encouraged to partici- pate In controlled expenments. When the preferred design is He random ~- 11 See Tu~ner7 Miller9 and Moses (1989:Chapter 5) for a discussion of precedents in other areas regard- ing the random assignment of entities (e.g.7 sites) tO altemative regimens.
118 ~ EVALUATING AIDS PREVENTION PROGRAMS assignment of individuals within a project to alternative regimens, the panel believes Mat at least two or three projects in different communities should be encouraged to cooperate in uniform tests. Because detectable relative differences may be small but important, the number of clients and sites involved in the tests must be large enough to estimate relative differences with confidence. Experimental Regimens There are a variety of experimental regimens that might be tested, but In designing randomized experiments of alternative counseling and testing treatments, it is best to test alternatives in which one regimen is not obviously better than the others. In the three examples noted In this section (below), the unit of analysis varies. These regimens test the effects on behavior of alternative modes of providing counseling and testing. Some of the alternatives fall into He category of structural variables. For example, the setting in which testing and counseling is delivered may have an effect on whether an individual returns for test results and foDow-up counseling. As noted above, a client's initial purpose in visiting a site may not be HIV testing; similarly, some sites and their staffs may be geared primarily to providing services other than counseling and testing (a drug ~eatrnent facility is one such example). Particular service delivery aspects of sites may also produce different effects. For example, Rugg and colleagues (1988) found that higher return rates for REV test results were associated win such site characteristics as a shorter wait for testing and the comfort and nature of He setting. The number of sessions as well as the content of counseling may have an effect on cognition and behavior. Other process variables Hat may influence return rates involve the adequacy of services In teas of emotional support and medical service referral. The pane! recommends that evaluations of "What works better?" focus on the comparative effectiveness of testing and counseling services that (~) are delivered in different settings, (2) have different content, tIuration, and intensity, and (3) are accompanied by different types of supportive services. Service Delivery Setting. As discussed above, the accessibility and suitability of testing and counseling projects are critical issues. Projects now exist to serve gay men, IV drug users in treatment, and users of general public health agencies such as STD or family planning clinics. These projects are widely distributed, but they are not necessarily acces- sible in all communities or equally accessible for all types of individuals.
HIV TESTING AND COUNSELING ~ 119 In addition to assessing whether particular projects serve more of the individuals who desire testing than other projects, it is important to un- derstand whether particular service settings are more effective contexts for providing testing and counseling to different types of clients. Content, Duration, and Intensity of Counseling. The experimental regimen could consider a number of potentially more effective ~nterven- tion approaches, including more frequent, longer, or intensive counseling. The content of counseling messages could be vaned, too: for example, the elimination of risk behaviors or the adoption of protective behaviors could be stressed. Other enhancements may include the use of support groups, the involvement of partners, and outreach services for individuals who continue to engage In high-risk behavior. As the standard protocol for counseling and testing evolves and is improved, experiments can ex- am~ne Be increased effectiveness of enhanced programs compared with the costs of their implementation. Additional Services. The third major area for exploration is the effect of providing services beyond the basic counseling and testing intervention now being offered. Increased relapse prevention services for IV drug users In treatment who become seropositive are a good example. A common reaction to stress by IV drug users is to seek and use drugs for stress reduction. Relapse prevention projects address this issue In general and could include components that are specially geared toward seropositive clients. Another example of potentially risk- reducing services is psychological counseling (beyond the counseling provided with HIV test results), which might help diminish adverse stress reactions. A creative evaluator is likely to identify many more Interventions whose effectiveness can be assessed using alternative regimens (e.g., videotapes, group sessions, cognitive interventions, etc.~. Yet not all sites will be suitable for testing augmented regimens. Consistent with the overall recommendations, the panel believes that sites should be selected for randomized teals on the basis of their willingness to cooperate and He potential effectiveness and replicability of the augmented intervention programs they would offer. Methodological Issues Conducting randomized experiments can present various problems, in- cluding cost, impediments to random selection of treatment units, and difficulties In collecting complete data from participants and to retaining participants in He study.
120 ~ EVALUATING AIDS PREVENTION PROGRAMS As noted in Chapter 1, offering alternative services to individuals within a project deserves a special note of caution. Problems can arise in that both clients and project staff may be uncomfortable with random assignment, and this discomfort could preclude randomized assignment. A practical solution In such cases is making the site He unit of assignment: that is, all clients at site ~ receive service regimen A, all clients at site 2 receive regimen B. etc. Data collection poses another challenge to study designers. Although eliciting sensitive information may seem feasible given the confidentiality assurances available under Section 303 of the Public Health Service Act, data gathering at HIV testing and counseling sites may be difficult. At sites that offer anonymous testing, it will not be possible to do follow-up interviews unless either He clients agree to confidential (rather than anonymous) data collection or the study uses a follow-up method that preserves client anonymity. In the latter regard, O 'Reilly (CDC, 1989) has reported recontact rates of between 50 and 80 percent using one scheme that preserves client anonymity. These recontact rates are certainly impressive, but they leave considerable room for uncertainty about the effects of attrition on the outcome measures. Although these uncertainties are troubling, the pane} points out that some important crucial outcome measures e.g., the proportion of clients that return for HIV test results and subsequent stages of the protocolcan be known with certainty even in an anonymous testing program. Although the pane} recommends that a skilled evaluation team care- fully design and conduct randomized studies, the evaluation team and the sponsoring agencies should be prepared for a certain number of failures in carrying out experiments. In the panel's option, a failure rate of 20 percent or more should not be surprising. ~ the event of a failure (e.g., the contamination of individuals at the point of their selection, substantial attrition from the study) it will be useful to have a faIl-back position. For example, if the randomization of individuals fails, a randomized ex- penment might still be conducted at the clinic level. Such a redesigned experiment may not be as "clean" as one that uses individual participants, but it might still provide useful information. The panel's experience with counseling and testing sites suggests that sites will be willing to cooperate In experimental studies if they have substantial involvement in the implementation of new approaches and of the evaluation strategy. Most administrators are sophisticated enough to understand the need for evaluation in the interest of improving interventions, even if they do not necessarily understand the statistical aspects of such studies.
HIV TESTING AND COUNSELING ~ 121 Resources and Aspirations Three types of personnel resources will be beneficial to conduct ran- domized field experiments of testing and counseling interventions: (~) a qualified team of behavioral, social, and statistical scientists to design and conduct the studies; (2) an independent scientific oversight group for quality control (as recommended in Chapter 1~; and (3) appropriately trained CDC staff to design and conduct the studies and to monitor stud- ies undertaken by outside groups (also as recommended above. The first type of personnel, the individual investigator or investigative team, would be responsible for developing and implementing interventions. A un~versity-based or non-public health system contractor might be a good choice; however, because some elements of the evaluation of HIV testing and counseling will require the direct involvement of the public health system and community-based organizations, personnel from these settings should not be excluded from consideration. The oversight group should be an academic or other scientific re- search agency team that is independent of the project investigator or investigators. (Appendix B descnbes one such oversight approach, the "Project Review Team.") This oversight body can be used at the outset to facilitate consensus on evaluation protocols and to approve or develop outcome measures. Further along in the evaluation process, it can pro- vice strong, centralized oversight and quality control of the work. Past experience with large-scale, decentralized social research and evaluation programs indicates that without vigorous oversight the research may be of poor quality. This problem seems to occur for a variety of reasons, including a lack of coordination together with the inherent difficulties of conducting methodologically rigorous research In the context of a social action program (see Betsey, HoDister, and Papageorgiou, 1985~. Finally, evaluation studies that are cattier out as randomized trials by outside experts will require appropriately trained CDC staff to interact win the investigators and to interpret study results. In addition, CDC's personnel expertise and workload should also permit staff to conduct evaluation studies themselves. The types of staff needed for such tasks are behavioral, social, and statistical scientists trained In evaluation research (see Chapter I). \2 Still a fourth resource for a sponsoring agency would be an interagency coordinating body to draw upon He expertise of the federal agencies Hat are knowledgeable in relevant areas (e.g., CDC, the National Institute of Mental Health (NIMH), the National Institute on Drug Abuse (NINA), etc.). A body of this kind could draw on one agency's expertise for instance, that of NIDA in the field of HIV prevention with IV drug usersand on CDC's expertise in providing services. Together, such a body could facilitate the development of creative interventions that are theoretically based and that could then be empirically tested.
122 ~ EVALUATING AIDS PREVENTION PROGRAMS In addition to personnel and funds, conducting experiments requires time. Results will not be available quickly; indeed, when positive effects are found, they will need to be measured again at intervals to guard against erosion. Consequently, investigators and policy makers alike must find the patience needed to calTy out such research and a commitment to using evaluation research as a too] for the Tong-term improvement of testing and counseling programs. The pane} also notes that advances In the treatment of asymptomatic HIV-infected individuals may increase the demand for testing and coun- sel~ng services. Recent research suggests that some early treatments of persons infected with HIV may postpone the onset of AIDS and decrease morbidity and mortality from opportunistic infections. Thus, there may be a substantial increase in the demand for HIV testingas well as for medical monitoring of seropositive persons. ~ a rapidly changing envi- ronment, an ongoing program of evaluations will be essential to assess progress toward both the goal of adequate service delivery and He goal of reduced HIV transmission. ~ keeping with its general recommenda- tions, the pane} urges that evaluation of HIV testing and counseling be an ongoing activity and that selected projects be reevaluated periodically to monitor their continued effectiveness. REFERENCES Betsey, C. L., Hollister, R. G., and Papageorgiou, M. R., eds. (1985) Youth Employment and Training Programs: The YEDPA Years. Report of the ARC Committee on Youth Employment Programs. Washington, D.C.: National Academy Press. Bradbum, N. M., and Sudman, S. (1979) Improving Interview Method and Questionnaire Design. San Francisco: Jossey Bass. Centers for Disease Control (CDC) (1986) Additional recommendations to reduce sexual and dog abuse-related transmissional human T-lymphotropic virus type m/lymphadenopathy-associated virus. Morbidity and Mortality Weekly Report 35:152-155. Centers for Disease Control (CDC) (1987) Public Health Service guidelines for testing and counseling to prevent HIV infection and AIDS. Morbidity awl Mortality Weekly Report 36:509-514. Centers for Disease Control (CDC) (1988) Announcement No. 901. Federal Register 53~182~:36492-36493. September 20. Centers for Disease Control (CDC) (1989) AIDS Community Demonstration Projects Progress Report 1989. Centers for Disease Control, Atlanta, Ga. Centers for Disease Control (CDC) (n.d.) Prototype: Process Performance Standards for Personnel Performing HIV Disease Intervention (Counseling and Partner Notification). Centers for Disease Control, Atlanta, Ga. Coates, T. J., Morin, S. F., and McKusick, L. (1987) Behaviom1 consequences of AIDS antibody testing among gay men. Journal of the American Medical Association 258:1889.
HIV TESTING AND COUNSELING | 123 Fitti, J. E. (1989) AIDS knowledge and attitudes for September, 1988. Advance Data. No. 164. DHHS Pub. No. (PHS) 89-1250. January 3. Institute of Medicine/National Academy of Sciences (IOM/NAS) (1988) Confronting AIDS: Update 1988. Committee for the Oversight of AIDS Activities. Washington, D.C.: National Academy Press. Presidential Commission on the Human Immunodeficiency Virus Epidemic (1988) Final Report of the Presidential Commission on the Human Immunodeficiency Virus Epidemic. Washington, D.C.: U.S. Government Printing Office. Rugg, D., Sweet, D., Hovell, M., and Pagan, R. (1988) Factors Affecting the Decision to Learn HIV Test Results. Paper presented at He Fourth International AIDS Conference, Stockholm, June 12-16. Smith, T. (1984) Nonattitudes: A review and evaluation. In Turner, C. F., and E. Martin, eds. Surveying Subjective Phenomena. Vol. 2. New York: Russell Sage Foundation. Turner, C. F., and Martin, E., eds. (1984) Surveying Subjective Phenomena. 2 vole. New York: Russell Sage Foundation. Turner, C. F., Miller, H. G., and Moses, L. E., eds. (1989) AIDS, Sexual Behavior and Intravenous Drug Use. Report of the NRC Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences. Washington, D.C.: National Academy Press.