Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Evaluating AIDS Prevention Programs Expanded Edition Susan L. Coyle, Robert F. Boruch, and Charles F. Turner, Editors PANEL ON THE EVALUATION OF AIDS INTERVENTIONS COMMITTEE ON AIDS RESEARCH AND THE BEHAVIORAL, SOCIAL, AND STATISTICAL SCIENCES COMMISSION ON THE BEHAVIORAL AND SOCIAL SCIENCES AND EDUCATION NATIONAL RESEARCH COUNCIL NATIONAL ACADEMY PRESS WASHINGTON, D.C. 1991
NATIONAL ACADEMY PRESS · 2101 Constitution Avenue, N.W. · Washington, D.C. · 20418 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Frank Press is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Robert M. White is president of the National Academy of Engineering. The Lnstitute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and upon its own initiative, to identify issues of medical care, research, and education. Dr. Samuel O. Thier is president of the Institute of Medicine. The National Research Council was established by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and of advising the federal government. Functioning in accordance with general policies determined by He Academy, the Council has become the principal operating agency of bow the National Academy of Sciences and the National Academy of Engineenng in providing services to the govemment, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Frank Press and Dr. Robert M. White are chairman and vice-chauman, respectively, of the National Research Council. The work that provided the basis for this volume was supported by a conuact from He U.S. Public Heals Service Library of Congress Cataloging-in-Publication Data Evaluating AIDS prevention programs I Susan L. Coyle, Robert F. Boruch, and Charles F. Turner, editors; Panel on the Evaluation of AIDS Interventions, Committee on AIDS Research and the Behavioral, Social and Statistical Sciences, National Research Council. - Expanded em p. cm. Includes bibliographical references and index. ISBN ~309 (}4281-X (paper). - ISBN ~309 04288-7 (clod) 1. AIDS (Disease - Prevention-Evaluation-Methodology. I. Coyle, Susan L. II. Boruch, Robert F. m. burner, Charles F. IV. mile. [DNLM: 1. Acquired Immunodeficiency Syndrome-prevention & control. 2. Program Evaluation. WD 308 N27725e] RA644.A25N27 1991 614.5'993~c20 DNLMJDLC for Library of Congress Copyright (~)1991 by the National Academy of Sciences Printed in the United States of America First Printing, January 1991 Second Printing, December 1993 9~13649 CIP
Panel on the Evaluation of AIDS Interventions ROBERT F. BORUCH (Chairs, Graduate School of Education and Department of Statistics, University of Pennsylvania THOMAS J. COATES (Liaison to Committee), Division of General Internal Medicine and Center for AIDS Prevention Studies, University of California at San Francisco JEFFREY P. DAVIS, Bureau of Community Health and Prevention, Wisconsin Division of Health BRIAN R. FLAY, Prevention Research Center, University of Illinois at Chicago ROBERT L. HUBBARD, Center for Social Research and Policy Analysis, Research Triangle Institute RONALD C. KESSLER,* Institute for Social Research, University of Michigan NAN M. LAIRD, Department of Biostatistics, Haward University School of Public Health JOHN L. MARTIN, School of Public Health, Columbia University LINCOLN E. MOSES, Department of Statistics, Stanford University JESSICA M. UTTS, Division of Statistics, University of California at Davis JOHN B. WAILER, JR., Department of Community Medicine, Wayne State University School of Medicine National Research Council Staff TRACY L. BRANDT, Research Assistant, Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences SUSAN L. COYLE, SIUdY Director, Panel on the Evaluation of AIDS Interventions KIRSTEN J. JOHNSON, Sr. Project Assistant, Panel on the Evaluation of AIDS Interventions CHARLES F. TURNER, Director, Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences * Served through Mary 31, 1990. . . .
Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences MARSHALL H. BECKER, School of Public Health, University of Michigan ROBERT F. BORUCH, Graduate School of Education and Depamnent of Statistics, University of Pennsylvania THOMAS J. COATES, Division of General Internal Medicine, and Center for AIDS Prevention Studies, University of California at San Francisco RAMON C. CORTINES,* Superintendent of Schools, San Francisco Unified School District ROBYN M. DAWES, Department of Social and Decision Sciences, Carnegie Mellon University DON C. DES JARLAIS (Vice ChairJ, Chemical Dependency Institute, Beth Israel Medical Center, New York, and Mount Sinai School of Medicine JOHN H. GAGNON, Department of Sociology, State University of New York at Stony Brook ALBERT R. JONSEN, Department of Medical History and Ethics, University of Washington at Seattle SHIRLEY LINDENBAUM, Department of Anthropology, Graduate Center, City University of New York JANE MENKEN (Chair), Department of Sociology, and Population Studies Center, University of Pennsylvanua LINCOLN E. MOSES, Department of Statistics, Stanford University CLADD E. STEVENS, The New York Blood Center BAILUS WALKER, School of Public Health, University of Oklahoma National Research Council Staff TRACY L. BRANDT, Research Assistant, Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences SUSAN L. COYLE, Study Director, Panel on He Evaluation of AIDS Interventions KIRSTEN J. JOHNSON, Sr. Project Assistant, Panel on the Evaluation of AIDS Interventions HEATHER G. MILLER,! Study Director, Panel on AIDS Interventions and Research KAREN E. ORLANDO,$ Sr. Project Assistant, Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences JEFFREY S. STRYKER, Study Director, Panel on Monitoring the Social Impact of He AIDS Epidemic CHARI,ES F. TURNER, Director, Committee on AIDS Research and tile Behavioral, Social, and Statistical Sciences * Served through September 1, 1989. ~ Served through July 1, 1990. ~ Served through September 1, 1990. 1V
Liaison Representatives to the Pane! from the U.S. Public Health Service ZILI AMSEL, National Institute on Drug Abuse G. STEPHEN BOWEN, Centers for Disease Control DAVID BROWNELL, Centers for Disease Control ISABEL FERNANDEZ, Centers for Disease Control JACOB GAYLE, Centers for Disease Control JANINE JASON, Centers for Disease Control C. FREDERIC KROGER, Centers for Disease Control SUSAN MIDDLESTADT, Centers for Disease Control KEVIN O 'REILLY, Centers for Disease Control DEBORAH L. RUGG, Centers for Disease Control ELLEN STOVER, National Institute of Mental Health RONALD W. WILSON, National Center for Heals Statistics, Centers for Disease Control Consultants MICHAEL L. COHEN, School of Public Affairs, University of Maryland SAHR J. KPUNDEH, Depamnent of Political Science, Howard University LEAH MAZADE, National Academy Press LAURA RUDKIN-MINIOT, Department of Sociology, Princeton University MICHAEL A. STOTO, Division of Heals Promotion and Disease Prevention, Institute of Medicine v
Preface As the second decade of the epidemic of acquired immune deficiency syndrome (AIDS) begins, behavioral change remains our primary weapon in the fight against the spread of the human immunodeficiency virus (HIV) and AIDS. It is unfortunate, however, that we know relatively little as scientists about the comparative effectiveness of different strategies intended to encourage and facilitate protective behavioral change among persons who are at risk of HIV transmission. It is our strong belief that such knowledge is an urgent national priority and that it will require a long-term commitment to a course of rigorous (and selective) research assessing current and future AIDS prevention programs. Such research will enable us to learn from both our failures and our successes in inducing protective changes in sexual and ~ug-using behaviors. The present report provides a bluepnnt for action. It has been under- takenat the behest of He U.S. Public Health Service, which requested assistance in developing strategies for evaluating the three major AIDS prevention programs sponsored by the Centers for Disease Control. This reports was prepared by the Pane] on the Evaluation of AIDS ~terven- lions, which was convened by the National Research Council Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences. The panel's work extends the committee's previous assessment of ~nter- vention needs, reported in AIDS, Sexual Behavior, and Intravenous Drug Use (Turner, Miller, and Moses, 1989~. That report argued strongly for a national commitment to the careful design and evaluation of AIDS intervention strategies. We recognize Mat the high quality evaluation research recommended in this report will be costly, but we believe that uncertainty about program effectiveness is even more costly. The price of continued ignorance will be measured both in dollars spent on ineffective programs and in deaths and disease that might have been prevented. A commitment to 1 The present volume is an expanded edition of the Panel's 1989 report. This edition incorporates new sections on nonexperimental approaches to intervention (Chapter 6 and Appendix F) and methodolog- ical issues in mounting evaluations (Appendixes C-E). . . V11
Viii ~ PR EFACE well-conducted evaluation research must be made now, and it must be maintained if we are to improve our understanding of how to control this deadly epidemic. There is good reason to believe that this nation can and will do a better job in the future in determining which AIDS prevention programs do change behavior and which do not. Federal officials and people who have been on the front lines of AIDS prevention activities since the early days of the epidemic all express a considerable desire for evaluation of their work. The paucity of rigorous evaluations conducted to date is not due to individuals' unwillingness to undertake the research but to a lack of resources and expertise. It is evident Hat past links between those who provide services and manage programs, on the one hand, and those who conduct research and evaluation, on the over, have not been strong. It is our hope that this volume win foster such collaboration and thereby advance our understanding of and ability to facilitate the behav- ioral changes required to retard Be spread of REV in the fixture. SUSAN L. COYLE, STUDY Director ROBERT F. BORUCH, Chair Pane} on the Evaluation of AIDS Interventions CHARLES F. TURNER, Director Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences
Acknowledgments Duling the course of this study, the panel was assisted by a number of scientists who took time to share their insights and expertise. To those who assisted us in our work, the pane} extends its sincere tiaras and appreciation. The panel's work was supported by the U.S. Public Health Service. Note on Contributions This report is the collective product of pane! members and staff. The content of this report reflects the deliberations of the panel, and the report presents the panel's recommendations. The list below identifies the persons who shared major responsibility for preparing initial drafts of materials for each chapter in this report. The panel reviewed all contributions, and they have been revised and edited in light of the panel's discussions and the comments of outside reviewers. The purpose of the following alphabetical list, therefore, is to give credit to individuals but not to assign final responsibility for the published text. It should also be noted that, although the list covers major sections of this volume, these sections frequently contain additional paragraphs or pages from over hands. SUMMARY: Coyle, Turner CHAPTER I: Boruch, Coates, Coyle, Laird, Moses, Turner CHAPTER 2: Margin, win Coyle CHAPTER 3: Flay and Kessler, with Utts CHAPTER 4: Boruch, Hubbard, Turner, Wailer CHAPTER 5: Boruch, Coates, Coyle, Davis, Hubbard, Martin, Turner CHAPTER 6: Coyle, Moses, and Turner, with Boruch, Cohen, Hubbard, Laird APPENDIX A: Waller APPENDIX B: Boruch APPENDIX C: repented from Miller, Turner, and Moses (1990), contributorsS. L`indenbaum, J. Sorensen, C. Turner, J. Wiley, with P. Blumstein, R. Dawes, L. Rub Moot 1X
APPENDIX D: Coyly APPENDIX E: Coyly APPENDIX F congas a bound pier Spud far He panel by Rood bomb; this per is Be sole ~sponsibUi~ of ilS =~0L
Contents SUMMARY .... ·.! 1 DESIGN AND IMPLEMENTATION OF EVALUATION RESEARCH 15 Awes of Evaluation, 16 Evaluation Research Design, 18 The Management of Evaluation, 27 References, 32 2 MEASUREMENT OF OUTCOMES.. Program Objectives, 34 Outcomes for Evaluations of HIV Prevention Programs, 37 Evaluation Measures, 45 References, 48 3 EVALUATING MEDIA CAMPAIGNS.. Background and Objectives, 51 Formative Evaluation: What works better? 53 Efficacy Tnals: Can Me campaign make a difference? 61 Process Evaluation: What is actually delivered? 66 Outcome Evaluation: Does the campaign make a difference? 71 References, 81 4 EVALUATING HEALTH EDUCATION AND RISK REDUCTION PROJECTS Background and Objectives, 83 What Services Are Delivered? 86 Do the Projects Make a Difference? 93 What Works Better? 99 References, 100 X1 ....34 . .50 ..... 83
Xii ~ CONTENTS 5 EVALUATING HIV TESTING AND COUNSELING PROJECTS ....................................... Background and Objectives, 102 How Well Are Services Delivered? 103 Options for Evaluating Whether HIV Testing and Counseling Services Make a Difference, 114 What Works Better? 116 References, 122 6 RANDOMIZED AND OBSERVATIONAL APPROACHES TO EVALUATING THE EFFECTIVENESS OF AIDS PREVENTION PROGRAMS............. Overview, 125 Randomized Experimentation, 128 Designing Comparability into Nonrandomized Studies, 144 Modeling and Statistical Adjustments for Bias, 168 When Should Nonrandomized Approaches Be Considered? 182 Interpreting Evaluation Results, 186 References, 189 Appendixes A Collaborative Contracting Strategy ................... B Oversight and Coordination Strategy ........................ C Methodological Issues in AIDS Surveys ................... D Sampling and Randomization: Technical Questions about Evaluating CDC's Three Major AIDS Prevention Programs E Ancillaly, Emerging, and Related Projects .... F The Use of Selection Models to Evaluate AIDS Interventions with Observational Data ......................... INDEX ...... ...... 102 124 ........ 197 .200 .207 . . .317 335 342 .365
Evaluating AIDS Prevention Programs Expanded Edition