1
Introduction

This is an especially opportune time to consider current practices and future prospects for the evaluation of criminal justice programs. In recent years there have been increased calls from policy makers for “evidence-based practice” in health and human services that have extended to criminal justice as, for example, in the joint initiative of the Office of Justice Programs and the Coalition for Evidence-Based Policy on evidence-based crime and substance-abuse policy.1 This trend has been accompanied by various organized attempts to use the findings of evaluation research to determine “what works” in criminal justice. The Maryland Report (Sherman et al., 1997) responded to a request by Congress to review existing research and identify effective programs and practices. The Crime and Justice Group of the Campbell Collaboration has embarked on an ambitious effort to develop systematic reviews of research on the effectiveness of crime and justice programs. The OJJDP Blueprints for Violence Prevention project identifies programs whose effectiveness is demonstrated by evaluation research and other lists of programs alleged to be effective on the basis of research have proliferated (e.g., the National Registry of Effective Programs sponsored by the Substance Abuse and Mental Health Services Administration). In addition, the National Research



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 7
Improving Evaluation of Anticrime Programs 1 Introduction This is an especially opportune time to consider current practices and future prospects for the evaluation of criminal justice programs. In recent years there have been increased calls from policy makers for “evidence-based practice” in health and human services that have extended to criminal justice as, for example, in the joint initiative of the Office of Justice Programs and the Coalition for Evidence-Based Policy on evidence-based crime and substance-abuse policy.1 This trend has been accompanied by various organized attempts to use the findings of evaluation research to determine “what works” in criminal justice. The Maryland Report (Sherman et al., 1997) responded to a request by Congress to review existing research and identify effective programs and practices. The Crime and Justice Group of the Campbell Collaboration has embarked on an ambitious effort to develop systematic reviews of research on the effectiveness of crime and justice programs. The OJJDP Blueprints for Violence Prevention project identifies programs whose effectiveness is demonstrated by evaluation research and other lists of programs alleged to be effective on the basis of research have proliferated (e.g., the National Registry of Effective Programs sponsored by the Substance Abuse and Mental Health Services Administration). In addition, the National Research 1   Available: http://www.excelgov.org/displayContent.asp?Keyword=prppcPrevent.

OCR for page 7
Improving Evaluation of Anticrime Programs Council’s (NRC) Committee on Law and Justice has been commissioned to prepare reports assessing research evidence on such topics as the effectiveness of policing policies (NRC, 2004), firearms policies (NRC, 2005), illicit drug policies (NRC, 2001), and the prevention, treatment, and control of juvenile crime (NRC and Institute of Medicine, 2001). These developments reflect recognition that effective guidance of criminal justice policy and practice requires evidence about the effects of those policies and practices on the populations and conditions they are intended to influence. For example, knowledge of the ability of various programs to reduce crime or protect potential victims allows resources to be allocated in ways that support effective programs and efficiently promote these outcomes. The role of evaluation research is to provide evidence about these kinds of program effects and to do so in a manner that is accessible and informative to policy makers. Fulfilling that function, in turn, requires that evaluation research be designed and implemented in a manner that provides valid and useful results of sufficient quality to be relied upon by policy makers. In this context especially, significant methodological shortcomings would seriously compromise the value of evaluation research. And, it is methodological issues that are at the heart of what has arguably been the most influential stimulus for attention to the current state of evaluation research in criminal justice. A series of reports2 by the U.S. General Accounting Office has been sharply critical of the evaluation studies conducted under the auspices of the Department of Justice. Because several offices within the Department of Justice are major funders of evaluation research on criminal justice programs, especially the larger and more influential evaluation projects, this is a matter of concern not only to the Department of Justice, but to others who conduct and sponsor criminal justice evaluation research. CRITICISMS OF METHOD The GAO reports focus on impact evaluation, that is, assessment of the effects of programs on the populations or conditions they are intended 2   Juvenile Justice: OJJDP Reporting Requirements for Discretionary and Formula Grantees and Concerns About Evaluation Studies (GAO, 2001). Drug Courts: Better DOJ Data Collection and Evaluation Efforts Needed to Measure Impact of Drug Court Programs (GAO, 2002a). Justice Impact Evaluations: One Byrne Evaluation Was Rigorous; All Reviewed Violence Against Women Office Evaluations Were Problematic (GAO, 2002b). Violence Against Women Office: Problems with Grant Monitoring and Concerns About Evaluation Studies (GAO, 2002c). Justice Outcome Evaluations: Design and Implementation of Studies Require More NIJ Attention (GAO, 2003a). Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity (GAO, 2003b).

OCR for page 7
Improving Evaluation of Anticrime Programs to change. The impact evaluations selected for review cover a wide range of programs, most of which are directed toward a particular criminal justice problem or population and implemented in multiple sites (see Box 1-1). As such, these programs are relatively representative of the kinds of initiatives that a major funder of criminal justice programs might support and wish to evaluate for impact. The GAO review of the design and implementation of the impact evaluations for these programs identified a number of problem areas that highlight the major challenges that must be met in a sound impact evaluation. These generally fell into two categories: (a) deficiencies in the evaluation design and procedures that were initially proposed and (b) difficulties implementing the evaluation plan. It is indicative of the magnitude of the challenge posed by impact evaluation at this scale that, of the 30 evaluations for the programs shown in Box 1-1, one or both of these problems were noted for 20 of them, and some of the remaining 10 were still in the proposal stage and had not yet been implemented. The most frequent deficiencies in the initial plan or the implementation of the evaluation identified in the GAO reviews were as follows: The sites selected to participate in the evaluation were not representative of the sites that had received the program. The program participants selected at the evaluation sites were not representative of the population the program served. Pre-program baseline data on key outcome variables were not included in the design or could not be collected as planned so that change over time could not be assessed. The intended program outcomes (e.g., reduced criminal activity, drug use, or victimization in contrast to intermediate outcomes such as increases in knowledge) were not measured or outcome measures with doubtful reliability and validity were used. No means for isolating program effects from the influence of external factors on the outcomes, such as a nonparticipant comparison group or appropriate statistical controls, were included in the design or the planned procedure could not be implemented. The program and comparison groups differed on outcome-related characteristics at the beginning of the program or became different due to differential attrition before the outcomes were measured. Data collection was problematic; needed data could not be obtained or response rates were low when it was likely that those who responded differed from those who did not. No recent review of evaluation research in the general criminal justice literature provides an assessment of methodology that is as compre-

OCR for page 7
Improving Evaluation of Anticrime Programs BOX 1-1 Programs Represented in the Impact Evaluation Plans and Projects Reviewed in Recent GAO Reports Arrest Policies Program (treating domestic violence as a serious violation of law) Breaking the Cycle (comprehensive service for adult offenders with drug-use histories) Chicago’s Citywide Community Policing Program (policing organized around small geographic areas) Children at Risk Program (comprehensive services for high-risk youth) Comprehensive Gang Initiative (community-based program to reduce gang-related crime) Comprehensive Service-Based Intervention Strategy in Public Housing (program to reduce drug activity and crime) Corrections and Law Enforcement Family Support (CLEFS) (stress intervention programs for law enforcement officers and families) Court Monitoring and Batterer Intervention Programs (batterer counseling programs and court monitoring) Culturally Focused Batterer Counseling for African-American Men Domestic Violence Victims’ Civil Legal Assistance Program (legal services for victims of domestic violence) Drug Courts (specialized court procedures and services for drug offenders) Enforcement of Underage Drinking Laws Program Gang Resistance Education and Training (GREAT) Program (school-based gang prevention program) Intensive Aftercare (programs for juvenile offenders after release from confinement) Juvenile Justice Mental Health Initiative (mental health services to families hensive as that represented in the collection of GAO reports summarized above. What does appear in that literature in recent years is considerable discussion of the role and applicability of randomized field experiments for investigating program effects. In Feder and Boruch (2000), a special issue of Crime and Delinquency was devoted to the potential for experiments in criminal justice settings, followed a few years later by a special issue (Weisburd, 2003) of Evaluation Review on randomized trials in criminology. More recently, a new journal, Experimental Criminology, was launched with an explicit focus on experimental and quasi-experimental research for investigating crime and justice practice and policy. The view that research on the effects of criminal justice interventions would be improved by greater emphasis on randomized experiments, however, is by

OCR for page 7
Improving Evaluation of Anticrime Programs of delinquent youths with serious emotional disturbances) Juvenile Mentoring Program (volunteer adult mentors for at-risk youth) Multi-Site Demonstration for Enhanced Judicial Oversight of Domestic Violence Cases (coordinated response to domestic violence offenses) Multi-Site Demonstration of Collaborations to Address Domestic Violence and Child Maltreatment (community-based programs for coordinated response to families with co-occurring domestic violence and child maltreatment) Parents Anonymous (support groups for child abuse prevention) Partnership to Reduce Juvenile Gun Violence Program (coordinated community strategies for selected areas in cities) Project PATHE (school-based violence prevention) Reducing Non-Emergency Calls to 911: Four Approaches Responding to the Problem Police Officer: Early Warning Systems (identification and treatment for officers whose behavior is problematic) Rural Domestic Violence and Child Victimization Enforcement Grant Program (coordinated strategies for responding to domestic violence) Rural Domestic Violence and Child Victimization Grant Program (cooperative community-based efforts to reduce domestic violence, dating violence, and child abuse) Rural Gang Initiative (community-based gang prevention programs) Safe Schools/Healthy Students (school services to promote healthy development and prevent violence and drug abuse) Safe Start Initiative (integrated service delivery to reduce impact of family and community violence on young children) STOP Grant Programs (culture-specific strategies to reduce violence against Indian women) Victim Advocacy with a Team Approach (domestic violence teams to assist victims) no means universal. The limitations of experimental methods for such purposes and alternatives using econometric modeling have also received critical attention (e.g., Heckman and Robb, 1985; Manski, 1996). OVERVIEW OF THE WORKSHOP AND THIS REPORT In the context of these various concerns about evaluation methods and quality, the National Institute of Justice asked the NRC Committee on Law and Justice to organize a workshop on improving the evaluation of criminal justice programs and to follow up with a report that extracted guidance for effective evaluation practices from those proceedings. The Academies appointed a small steering committee to guide workshop de-

OCR for page 7
Improving Evaluation of Anticrime Programs velopment. The workshop was held in September 2003, and this report is the result of the efforts of the steering committee to further develop the themes raised there and integrate them as constructive advice about conducting evaluations of criminal justice programs. The purpose of the Workshop on Improving the Evaluation of Criminal Justice Programs was to foster broader implementation of credible evaluations in the field of criminal justice by promoting informed discussion of: the repertoire of applicable evaluation methods; issues in matching methods to program and policy circumstances; and the organizational infrastructure requirements for supporting sound evaluation. This purpose was pursued through presentation and discussion of case examples of evaluation-related studies selected to represent the methods and challenges associated with research at each of three different levels of intervention. The three levels are distinguished by different social units that are the target of intervention and thus constitute the units of analysis for the evaluation design. The levels and the exemplary evaluation studies and assigned discussant for each were as follows: Interventions directed toward individuals, a situation in which there are generally a relatively large number of units within the scope of the program being evaluated and potential for assigning those units to different intervention conditions. Multidimensional Family Foster Care (Patricia Chamberlain) A Randomized Experiment: Testing Inmate Classification Systems (Richard Berk) Discussant (Adele Harrell) Interventions with neighborhoods, schools, prisons, or communities, a situation generally characterized by relatively few units within the scope of the program and often limited potential for assigning those units to different intervention conditions. Hot Spots Policing and Crime Prevention (Anthony Braga) Communities Mobilizing for Change (Alex Wagenaar) Discussant (Edward Zigler)

OCR for page 7
Improving Evaluation of Anticrime Programs Interventions at the broad local, state, or national level where the program scope encompasses a macro unit and there is virtually no potential for assigning units to different intervention conditions. An Empirical Analysis of LOJACK (Steven Levitt) Racial Bias in Motor Vehicle Searches (Petra Todd) Discussant (John V. Pepper) After the research case studies in each category were presented, their implications for conducting high-quality evaluations were discussed. A final panel at the end of the workshop then discussed the infrastructure requirements for strong evaluations. Infrastructure Requirements for Consumption (and Production) of Strong Evaluations (Lawrence Sherman) Recommendations for Evaluation (Robert Moffitt) Bringing Evidence-Based Policy to Substance Abuse and Criminal Justice (Jon Baron) Papers presented at the workshop are provided on the Committee on Law and Justice Website at http://www7.nationalacademies.org/claj/. The intent of this report is not to summarize the workshop but, rather, to draw upon its contents to highlight the major considerations in developing and implementing evaluation plans for criminal justice programs. In particular, the report is organized around five interrelated questions that require thoughtful analysis in the development of any evaluation plan, with particular emphasis on impact evaluation: What questions should the evaluation address? When is it appropriate to conduct an impact evaluation? How should an impact evaluation be designed? How should the evaluation be implemented? What organizational infrastructure and procedures support high-quality evaluation? In the pages that follow, each of these questions is examined and advice is distilled from the workshop presentations and discussion, and from subsequent committee deliberations, for answering them in ways that will help improve the evaluation of criminal justice programs. The intended audience for this report includes NIJ, the workshop sponsor and a major funder of criminal justice evaluations, but also other federal, state, and local agencies, foundations, and other such organizations that plan, sponsor, or administer evaluations of criminal justice programs.