Improving Evaluation of Anticrime Programs (2005)

This is an especially opportune time to consider current practices and future prospects for the evaluation of criminal justice programs. In recent years there have been increased calls from policy makers for “evidence-based practice” in health and human services that have extended to criminal justice as, for example, in the joint initiative of the Office of Justice Programs and the Coalition for Evidence-Based Policy on evidence-based crime and substance-abuse policy.¹ This trend has been accompanied by various organized attempts to use the findings of evaluation research to determine “what works” in criminal justice. The Maryland Report (Sherman et al., 1997) responded to a request by Congress to review existing research and identify effective programs and practices. The Crime and Justice Group of the Campbell Collaboration has embarked on an ambitious effort to develop systematic reviews of research on the effectiveness of crime and justice programs. The OJJDP Blueprints for Violence Prevention project identifies programs whose effectiveness is demonstrated by evaluation research and other lists of programs alleged to be effective on the basis of research have proliferated (e.g., the National Registry of Effective Programs sponsored by the Substance Abuse and Mental Health Services Administration). In addition, the National Research

Council’s (NRC) Committee on Law and Justice has been commissioned to prepare reports assessing research evidence on such topics as the effectiveness of policing policies (NRC, 2004), firearms policies (NRC, 2005), illicit drug policies (NRC, 2001), and the prevention, treatment, and control of juvenile crime (NRC and Institute of Medicine, 2001).

These developments reflect recognition that effective guidance of criminal justice policy and practice requires evidence about the effects of those policies and practices on the populations and conditions they are intended to influence. For example, knowledge of the ability of various programs to reduce crime or protect potential victims allows resources to be allocated in ways that support effective programs and efficiently promote these outcomes. The role of evaluation research is to provide evidence about these kinds of program effects and to do so in a manner that is accessible and informative to policy makers. Fulfilling that function, in turn, requires that evaluation research be designed and implemented in a manner that provides valid and useful results of sufficient quality to be relied upon by policy makers.

In this context especially, significant methodological shortcomings would seriously compromise the value of evaluation research. And, it is methodological issues that are at the heart of what has arguably been the most influential stimulus for attention to the current state of evaluation research in criminal justice. A series of reports² by the U.S. General Accounting Office has been sharply critical of the evaluation studies conducted under the auspices of the Department of Justice. Because several offices within the Department of Justice are major funders of evaluation research on criminal justice programs, especially the larger and more influential evaluation projects, this is a matter of concern not only to the Department of Justice, but to others who conduct and sponsor criminal justice evaluation research.

The GAO reports focus on impact evaluation, that is, assessment of the effects of programs on the populations or conditions they are intended

to change. The impact evaluations selected for review cover a wide range of programs, most of which are directed toward a particular criminal justice problem or population and implemented in multiple sites (see Box 1-1). As such, these programs are relatively representative of the kinds of initiatives that a major funder of criminal justice programs might support and wish to evaluate for impact.

The GAO review of the design and implementation of the impact evaluations for these programs identified a number of problem areas that highlight the major challenges that must be met in a sound impact evaluation. These generally fell into two categories: (a) deficiencies in the evaluation design and procedures that were initially proposed and (b) difficulties implementing the evaluation plan. It is indicative of the magnitude of the challenge posed by impact evaluation at this scale that, of the 30 evaluations for the programs shown in Box 1-1, one or both of these problems were noted for 20 of them, and some of the remaining 10 were still in the proposal stage and had not yet been implemented.

The most frequent deficiencies in the initial plan or the implementation of the evaluation identified in the GAO reviews were as follows:

• The sites selected to participate in the evaluation were not representative of the sites that had received the program.

• The program participants selected at the evaluation sites were not representative of the population the program served.

• Pre-program baseline data on key outcome variables were not included in the design or could not be collected as planned so that change over time could not be assessed.

• The intended program outcomes (e.g., reduced criminal activity, drug use, or victimization in contrast to intermediate outcomes such as increases in knowledge) were not measured or outcome measures with doubtful reliability and validity were used.

• No means for isolating program effects from the influence of external factors on the outcomes, such as a nonparticipant comparison group or appropriate statistical controls, were included in the design or the planned procedure could not be implemented.

• The program and comparison groups differed on outcome-related characteristics at the beginning of the program or became different due to differential attrition before the outcomes were measured.

• Data collection was problematic; needed data could not be obtained or response rates were low when it was likely that those who responded differed from those who did not.

No recent review of evaluation research in the general criminal justice literature provides an assessment of methodology that is as compre-

hensive as that represented in the collection of GAO reports summarized above. What does appear in that literature in recent years is considerable discussion of the role and applicability of randomized field experiments for investigating program effects. In Feder and Boruch (2000), a special issue of Crime and Delinquency was devoted to the potential for experiments in criminal justice settings, followed a few years later by a special issue (Weisburd, 2003) of Evaluation Review on randomized trials in criminology. More recently, a new journal, Experimental Criminology, was launched with an explicit focus on experimental and quasi-experimental research for investigating crime and justice practice and policy. The view that research on the effects of criminal justice interventions would be improved by greater emphasis on randomized experiments, however, is by

no means universal. The limitations of experimental methods for such purposes and alternatives using econometric modeling have also received critical attention (e.g., Heckman and Robb, 1985; Manski, 1996).

In the context of these various concerns about evaluation methods and quality, the National Institute of Justice asked the NRC Committee on Law and Justice to organize a workshop on improving the evaluation of criminal justice programs and to follow up with a report that extracted guidance for effective evaluation practices from those proceedings. The Academies appointed a small steering committee to guide workshop de-

velopment. The workshop was held in September 2003, and this report is the result of the efforts of the steering committee to further develop the themes raised there and integrate them as constructive advice about conducting evaluations of criminal justice programs.

The purpose of the Workshop on Improving the Evaluation of Criminal Justice Programs was to foster broader implementation of credible evaluations in the field of criminal justice by promoting informed discussion of:

• the repertoire of applicable evaluation methods;

• issues in matching methods to program and policy circumstances; and

• the organizational infrastructure requirements for supporting sound evaluation.

This purpose was pursued through presentation and discussion of case examples of evaluation-related studies selected to represent the methods and challenges associated with research at each of three different levels of intervention. The three levels are distinguished by different social units that are the target of intervention and thus constitute the units of analysis for the evaluation design. The levels and the exemplary evaluation studies and assigned discussant for each were as follows:

1. Interventions directed toward individuals, a situation in which there are generally a relatively large number of units within the scope of the program being evaluated and potential for assigning those units to different intervention conditions.

   • Multidimensional Family Foster Care (Patricia Chamberlain)

   • A Randomized Experiment: Testing Inmate Classification Systems (Richard Berk)

   • Discussant (Adele Harrell)

2. Interventions with neighborhoods, schools, prisons, or communities, a situation generally characterized by relatively few units within the scope of the program and often limited potential for assigning those units to different intervention conditions.

   • Hot Spots Policing and Crime Prevention (Anthony Braga)

   • Communities Mobilizing for Change (Alex Wagenaar)

   • Discussant (Edward Zigler)

3. Interventions at the broad local, state, or national level where the program scope encompasses a macro unit and there is virtually no potential for assigning units to different intervention conditions.

   • An Empirical Analysis of LOJACK (Steven Levitt)

   • Racial Bias in Motor Vehicle Searches (Petra Todd)

   • Discussant (John V. Pepper)

After the research case studies in each category were presented, their implications for conducting high-quality evaluations were discussed. A final panel at the end of the workshop then discussed the infrastructure requirements for strong evaluations.

• Infrastructure Requirements for Consumption (and Production) of Strong Evaluations (Lawrence Sherman)

• Recommendations for Evaluation (Robert Moffitt)

• Bringing Evidence-Based Policy to Substance Abuse and Criminal Justice (Jon Baron)

Papers presented at the workshop are provided on the Committee on Law and Justice Website at http://www7.nationalacademies.org/claj/.

The intent of this report is not to summarize the workshop but, rather, to draw upon its contents to highlight the major considerations in developing and implementing evaluation plans for criminal justice programs. In particular, the report is organized around five interrelated questions that require thoughtful analysis in the development of any evaluation plan, with particular emphasis on impact evaluation:

1. What questions should the evaluation address?

2. When is it appropriate to conduct an impact evaluation?

3. How should an impact evaluation be designed?

4. How should the evaluation be implemented?

5. What organizational infrastructure and procedures support high-quality evaluation?

In the pages that follow, each of these questions is examined and advice is distilled from the workshop presentations and discussion, and from subsequent committee deliberations, for answering them in ways that will help improve the evaluation of criminal justice programs. The intended audience for this report includes NIJ, the workshop sponsor and a major funder of criminal justice evaluations, but also other federal, state, and local agencies, foundations, and other such organizations that plan, sponsor, or administer evaluations of criminal justice programs.