7
On the Use of Aggregate Crime Regressions in Policy Evaluation

Steven N. Durlauf, Salvador Navarro, and David A. Rivers


Despite recent efforts to employ microeconomic data and natural experiments, aggregate crime regressions continue to play a significant role in criminological analyses. One use of these regressions is predictive, as illustrated by the papers in this volume that employ aggregate crime trends regressions—Baumer (Chapter 5) and Pepper (Chapter 6). A second use involves policy evaluation: Prominent and controversial cases include the deterrent effect of shall-issue concealed weapons legislation (e.g., Ayres and Donohue, 2003; Black and Nagin, 1998; Lott, 1998; Lott and Mustard, 1997; Plassmann and Whitley, 2003) and the deterrent effect of capital punishment (e.g., Dezhbakhsh, Rubin, and Shepherd, 2003; Donohue and Wolfers, 2005). These uses are interrelated, as is evident from the effort to evaluate how changes in criminal justice policies explain the great reduction of crime in the 1990s.

The goal of this chapter is to examine the construction and interpretation of aggregate crime regressions. Specifically, we employ contemporary economic and econometric reasoning to understand how aggregate crime regressions may be appropriately used to inform positive and normative questions. While by no means comprehensive, we hope our discussion will prove useful in highlighting some of the limitations of the use of these regressions and in particular will indicate how empirical findings may be misinterpreted when careful attention is not given to the link between the aggregate data and individual behavior.1

1

The interpretation of aggregate data continues to be one of the most difficult questions in social science; Stoker (1993) and Blundell and Stoker (2005) provide valuable overviews.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 211
7 On the Use of Aggregate Crime Regressions in Policy Evaluation Steen N. Durlauf, Salador Naarro, and Daid A. Riers Despite recent efforts to employ microeconomic data and natural experiments, aggregate crime regressions continue to play a significant role in criminological analyses. One use of these regressions is predictive, as illustrated by the papers in this volume that employ aggregate crime trends regressions—Baumer (Chapter 5) and Pepper (Chapter 6). A second use involves policy evaluation: Prominent and controversial cases include the deterrent effect of shall-issue concealed weapons legislation (e.g., Ayres and Donohue, 2003; Black and Nagin, 1998; Lott, 1998; Lott and Mustard, 1997; Plassmann and Whitley, 2003) and the deterrent effect of capital punishment (e.g., Dezhbakhsh, Rubin, and Shepherd, 2003; Donohue and Wolfers, 2005). These uses are interrelated, as is evident from the effort to evaluate how changes in criminal justice policies explain the great reduction of crime in the 1990s. The goal of this chapter is to examine the construction and interpreta- tion of aggregate crime regressions. Specifically, we employ contemporary economic and econometric reasoning to understand how aggregate crime regressions may be appropriately used to inform positive and normative questions. While by no means comprehensive, we hope our discussion will prove useful in highlighting some of the limitations of the use of these regressions and in particular will indicate how empirical findings may be misinterpreted when careful attention is not given to the link between the aggregate data and individual behavior.1 1 The interpretation of aggregate data continues to be one of the most difficult questions in social science; Stoker (1993) and Blundell and Stoker (2005) provide valuable overviews. 

OCR for page 211
 UNDERSTANDING CRIME TRENDS The chapter is organized as follows. We begin by describing a stan- dard choice-based model of crime. We then discuss how this individual- level model can be aggregated to produce crime regressions of the type found in the literature. In the next three sections we discuss the analysis of counterfactuals, issues of model uncertainty in crime regressions, and the relationship between statistical models and policy evaluation. We then apply our general arguments to areas in the empirical criminology litera- ture: the convergence of crime rates, capital punishment, and shall-issue concealed weapons laws. The next section discusses whether the limitations that exist in using crime regressions mean that they should be replaced by quasi-experimental methods, and a final section concludes the chapter. Our discussion is conceptual; Durlauf, Navarro, and Rivers (2008) provide a more systematic treatment of many of the issues we raise as well as an empirical application. CRIME AS A CHOICE From the vantage point of economics, the fundamental idea underlying the analysis of crime is that each criminal act constitutes a purposeful choice on the part of the criminal. In turn, this means that the development of a theory of the aggregate crime rate should be explicitly understood as deriving from the aggregation of individual decisions. The basic logic of the economic approach to crime was originally developed by Gary Becker (1968) and extended by Isaac Ehrlich (1972, 1973). This logic underlies the renaissance of crime research in economics, exemplified in the work of, for example, Levitt (1996) and Donohue and Levitt (2001). In constructing a formal model, the idea that crime is purposeful means that an observed criminal act is understood as the outcome of a decision problem in which a criminal maximizes an expected utility function sub- ject to whatever constraints he faces. The utility function is not a primitive assumption about behavior (i.e., no economist thinks that agents carry explicit representations of utility functions in their heads); rather, it is a mathematical representation of an individual’s preferences, one that consti- tutes a rank ordering across the potential actions the individual may take. The choice-theoretic conception does not, by itself, have any implica- tions for the process by which agents make these decisions, although cer- tain behavioral restrictions are standard for economists. For example, to say that the commission of a crime is a purposeful act says nothing about how an individual assesses the various probabilities that are relevant to the choice, such as the conditional probability of being caught given that the crime is committed. That said, the economic analyses typically assume that an individual’s subjective beliefs—that is, the probabilities that inform his decision—are rational in the sense that they correspond to the probabili-

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS ties generated by the optimal use of the individual’s available information. While the relaxation of this notion of rationality has been a major theme in recent economic research (behavioral economics is now an established field of the discipline), it has not generally been a central focus in crime research, at least as conducted by economists. But we emphasize that the choice- based approach does not require rationality as conventionally understood. As Becker (1993, p. 386) has written: “The analysis assumes that individu- als maximize welfare as they conceive it, whether they be selfish, altruistic, loyal, spiteful, or masochistic. Their behavior is forward looking, and it is also assumed to be consistent over time. In particular they try as best they can to anticipate the consequences of their actions.” To see how crime choice may be formally described, we follow the standard binary choice model of economics. We consider the decision problem of individuals indexed by i each of whom decides at each period t whether or not to commit a crime. Individuals live in locations l, and it is assumed that a person commits crimes only within the location in which he lives. Individual behaviors are coded as wi,t = 1 if a crime is committed, 0 otherwise. A common form for the expected utility associated with the () choice ui,t ω i,t is () () () ui, t ω i, t = Zl , t βω i, t + Xi, t γω i, t + ξl , t ω i, t + ε i, t ω i, t . (1) In this expression, Zl,t denotes a set of observable (to the modeler) location-specific characteristics, and Xi,t denotes a vector of observable individual-specific characteristics. The multiplication of the terms Zl,t b and Xi,t g by wi,t capture the idea that the utility effect of these variables depends on whether the crime is committed. For example, the effect of a particular set of punishments on an individual’s utility will differ according () () to whether or not he commits a crime. The terms ξl ,t ω i,t and ε i,t ω i,t denote unobservable (to the modeler) location-specific and individual-spe- cific utility terms. These are functions of wi,t because these effects also depend on whether a crime was committed. From the perspective of a modeler, an individual’s sense of guilt is unobservable, and may be thought of as a utility consequence that occurs if he commits a crime. Similarly, the quality of the police force in a location is not observable (even if empirical proxies exist) and will affect utility only if a crime is committed, in this case via the effect on the likelihood of apprehension and punishment. The assumption of linearity of the utility function, while common in binary choice analysis, represents a statistical simplification and does not derive from choice-based reasoning per se. It is possible to consider nonparametric forms of the utility function (see Matzkin, 1992). We focus on the linear case both because it is the empirical standard in much of

OCR for page 211
 UNDERSTANDING CRIME TRENDS social science and because it is not clear that more general forms will be particularly informative for the issues we wish to address. Some forms of nonlinearity may be trivially introduced, such as including the products of elements of any initial choice of Xi,t as additional observables. The distinction between observable and unobservable variables is fun- damental to the relationship between choice-based theories of crime and their embodiment in a statistical framework. We assume that the indi- vidual and location-specific unobservables are independent of each other both contemporaneously and across time. We further assume that the individual-specific errors are independent of both the individual-specific and location-specific observables. We do not assume that the location-specific unobservables are independent of the location-specific observables; there is no good theoretical reason why they should be so and, unlike the other independence assumptions, whether it holds or not is important in the interpretation of aggregate regressions. Under our specification, the net expected utility from committing a crime is ν i, t = Zl , t β + Xi, t γ + ξl , t (1) − ξl , t (0) + ε i, t (1) − ε i, t (0), (2) and the choice-based perspective amounts to saying that a person chooses to commit a crime if the net utility is positive, that is, wi,t = 1, if and only if Zl ,t β + Xi,t γ + ξl ,t (1) − ξl ,t (0) > ε i,t (0) − ε i,t (1) . (3) Inequality (3) is useful as it provides a way of assigning probabilities to crime choices. Conditional on Xi,t , Zl,t , and ξl ,t (1) − ξl ,t (0) , the individual choices are stochastic; the distribution function of ε i,t (0) − ε i,t (1) , which we denote by Gi,t , determines the probability that a crime is committed. Formally, ( ) ( ) Pr ω i, t = 1 Zl , t , Xi, t , ξl , t (1) − ξl , t (0) = Gi, t Zl , t β + Xi, t γ + ξl , t (1) − ξl , t (0) . (4) This conditional probability structure captures the microfoundations of the economic model we wish to study. This formulation is in fact a rela- tively simple behavioral model, in that we ignore issues such as (1) selection into and out of the population generated by the dynamics of incarcera- tion and (2) those aspects of a crime decision at t in which a choice is a single component in a sequence of decisions that collectively determine an individual’s utility; that is, a more general preference specification is one in which agents make decisions to maximize a weighted average of current and future utility, accounting for the intertemporal effects of their deci- sions in each period. While the introduction of dynamic considerations

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS into the choice problem raises numerous issues, such as state dependence, heterogeneity, and dynamic selection, these can in principle be dealt with using the analysis of Heckman and Navarro (2007), albeit at the expense of considerable complication of the analysis. AGGREGATION How do the conditional crime probabilities for individuals described by (4) aggregate within a location? Let rl,t denote the realized crime rate in locality l at time t. Notice that we define the crime rate as the percent- age of individuals committing crimes, not the number of crimes per se, so we are ignoring multiple acts by a single criminal. Given our assumptions, for the location-specific choice model (4), if individuals are constrained to commit crimes in the location of residence, then the aggregate crime rate in a locality is determined by integrating over the observable individual-spe- cific heterogeneity in the location’s population. Let FX denote the empirical l ,t distribution function of Xi,t within l. The expected crime rate in a location at a given time is ( ) ( ) E ρl , t Zl , t , FX , ξl , t (1) − ξl , t (0) = ∫ Gi, t Zl , t β + Xγ + ξl , t (1) − ξl , t (0) dFX (5) l ,t l ,t In order to convert this aggregate relationship into a linear regres- sion form, it is necessary to further restrict the distribution function Gi,t. Suppose that the associated probability densities dGi,t are uniform; a uni- form density produces what is known as a linear probability model for the individual choices. In this case, the crime rate in locality l at time t obeys ρl ,t = Zl ,t β + X l ,t γ + ξl ,t (1) − ξl ,t (0) + θl ,t , (6) where is the empirical mean of w ithin a nd X l ,t X i,t l ( ) θl ,t = ρl ,t − E ρl ,t Zl ,t , FX , ξl ,t (1) − ξl ,t (0) captures the difference between l ,t the realized and expected crime rate in a locality. This is the model typically employed in aggregate crime regressions. Our construction of equation (6) from choice-based foundations illus- trates how standard aggregate crime regressions require a number of statis- tical assumptions if they are to be interpreted as aggregations of individual behavior. The assumption of a uniform density for the individual specific heterogeneity is of concern; in order to ensure that the probabilities of each choice are bounded between 0 and 1, the support of the uniform density may need to be agent-specific.2 Unfortunately, other random utility speci- 2 See Aldrich and Nelson (1984, Chapter 1) for an accessible discussion of the problems of the linear probability model.

OCR for page 211
 UNDERSTANDING CRIME TRENDS fications do not aggregate in a straightforward manner. To illustrate the () problem, note that if one assumes that ε i,t ω i,t has a type-I extreme value distribution, which is the implicit assumption in the logit binary choice ( )  Pr ω = 1 Z , X , ξ (1) − ξ (0)  i ,t i ,t l ,t i ,t l ,t l ,t model, then log   will be linear in ( )  1 − Pr ω = 1 Z , X , ξ (1) − ξ (0)    i ,t i ,t l ,t i ,t l ,t l ,t the various payoff components but will not produce a closed form solution for the aggregate crime rate. Methods are available to allow for analysis of aggregate data under logit type assumptions (see Berry, Levinsohn, and Pakes, 1995) but have not been applied, as far as we know, to the crime context. On its own terms, our development of a linear crime regression indi- cates how aggregation affects the consistency of particular estimators. While we have assumed that the individual-specific unobserved and observed determinants of crime choices are independent, we have not made an () analogous assumption on the location-specific unobservables ξl ,t ω i,t . In the aggregate regression, these may be correlated with either the aggregate observables that appear in the utility function Zl,t or those variables that appear as a consequence of aggregation X l ,t . From the perspective of theorizing about individual behavior, there is no reason why the regression residual ξl ,t (1) − ξl ,t (0) + θl ,t should be orthogonal to any of the regressors in equation (6). By implication, this means that all the variables in equation (6) should be instrumented. Hence in our judgment the focus on instru- menting endogenous regressors that one finds in empirical crime analyses is often insufficient, in that, while this strategy addresses endogeneity, it does not address unobserved location-specific heterogeneity. Notice that if individual-level data were available, this problem would not arise, since one would normally allow for location-specific, time-specific, and location- time-specific fixed effects for a panel. COUNTERFACTUAL ANALYSIS How can an aggregate crime regression be used to evaluate counter- factuals such as a change in policy? Given our choice-theoretic framework, a counterfactual analysis may be understood as a comparison of choices under alternative policy regimes A and B. The net utility to the commission of a crime will depend on the regime, so that ν iAt = ZlAt β A + XiAt γ A + ξlAt (1) − ξlAt (0) + ε iAt (1) − ε iAt (0) (7) , , , , , , , and

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS ν iBt = ZlBt β B + XiBt γ B + ξlBt (1) − ξlBt (0) + ε iBt (1) − ε iBt (0) (8) , , , , , , , respectively. The net utility to individual i of committing a crime equals ν i ,t = ZlAt β A + XiAt γ A + ξlAt (1) − ξlAt (0) + ε iAt (1) − ε iAt (0) + , , , , , , ( ) ( ) Dl ,t ZlBt β B − ZlAt β A + Dl ,t XiBt γ B − XiAt γ A + (9) , , , , ( )) ( Dl ,t ξlBt (1) − ξlBt (0) − ξlAt (1) − ξlAt (0) + , , , , D (ε (0 ))) . (0 ) − ( ε (1) − εiBt (1) − εiAt B A l ,t i ,t , i ,t , where Dl,t = 1 if regime B applies to locality l at t; 0 otherwise. The analo- gous linear aggregate crime rate regression is ρl , t = ( ) + Dl , t ZlBt β B − ZlAt β A + Dl , t  X l , t γ B − X l , t γ A  + A B A (10) ZlAt β A + X l,t γ A   , , , ( ) ( ) ξlAt (1) − ξlAt (0) + θlAt + Dl , t ξlBt (1) − ξlBt (0) − ξlAt (1) − ξlAt (0) + θlBt − θlAt .. , , , , , , , , , The standard approach measuring how different policies affect the crime rate, in this case regimes A versus B, is to embody the policy change in ZlAt versus ZlBt and to assume that all model parameters are con- , , stant across regimes. This allows the policy effect to be measured by ( ) ZlBt − ZlAt β . Equation (10) indicates how a number of assumptions are , , embedded in the standard approach, in particular the requirement that ( ) ξlBt (1) − ξlBt (0) − ξlAt (1) − ξlAt (0) = 0 , that is, that the change of regime does , , , , not change the location-specific unobserved utility differential between committing a crime and not doing so. This requirement seems problematic, as it means that the researcher must be willing to assume that the regime change is fully measured by the changes in X l ,t and Zl,t. Changes in the detection probabilities and penalties for crimes typically come in bundles, and we argue below that there are cases, specifically capital punishment, in which this does not receive adequate attention in the relevant empirical formulations. MODEL UNCERTAINTY Our derivation of aggregate crime rates from microfoundations assumed that the researcher had strong prior information about the individual deci- sion process. Put differently, our derivation of an aggregate crime regression

OCR for page 211
 UNDERSTANDING CRIME TRENDS was based on certainty about the underlying model of criminal behavior. In this section, we discuss ways to relax this assumption, that is, we con- sider the case of model uncertainty. In raising this, we emphasize that the problem of inadequate attention to model uncertainty is in no way unique to criminology. Nor do we mean to suggest that criminological studies are unique in the extent to which authors fail to investigate how modifications in baseline models affect inferences. Characterizing Model Uncertainty Our reading of the criminology literature suggests several general sources of model uncertainty. The categories we describe have previously been proposed by Brock, Durlauf, and West (2003) for economic growth models and Brock, Durlauf, and West (2007) for business cycle models. These categories are meant to identify general types of model uncertainty that are common in social science analyses. At the same time, our decompo- sition of model uncertainty is not unique; one can well imagine alternative divisions. Theory Uncertainty Social science theories for a given phenomenon are often open-ended (Brock and Durlauf, 2001), which means that one theory does not logically exclude another as having additional explanatory power. Hence there is often no justification for focusing on a subset of plausible explanations in empirical work. Some evidence of why this matters is suggested by Levitt’s (2004) evaluation of sources of the crime decline of the 1990s. Levitt iden- tifies 10 alternative theories of the crime decline, all of which are mutually consistent. Without questioning any of his substantive conclusions, we do note that Levitt is to a large extent forced to evaluate the roles of the dif- ferent theories based on studies that, typically, do not account for the full range of the competing explanations when measuring the empirical salience of a particular one. Statistical Instantiation Models may differ with respect to details of statistical specification that have nothing to do with the underlying social science theories that moti- vate them, but rather are employed in order to translate these theories into representations that are amenable to data analysis. This is typically so even when the social science theories are themselves expressed mathematically. Differences in these assumptions can lead to different findings. A good example of how differences in statistical assumptions can

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS affect substantive conclusions is specification of time trends. In the context of the deterrence effects of shall-issue concealed weapons carry laws, dif- ferent time trend choices have proven to be important. Specifically, Black and Nagin (1998) found that the use of quadratic time trends in place of state-specific linear time trends eliminates the evidence of a link between liberalization of concealed weapons laws and crime rates found in Lott and Mustard (1997). Lott’s rejoinder (1998) argues that it is hard to identify the effects of a policy change (in this case, concealed weapons legality) because a quadratic trend will mask it; intuitively, if crime is rising before a law is passed and decreases thereafter, this will be approximated by the quadratic trend.3 Lott’s intuition may be reasonable, but his argument is question begging, as it applies in both directions. If crime follows an exogenously determined quadratic trend over some time interval and rising crime levels lead to a change in legislation, then Lott’s approach will spuriously identify a causal effect from the legislation. This is true even if state-specific trends are employed. From the perspective of model uncertainty, Black and Nagin and Lott are working with different statistical instantiations of unexplained temporal heterogeneity. Under the Black and Nagin specification, there may be, as Lott argues, substantial collinearity between the variable used to measure temporal heterogeneity and the variables used to measure the effects of shall-issue concealed weapons legislation. This multicollinearity does not invalidate the Black and Nagin model on logical grounds. In our judgment, the differences between Black and Nagin and Lott on this issue reflect the absence of good explanations for much of the temporal evolution of crime rates. Neither a linear specification nor a quadratic specification (or for that matter, more flexible splines or alternative semiparametric methods) instantiate substantive ideas about the crime process. Rather, they con- stitute efforts to purge the data so that the residual components may be analyzed. Trend specification also matters in the analysis of unemployment rates and crime. Greenberg (2001) criticizes Cantor and Land (1985) for model- ing trends using deterministic rather than unit root methods. Again, social science theory does not dictate a preference for one type of trend over another. While both Greenberg and Cantor suggest justifications in favor of their trend specifications that derive from individual behavioral determi- nants, neither of them demonstrates a one-to-one or even precise mapping from these determinants to their statistical modeling assumptions. Other examples of this type of model uncertainty include assumptions about additivity, linearity, and the use of logarithms versus levels. 3 This argument is further developed in Plassmann and Whitley (2003).

OCR for page 211
0 UNDERSTANDING CRIME TRENDS Parameter Heterogeneity A third type of model uncertainty concerns parameter heterogeneity. Researchers often disagree on whether or not observations are simply draws from a common data-generating process, so that any heterogeneity in the observations derives from differences in values of some set of observable control variables and different realizations of the model errors. Social sci- ence theory typically does not impose that parameters are constant across observations. For example, the argument that there is a deterrent effect from a given penalty does not imply that the effect is independent of the geographical unit in which the penalty is present. Parameter heterogene- ity may be linked to deep questions about the interpretation of statistical models; see Brock and Durlauf (2001) for a discussion of parameter hetero- geneity and the concept of exchangeability of observations. Exchangeability, roughly speaking, captures the idea that observations, such as state-specific crime rates, may be treated as draws from a common statistical process. One example of sensitivity of empirical claims to assumptions about parameter heterogeneity is again found in the controversy between Black and Nagin and Mustard and Lott. Black and Nagin found that evidence of crime reductions associated with shall-issue laws are sensitive to the presence of Florida in the dataset. They found that eliminating data from Florida eliminated the evidentiary support for a handgun-crime link from some of the Lott and Mustard specifications. Another example appears in the capital punishment literature. Donohue and Wolfers (2005) challenge findings of Dezhbakhsh, Rubin, and Shepherd (2003) on the grounds that the findings are not robust to the exclusion of California and Texas. As argued by Cohen-Cole et al. (2008), this disagreement may be understood as a disagreement about parameter homogeneity. Model Averaging How can the dependence of empirical claims on model specification be constructively addressed? We describe a strategy based on model averag- ing; ideas associated with model averaging appear to originate in Leamer (1978). They have become prominent in the past decade in statistics; a valu- able conceptual argument is made in Draper (1995), and the development of formal methods has been greatly advanced by Raftery (e.g., Raftery, Madigan, and Hoeting, 1997). We proceed using Bayesian language for expositional convenience, although the analysis can be done using frequen- tist estimators. For a given exercise, suppose that the objective of the researcher is to construct a conditional density of crime rates rl,t+ based on data Dt and

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS ( ) model m, that is, Pr ρl ,t +1 Dt , m . Many disagreements about substantive empirical questions, such as forecasts or the effects of alternative policies, derive from disagreements about the choice of model m. This is, of course, why model selection plays such a significant role in empirical work. From the perspective of some empirical questions, it is not obvious that this is the appropriate role for model choice. If the goal of an exercise is to compare policies, the model choice is a nuisance parameter. Similarly, if one wants to construct a forecast, then the model itself is not intrinsically interesting. In order to avoid dependence on a particular model specification, an alternative strategy is to develop conclusions based on a space of candidate models; denote this space as M. Probability statements about a future out- come such as rl,t+ can then be constructed conditioning on the entire model space rather than on one of its elements. In other words, one computes the ( ) probability density Pr ρl ,t +1 Dt , M , which is the conditional density of the crime rate given the data and a model space. From this perspective, the true model is an unknown that needs to be integrated out of the probability density. Formally, ( ) ∑ Pr ( ρ )( ) Pr ρl , t +1 Dt , M = Dt , m Pr m Dt .. (11) l , t +1 m ∈M ( ) Here Pr m Dt denotes the posterior probability that m is the correct model given the data. Conditioning on M means that the analyst knows which models comprise M. Intuitively, one constructs probability state- ments about an outcome, such as a crime rate, based on aggregating the information available across each of the models under consideration. This aggregation places greater weight on models that are more likely, as mea- ( ) sured by Pr m Dt . The linear structure in equation (11) derives from the law of conditional probability, hence the term averaging. Model averaging is emerging as a common methodology in econom- ics; its increasing popularity reflects a combination of improved computa- tional capacity and theoretical advances. The approach has been used to study economic growth (Brock, Durlauf, and West, 2003; Doppelhofer, Miller, and Sala-i-Martin, 2004; Fernandez, Ley, and Steel, 2001), finance (Avramov, 2002), forecasting (Garratt et al., 2003), and monetary policy (Brock, Durlauf, and West, 2003). An application to a crime context, the deterrent effect of capital punishment, is Cohen-Cole et al. (2008). While we regard model-averaging methods as very promising, we also emphasize that the methodology is still being developed and a number of outstand- ing theoretical questions still exist.4 And of course, model averaging still 4 One issue concerning model priors that is worth noting concerns the assignment of priors to similar models. Most of the model-averaging literature has employed diffuse priors, that is, all models are assigned equal prior weights. However, it can be the case that some models in

OCR for page 211
 UNDERSTANDING CRIME TRENDS deterrence in the sample studied by Dezhbakhsh, Rubin, and Shepherd is weak. Policy-Relevant Calculations Following our general discussion, the statistical significance of the capital punishment variables in a murder regression does not produce the appropriate information needed to make policy comparisons. This has implications for the way such evidence is employed in death penalty debates. Sunstein and Vermeule (2005) argue that evidence of a deterrent effect can produce a moral case for capital punishment, in that the decision of a government to fail to implement a life-saving policy is equivalent to the decision to implement a policy that costs lives. Sunstein and Vermeule (2005) develop their argument conditioning on evidence of a deterrence effect. Leaving aside the insouciance with which they treat the empirical literature,7 their argument lacks attention to the appropriate nature of the policy maker’s loss function and the nature of the uncertainty of the empirical evidence. The Sunstein and Vermeule analysis treats the expected number of lives saved as the variable of interest to the policy maker; in Dezhbakhsh, Rubin, and Shepherd, this value is a function of the estimated parameter bE in (20). The expected number of lives saved is not necessarily sufficient in describing a policy maker’s utility function, even if this function is a monotonically increasing function of the number of lives saved. As such, their attention to this figure is analogous to making a utilitarian as opposed to a welfarist calculation (see Sen, 1979). While Sunstein and Vermeule would presum- ably respond that they are assuming that the precision associated with estimates of the expected number of lives saved is high, precision needs to be defined with respect to the policy maker’s utility function. It is not an independent object. The sensitivity of deterrence evidence to model choice, as demon- strated by Donohue and Wolfers and extended in Cohen-Cole et al. (2008), raises the issues we have discussed with respect to decision making under ambiguity and the evaluation of policies when one does not wish to base them on a choice of model priors. Without a justification of the choice of priors, there is no expected deterrence effect on which Sunstein and Vermeule can even rely. Our impression of the philosophy literature is that 7 At the same time they also state that “The foundation of our argument is a large and growing body of evidence that capital punishment may well have a deterrent effect, possibly a quite powerful one. . . . The particular numbers do not much matter” (p. 706).

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS the issue of policy evaluation under ambiguity has generally not been dis- cussed, although Gaus (2006) makes an interesting argument in favor of following principles rather than expected-effect calculations when assessing policies, the effects of which are associated with substantial uncertainty. To be clear, none of this means that Sunstein and Vermeule (2005) are incorrect in their conclusions about the ethical implications of a certain deterrent effect for a policy maker or that the death penalty is either moral or immoral per se. Rather, our claim is that the policy implications of the uncertainty associated with deterrence effects cannot be assessed outside of the policy maker’s preferences. Right-to-Carry Laws and Crime: Firearms and Violence Revisited Our third example is the controversy over the effects of shall-issue concealed weapons laws in the National Academies report Firearms and Violence (National Research Council, 2005). This report concluded (pp. 150-151): with the current evidence it is not possible to determine that there is a causal link between the right-to-carry laws and crime rates. It is also the committee’s view that additional analysis along the lines of the current literature is unlikely to yield results that will persuasively demonstrate a causal link between right-to-carry laws and crime rates (unless substantial numbers of states were to adopt or repeal right-to-carry laws), because of the sensitivity of the results to model specification. Committee member James Q. Wilson dissented from this part of the study, on the grounds that the sensitivity to specification found in the report did not account for the sensibility of different models; in particular, he ques- tioned whether the failure of models that excluded socioeconomic control variables to find deterrent effects was of importance in assessing the deter- rent effect. Wilson observes (National Research Council, 2005, p. 270): Suppose Professor Jones wrote a paper saying that increasing the number of police in a city reduced the crime rate and Professor Smith wrote a rival paper saying that cities with few police officers have low crime rates. Suppose that neither Smith nor Jones used any control variables, such as income, unemployment, population density, or the frequency with which offenders are sent to prison in reaching their conclusions. If such papers were published, they would be rejected out of hand by the committee for the obvious reason that they failed to supply a complete account of the factors that affect the crime rate.

OCR for page 211
 UNDERSTANDING CRIME TRENDS The committee’s rejoinder to Wilson argued (National Research Council, 2005, pp. 273-274): Everyone (including Wilson and the rest of the committee) agrees that control variables matter, but there is disagreement on the correct set. Thus, the facts that there is no way to statistically test for the correct specifica- tion and that researchers using reasonable specifications find different answers are highly relevant. Given the existing data and methods, the rest of the committee sees little hope of resolving this fundamental statistical problem. We believe that this conclusion is too pessimistic. The disagreement between Wilson and the rest of the National Academies committee reflects the absence in the report of an explicit evaluation of how model uncertainty interacts with evidence of shall-issue laws. While the assertion that it is impossible to statistically identify the correct specification of a statistical model is true at some level of generality (although the report is frankly unclear on what is meant by this), this argument is hardly novel; it is known in the philosophy literature as the Duhem-Quine hypothesis (Quine, 1951, is the classic statement) and refers to the idea that all theories are undeter- mined by available data. At this level of generality the National Academies committee claim is an uninteresting observation with respect to social science research, since it begs the question of the relative plausibility of assumptions.8 For the dispute at hand, we believe that Wilson is correct in his argument that a model whose specification includes controls suggested by social science theory should receive greater weight than one that does not. Furthermore, these two models are statistically distinguishable. To conclude that one should regard evidence of a deterrent effect as persuasive only if both models produce the same findings makes little sense. The report implicitly suggests that the models without control variables are intrinsically interest- ing: “No link between right-to-carry laws and changes in crime is appar- ent in the raw data . . . ; it is only once numerous covariates are included that the . . . effects . . . emerge” (p. 150). This remark ignores the classic Simpson’s paradox, in which a bivariate relationship has one direction, whereas a multivariate relationship does not. The standard example of Simpson’s paradox is the positive relationship between admission to the hospital and the probability of death. 8 The report’s suggestion that randomized experiments represent the gold standard for research ignores the assumptions required for their conduct—integrity of the researcher, accu- racy of data collection, etc. An advocate of randomized experiments would presumably dismiss concerns about such factors as implausible—but this is precisely our point.

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS Model averaging provides a natural way of integrating the information across the alternative specifications considered in the National Academies report. As we see it, the committee could have addressed the sensitivity of shall-issue deterrence effects by constructing a set of specifications that included those found in the literature as well as others that are formed by combining the assumptions underlying these models. Intuitively, one thinks of the assumptions that differentiate models as the axes of the model space, and one fills the model space out with those combinations of assumptions that are coherent with one another. Averaging over this space would have integrated the information in the different models and indicated whether evidence of a shall-issue deterrent effect is present when one conditions on a model space rather than a particular model. One answer to our advocacy of model averaging as a tool to address model uncertainty of the type facing the National Academies committee is that a given body of empirical studies captures only a small fraction of the universe of potential models (and indeed might represent a measure 0 set). This is certainly a tenable position. But if this position is taken, then it would be irrelevant whether a given body of studies produced similar or conflicting results. If it is then claimed that the degree of consistency in results across models contained in a subspace is informative about the results that would be ascertained were the model space expanded, then it is difficult to see why the relative prior plausibility and relative evidentiary support within an initial model space are not informative as well. A second answer to the use of model averaging might rely on the absence of a principled basis for assigning prior model probabilities. We are certainly sympathetic to this view. But if this position is taken, then the implications of the body of model-specific findings of an effect of shall- issue laws to policy need to be explicitly considered. It is not obvious, for example, that the fragility that the National Academies report claims to be present in concealed weapons regressions is even an argument against the laws. Suppose that a policy maker possesses minimax preferences with respect to model uncertainty. Fragility of deterrence evidence does not logically lead to rejection of the policy; one needs to know the payoffs under the different models under consideration. The National Academies report seems to take the position that, in absence of strong evidence that the laws reduce crime, they should not be implemented. But minimax preferences do not, by themselves, generate this conclusion, which really is based on the presumption that the law should not be implemented unless there is compelling evidence of crime reduction. This line of reasoning can be justified (e.g., Brock, Durlauf, and West, 2003), but it requires context- specific argumentation. Therefore, a recommendation we make for policy evaluation studies

OCR for page 211
 UNDERSTANDING CRIME TRENDS such as Firearms and Violence is that claims about the robustness or fragility of various findings be evaluated with respect to different loss func- tions, with particular attention to minimax and minimax regret calculations as supplements to the standard Bayesian ones. SHOULD AGGREGATE CRIME REGRESSIONS BE ABANDONED? One response to the discussion in this paper would be to search for alternative ways of uncovering aggregate criminological facts. The critiques we have raised are part of the source of interest in so-called natural experiments, in which an exogenous event of some type allows a comparison of aggregate crime outcomes (see Levitt, 1996, for a nice example). In his appendix to the Firearms and Violence study, Horowitz (2005) makes a broad general argument against the use of regression models to elucidate the determinants of crime, specifically in terms of evaluating policy effects. While his focus is on concealed weapons laws, his claims apply with equal force to other crime contexts. According to Horowitz, “In summary, the problems posed by high-dimensional estimation, misspecified models, and lack of correct knowledge of the correct set of explanatory variables seem insurmountable with observational data” (National Research Council, 2005, p. 308). In contrast, he argues that random assignment of policies could in principle reveal their effects; in particular, he discusses how random assignment can allow for the estimation of average treatment effects (a par- ticular piece of legislation, such as shall-issue concealed weapons laws, is an example of a treatment). We of course concur that there does not exist an algorithm to infallibly identify the “true” model of crime (or for that matter, other phenomena) when the universe of candidate models is broad enough. However, we do not believe this means that crime regressions cannot be informative about policy. Different models have both different ex ante levels of plausibility and ex post levels of goodness of fit for a given body of observational data. The different concealed weapons regressions with and without socio- economic controls are not equally ex ante plausible, given the state of social science. And we do not know, given our priors, how the relative goodness of fit of the different models analyzed in the National Academies report would translate into different posterior model probabilities. Our discussion of the assumptions that underlie the interpretation of aggregate crime regressions may all be interpreted as examples for Horowitz’s arguments about the limitations of regression analysis of crime. We do not claim to have an answer to the question of how to integrate the different types of model uncertainty we have discussed into a single integrated framework, let alone introduce such factors as the extension of

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS the basic crime model to intertemporal decision making. Our disagreement with Horowitz is that we see a role for empirical models in informing policy discussion, even though the researcher is aware of untestable or unappeal- ing assumptions underlying them. The way in which models are used to inform beliefs necessarily requires judgments; this necessity does not mean that the models are uninformative. A researcher brings a body of social science and statistical knowledge to bear in the assessment of empirical results; this knowledge matters in assessing the dependence of a result on an assumption. Put differently, not all assumptions are equally arbitrary. The need for assumptions is not unique to regression analysis with obser- vational data; all empirical work is theory-laden (to use Quine’s phrase). An experiment of the type proposed by Horowitz with respect to shall-issue weapons permit laws—randomized legalization across states—would, if one is to use the findings to inform policy makers, require assumptions about (1) the degree to which potential criminals can alter the locations in which crimes are committed, (2) the nature of migration by potential criminals across state boundaries both before the experiment and in response to it, (3) the effect on the current crime choices of potential criminals of the knowledge that an experiment that may affect future laws in their state of residence is being conducted, etc. Also, the translation of findings from such an experiment into a recommendation for those states that did not imple- ment the policy requires exchangeability assumptions on the states. Does one assume that the deterrent effect of the law is identical across states? If state-level deterrent effects are heterogeneous, how is this heterogeneity to be modeled—via random effects, varying coefficients, or some other method?9 Randomized experiments cannot avoid the need for judgments; as described in detail in Heckman (2000, 2005), judgment is intrinsic to social scientific inquiry. Overall, we do not see good reasons to regard natural experiments as superior to regressions with observational data in terms of their relative utility as means of understanding crime.10 It is straightforward to construct examples in which one methodology can provide insights that the other does not. Each has a contribution to make in criminological research. 9 Abbring and Heckman (2007) provide a comprehensive overview of the assumptions required in developing estimates of treatment effects that account for considerations of the type hinted at in our discussion. 10 See Heckman (2005) and Manski (2007) for discussion of the limitations of experiments; Heckman and Navarro (2004) compare the strengths and weaknesses of different empirical strategies for uncovering the determinants of individual choice.

OCR for page 211
 UNDERSTANDING CRIME TRENDS CONCLUSION In this chapter, we have described some issues we regard as important in the econometric study of crime: microfoundations, aggregation, counter- factual analysis, and policy evaluation. We have tried to make clear the various assumptions that must be maintained to interpret aggregate crime regressions with respect to individual behavior and have emphasized how standard uses of these regressions to evaluate policy presuppose a number of assumptions. In light of disagreements about these assumptions, which ultimately underlie claims of fragility or robustness of an empirical result, we have outlined some ways of using model-averaging methods and statisti- cal decision theory to make progress. Throughout, we have emphasized the role of judgment in empirical work, for which no algorithm exists. ACKNOWLEDGMENTS We thank the National Science Foundation and University of Wisconsin Graduate School for financial support. Arthur Goldberger and Justin Wolfers provided immensely helpful comments on a previous draft. REFERENCES Abbring, Jaap H., and James J. Heckman. (2007). Econometric evaluation of social programs, part III: Distributional treatment effects, dynamic treatment effects, dynamic discrete choice, and general equilibrium policy evaluation. In J. Heckman and E. Leamer (Eds.), Handbook of econometrics, olume . Amsterdam: Elsevier. Aldrich, John H., and Forrest D. Nelson. (1984). Linear probability, logit, and probit models. Beverly Hills, CA: Sage. Avramov, Doron. (2002). Stock return predictability and model uncertainty. Journal of Finance, , 423-458. Ayres, Ian, and John Donohue, III. (2003). Shooting down the “more guns, less crime” hypothesis. Stanford Law Reiew, , 1193-1312. Becker, Gary S. (1968). Crime and punishment: An economic analysis. Journal of Political Economy, (2), 169-217. Becker, Gary S. (1993). Nobel lecture: The economic way of looking at behavior. Journal of Political Economy, 0(3), 385-409. Bernard, Andrew B., and Steven N. Durlauf. (1995). Convergence in international output. Journal of Applied Econometrics, 0(2), 97-108. Bernard, Andrew B., and Steven N. Durlauf. (1996). Interpreting tests of the convergence hypothesis. Journal of Econometrics, (1-2), 161-173. Berry, Steven, James Levinsohn, and Ariel Pakes. (1995). Automobile prices in market equi- librium. Econometrica, (4), 841-890. Black, Dan A., and Daniel S. Nagin. (1998). Do right-to-carry laws deter violent crimes? Journal of Legal Studies, (1), 209-219. Blundell, Richard, and Thomas Stoker. (2005). Heterogeneity and aggregation. Journal of Economic Literature, (2), 347-391.

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS Brock, William A., and Steven Durlauf. (2001). Growth economics and reality. World Bank Economic Reiew, , 229-272. Brock, William A., Steven Durlauf, and Kenneth West. (2003). Policy analysis in uncertain economic environments (with discussion). Brookings Papers on Economic Actiity, , 235-322. Brock, William A., Steven Durlauf, and Kenneth West. (2007). Model uncertainty and policy evaluation: Some theory and empirics. Journal of Econometrics, (2), 629-664. Brock, William A., Steven Durlauf, James Nason, and Giacomo Rondina. (2007). Simple versus optimal rules as guides to policy. Journal of Monetary Economics, (5), 1372-1396. Cantor, David, and Kenneth C. Land. (1985). Unemployment and crime rates in the post- World War II United States: A theoretical and empirical investigation. American Socio- logical Reiew, 0, 317-332. Cohen-Cole, Ethan, Steven Durlauf, Jeffrey Fagan, and Daniel Nagin. (2008). Model uncer- tainty and the deterrent effect of capital punishment. American Law and Economics Reiew, forthcoming. Dezhbakhsh, Hashem, Paul Rubin, and Joanna M. Shepherd. (2003). Does capital punishment have a deterrent effect? New evidence from post-moratorium panel data. American Law and Economics Reiew, (2), 344-376. Donohue, John, and Steven Levitt. (2001). The impact of legalized abortion on crime. Quarterly Journal of Economics, (2), 379-420. Donohue, John, and Justin Wolfers. (2005). Uses and abuses of empirical evidence in the death penalty debate. Stanford Law Reiew, (3), 791-846. Doppelhofer, Gernot, Ronald I. Miller, and Xavier Sala-i-Martin. (2004). Determinants of long-term growth: A Bayesian averaging of classical estimates (BACE) approach. American Economic Reiew, (4), 813-835. Draper, David. (1995). Assessment and propagation of model uncertainty. Journal of the Royal Statistical Society, Series B, , 45-70. Durlauf, Steven, Salvador Navarro, and David Rivers. (2008). Notes on the econometrics of deterrence. Unpublished working paper, Department of Economics, University of Wisconsin, Madison. Ehrlich, Isaac. (1972). The deterrent effect of criminal law enforcement. Journal of Legal Studies, (2), 259-276. Ehrlich, Isaac. (1973). Participation in illegal activities: A theoretical and empirical investiga- tion. Journal of Political Economy, (3), 521-565. Fernandez, Carmen, Eduardo Ley, and Mark Steel. (2001). Model uncertainty in cross-country growth regressions. Journal of Applied Econometrics, (5), 563-576. Garratt, Anthony, Kevin Lee, M. Hashem Pesaran, and Yongcheol Shin. (2003). Forecasting uncertainties in macroeconometric modelling: An application to the UK economy. Journal of the American Statistical Association, (464), 829-838. Gaus, Gerald F. (2006). Social complexity and eoled moral principles. Unpublished paper, University of Arizona. Gelman, Andrew, James S. Liebman, Valerie West, and Alexander Kiss. (2004). A broken sys- tem: The persistent patterns of reversals of death sentences in the United States. Journal of Empirical Legal Studies, (2), 209-261. Goldberger, Arthur S. (1991). A course in econometrics. Cambridge, MA: Harvard University Press. Greenberg, David. (2001). Time series analysis of crime rates. Journal of Quantitatie Crimi- nology, (4), 291-327. Hansen, Lars Peter, and Thomas J. Sargent. (2007). Robustness. Princeton, NJ: Princeton University Press.

OCR for page 211
0 UNDERSTANDING CRIME TRENDS Heckman, James L. (2000). Causal parameters and policy analysis in economics: A twentieth century retrospective. Quarterly Journal of Economics, (1), 45-97. Heckman, James L. (2005). The scientific model of causality. Sociological Methodology, , 1-98. Heckman, James L., and Salvador Navarro. (2004). Using matching, instrumental variables, and control functions to estimate economic choice models. Reiew of Economics and Statistics, (1), 30-57. Heckman, James L., and Salvador Navarro. (2007). Dynamic discrete choices and dynamic treatment effects. Journal of Econometrics, (2), 341-396. Horowitz, Joel L. (2005). Statistical issues in the evaluation of right-to-carry laws. In National Research Council, Firearms and iolence: A critical reiew (pp. 299-308), Committee to Improve Research Information and Data on Firearms, Charles F. Wellford, John V. Pepper, and Carol V. Petrie (Eds.), Committee on Law and Justice, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. LaFree, Gary. (2005). Evidence for elite convergence in cross-national homicide victimization trends, 1956 to 2000. The Sociological Quarterly, , 191-211. Leamer, Edward E. (1978). Specification searches. New York: John Wiley & Sons. Levitt, Steven D. (1996). The effect of prison population size on crime rates: Evidence from prison overcrowding litigation. Quarterly Journal of Economics, , 319-351. Levitt, Steven D. (2004). Understanding why crime fell in the 1990s: Four factors that explain the decline and six that do not. Journal of Economic Perspecties, (1), 163-190. Lott, John R. (1998). The concealed-handgun debate. Journal of Legal Studies, (1), 221-291. Lott, John R., and David B. Mustard. (1997). Crime, deterrence, and right-to-carry concealed handguns. Journal of Legal Studies, (1), 1-68. Mankiw, N. Gregory, David Romer, and David N. Weil. (1992). A contribution to the empirics of economic growth. Quarterly Journal of Economics, 0(2), 407-437. Manski, Charles F. (2005). Statistical treatment rules for heterogeneous populations. Econo- metrica, , 1221-1246. Manski, Charles F. (2006). Minimax-regret treatment choice with missing outcome data. Journal of Econometrics, (1), 105-115. Manski, Charles F. (2007). Identification for prediction and decision. Cambridge, MA: Harvard University Press. Marschak, Jacob. (1953). Economic measurements for policy and prediction. In William C. Hood and Tjalling C. Koopmans (Eds.), Studies in econometric method. New York: John Wiley & Sons. Matzkin, Rosa L. (1992). Nonparametric and distribution-free estimation of the binary thresh- old crossing and the binary choice models. Econometrica, 0(2), 239-270. National Research Council. (2005). Firearms and iolence: A critical reiew. Committee to Improve Research Information and Data on Firearms, Charles F. Wellford, John V. Pepper, and Carol V. Petrie (Eds.), Committee on Law and Justice, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. Navarro, Salvador. (2007). Control functions. In Lawrence E. Blume and Steven Durlauf (Eds.), New Palgrae dictionary of economics (revised ed.). London: Palgrave-Macmillan. O’Brien, Robert M. (1999). Measuring the convergence/divergence of “serious crime” arrest rates for males and females: 1960-1995. Journal of Quantitatie Criminology, (1), 97-114. Plassmann, Florenz, and John Whitley. (2003). Confirming “more guns, less crime.” Stanford Law Reiew, , 1313-1369. Quine, Willard Van Orman. (1951). Two dogmas of empiricism. Philosophical Reiew, 0, 20-43.

OCR for page 211
 ON THE USE OF AGGREGATE CRIME REGRESSIONS Raftery, Adrian E., David Madigan, and Jennifer A. Hoeting. (1997). Bayesian model averag- ing for linear regression models. Journal of the American Statistical Association, (437), 179-191. Savage, Leonard J. (1951). The theory of statistical decision. Journal of the American Statisti- cal Association, , 55-67. Sen, Amartya. (1979). Utilitarianism and welfarism, Journal of Philosophy, (9), 463-489. Stoker, Thomas M. (1993). Empirical approaches to the problem of aggregation over indi- viduals. Journal of Economic Literature, (4), 1827-1874. Sunstein, Cass R., and Adrian Vermeule. (2005). Is capital punishment morally required? Acts, omissions and life-life tradeoffs. Stanford Law Reiew, (3), 703-751. Wald, Abraham. (1950). Statistical decision functions. New York: John Wiley & Sons.

OCR for page 211