|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. 7
|
||||||||||||||||
 |
 |
Page 211 |
 |
|
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 211
Understanding Crime Trends: Workshop Report
7
On the Use of Aggregate Crime Regressions in Policy Evaluation
Steven N. Durlauf, Salvador Navarro, and David A. Rivers
Despite recent efforts to employ microeconomic data and natural experiments, aggregate crime regressions continue to play a significant role in criminological analyses. One use of these regressions is predictive, as illustrated by the papers in this volume that employ aggregate crime trends regressions—Baumer (Chapter 5) and Pepper (Chapter 6). A second use involves policy evaluation: Prominent and controversial cases include the deterrent effect of shall-issue concealed weapons legislation (e.g., Ayres and Donohue, 2003; Black and Nagin, 1998; Lott, 1998; Lott and Mustard, 1997; Plassmann and Whitley, 2003) and the deterrent effect of capital punishment (e.g., Dezhbakhsh, Rubin, and Shepherd, 2003; Donohue and Wolfers, 2005). These uses are interrelated, as is evident from the effort to evaluate how changes in criminal justice policies explain the great reduction of crime in the 1990s.
The goal of this chapter is to examine the construction and interpretation of aggregate crime regressions. Specifically, we employ contemporary economic and econometric reasoning to understand how aggregate crime regressions may be appropriately used to inform positive and normative questions. While by no means comprehensive, we hope our discussion will prove useful in highlighting some of the limitations of the use of these regressions and in particular will indicate how empirical findings may be misinterpreted when careful attention is not given to the link between the aggregate data and individual behavior.1
1
The interpretation of aggregate data continues to be one of the most difficult questions in social science; Stoker (1993) and Blundell and Stoker (2005) provide valuable overviews.
OCR for page 212
Understanding Crime Trends: Workshop Report
The chapter is organized as follows. We begin by describing a standard choice-based model of crime. We then discuss how this individual-level model can be aggregated to produce crime regressions of the type found in the literature. In the next three sections we discuss the analysis of counterfactuals, issues of model uncertainty in crime regressions, and the relationship between statistical models and policy evaluation. We then apply our general arguments to areas in the empirical criminology literature: the convergence of crime rates, capital punishment, and shall-issue concealed weapons laws. The next section discusses whether the limitations that exist in using crime regressions mean that they should be replaced by quasi-experimental methods, and a final section concludes the chapter. Our discussion is conceptual; Durlauf, Navarro, and Rivers (2008) provide a more systematic treatment of many of the issues we raise as well as an empirical application.
CRIME AS A CHOICE
From the vantage point of economics, the fundamental idea underlying the analysis of crime is that each criminal act constitutes a purposeful choice on the part of the criminal. In turn, this means that the development of a theory of the aggregate crime rate should be explicitly understood as deriving from the aggregation of individual decisions. The basic logic of the economic approach to crime was originally developed by Gary Becker (1968) and extended by Isaac Ehrlich (1972, 1973). This logic underlies the renaissance of crime research in economics, exemplified in the work of, for example, Levitt (1996) and Donohue and Levitt (2001).
In constructing a formal model, the idea that crime is purposeful means that an observed criminal act is understood as the outcome of a decision problem in which a criminal maximizes an expected utility function subject to whatever constraints he faces. The utility function is not a primitive assumption about behavior (i.e., no economist thinks that agents carry explicit representations of utility functions in their heads); rather, it is a mathematical representation of an individual’s preferences, one that constitutes a rank ordering across the potential actions the individual may take.
The choice-theoretic conception does not, by itself, have any implications for the process by which agents make these decisions, although certain behavioral restrictions are standard for economists. For example, to say that the commission of a crime is a purposeful act says nothing about how an individual assesses the various probabilities that are relevant to the choice, such as the conditional probability of being caught given that the crime is committed. That said, the economic analyses typically assume that an individual’s subjective beliefs—that is, the probabilities that inform his decision—are rational in the sense that they correspond to the probabili-
OCR for page 213
Understanding Crime Trends: Workshop Report
ties generated by the optimal use of the individual’s available information. While the relaxation of this notion of rationality has been a major theme in recent economic research (behavioral economics is now an established field of the discipline), it has not generally been a central focus in crime research, at least as conducted by economists. But we emphasize that the choice-based approach does not require rationality as conventionally understood. As Becker (1993, p. 386) has written: “The analysis assumes that individuals maximize welfare as they conceive it, whether they be selfish, altruistic, loyal, spiteful, or masochistic. Their behavior is forward looking, and it is also assumed to be consistent over time. In particular they try as best they can to anticipate the consequences of their actions.”
To see how crime choice may be formally described, we follow the standard binary choice model of economics. We consider the decision problem of individuals indexed by i each of whom decides at each period t whether or not to commit a crime. Individuals live in locations l, and it is assumed that a person commits crimes only within the location in which he lives. Individual behaviors are coded as ωi,t = 1 if a crime is committed, 0 otherwise. A common form for the expected utility associated with the choice ui,t (ωi,t) is
(1)
In this expression, Zl,t denotes a set of observable (to the modeler) location-specific characteristics, and Xi,t denotes a vector of observable individual-specific characteristics. The multiplication of the terms Zl,t β and Xi,t γ by ωi,t Capture the idea that the utility effect of these variables depends on whether the crime is committed. For example, the effect of a particular set of punishments on an individual’s utility will differ according to whether or not he commits a crime. The terms ξl,t (ωi,t) and εi,t (ωi,t) denote unobservable (to the modeler) location-specific and individual-specific utility terms. These are functions of ωi,t because these effects also depend on whether a crime was committed. From the perspective of a modeler, an individual’s sense of guilt is unobservable, and may be thought of as a utility consequence that occurs if he commits a crime. Similarly, the quality of the police force in a location is not observable (even if empirical proxies exist) and will affect utility only if a crime is committed, in this case via the effect on the likelihood of apprehension and punishment.
The assumption of linearity of the utility function, while common in binary choice analysis, represents a statistical simplification and does not derive from choice-based reasoning per se. It is possible to consider nonparametric forms of the utility function (see Matzkin, 1992). We focus on the linear case both because it is the empirical standard in much of
OCR for page 214
Understanding Crime Trends: Workshop Report
social science and because it is not clear that more general forms will be particularly informative for the issues we wish to address. Some forms of nonlinearity may be trivially introduced, such as including the products of elements of any initial choice of Xi,t as additional observables.
The distinction between observable and unobservable variables is fundamental to the relationship between choice-based theories of crime and their embodiment in a statistical framework. We assume that the individual and location-specific unobservables are independent of each other both contemporaneously and across time. We further assume that the individual-specific errors are independent of both the individual-specific and location-specific observables. We do not assume that the location-specific unobservables are independent of the location-specific observables; there is no good theoretical reason why they should be so and, unlike the other independence assumptions, whether it holds or not is important in the interpretation of aggregate regressions.
Under our specification, the net expected utility from committing a crime is
(2)
and the choice-based perspective amounts to saying that a person chooses to commit a crime if the net utility is positive, that is, ωi,t = 1, if and only if
(3)
Inequality (3) is useful as it provides a way of assigning probabilities to crime choices. Conditional on Xi,t, Zl,t, and ξl,t (1)−ξl,t (0), the individual choices are stochastic; the distribution function of εi,t (0)−εi,t (1), which we denote by Gi,t, determines the probability that a crime is committed. Formally,
(4)
This conditional probability structure captures the microfoundations of the economic model we wish to study. This formulation is in fact a relatively simple behavioral model, in that we ignore issues such as (1) selection into and out of the population generated by the dynamics of incarceration and (2) those aspects of a crime decision at t in which a choice is a single component in a sequence of decisions that collectively determine an individual’s utility; that is, a more general preference specification is one in which agents make decisions to maximize a weighted average of current and future utility, accounting for the intertemporal effects of their decisions in each period. While the introduction of dynamic considerations
OCR for page 215
Understanding Crime Trends: Workshop Report
into the choice problem raises numerous issues, such as state dependence, heterogeneity, and dynamic selection, these can in principle be dealt with using the analysis of Heckman and Navarro (2007), albeit at the expense of considerable complication of the analysis.
AGGREGATION
How do the conditional crime probabilities for individuals described by (4) aggregate within a location? Let ρl,t denote the realized crime rate in locality l at time t. Notice that we define the crime rate as the percentage of individuals committing crimes, not the number of crimes per se, so we are ignoring multiple acts by a single criminal. Given our assumptions, for the location-specific choice model (4), if individuals are constrained to commit crimes in the location of residence, then the aggregate crime rate in a locality is determined by integrating over the observable individual-specific heterogeneity in the location’s population. Let denote the empirical distribution function of Xi,t within l. The expected crime rate in a location at a given time is
(5)
In order to convert this aggregate relationship into a linear regression form, it is necessary to further restrict the distribution function Gi,t. Suppose that the associated probability densities dGi,t are uniform; a uniform density produces what is known as a linear probability model for the individual choices. In this case, the crime rate in locality l at time t obeys
(6)
where is the empirical mean of Xi,t within l and captures the difference between, the realized and expected crime rate in a locality. This is the model typically employed in aggregate crime regressions.
Our construction of equation (6) from choice-based foundations illustrates how standard aggregate crime regressions require a number of statistical assumptions if they are to be interpreted as aggregations of individual behavior. The assumption of a uniform density for the individual specific heterogeneity is of concern; in order to ensure that the probabilities of each choice are bounded between 0 and 1, the support of the uniform density may need to be agent-specific.2 Unfortunately, other random utility speci-
2
See Aldrich and Nelson (1984, Chapter 1) for an accessible discussion of the problems of the linear probability model.
OCR for page 216
Understanding Crime Trends: Workshop Report
fications do not aggregate in a straightforward manner. To illustrate the problem, note that if one assumes that εi,t (ωi,t) has a type-I extreme value distribution, which is the implicit assumption in the logit binary choice model, then will be linear in the various payoff components but will not produce a closed form solution for the aggregate crime rate. Methods are available to allow for analysis of aggregate data under logit type assumptions (see Berry, Levinsohn, and Pakes, 1995) but have not been applied, as far as we know, to the crime context.
On its own terms, our development of a linear crime regression indicates how aggregation affects the consistency of particular estimators. While we have assumed that the individual-specific unobserved and observed determinants of crime choices are independent, we have not made an analogous assumption on the location-specific unobservables ξl,t (ωi,t). In the aggregate regression, these may be correlated with either the aggregate observables that appear in the utility function Zl,t or those variables that appear as a consequence of aggregation . From the perspective of theorizing about individual behavior, there is no reason why the regression residual ξl,t (1)−ξl,t (0)+θl,t should be orthogonal to any of the regressors in equation (6). By implication, this means that all the variables in equation (6) should be instrumented. Hence in our judgment the focus on instrumenting endogenous regressors that one finds in empirical crime analyses is often insufficient, in that, while this strategy addresses endogeneity, it does not address unobserved location-specific heterogeneity. Notice that if individual-level data were available, this problem would not arise, since one would normally allow for location-specific, time-specific, and location-time-specific fixed effects for a panel.
COUNTERFACTUAL ANALYSIS
How can an aggregate crime regression be used to evaluate counter-factuals such as a change in policy? Given our choice-theoretic framework, a counterfactual analysis may be understood as a comparison of choices under alternative policy regimes A and B. The net utility to the commission of a crime will depend on the regime, so that
(7)
and
OCR for page 217
Understanding Crime Trends: Workshop Report
(8)
respectively. The net utility to individual i of committing a crime equals
(9)
where Dl,t = 1 if regime B applies to locality l at t; 0 otherwise. The analogous linear aggregate crime rate regression is
(10)
The standard approach measuring how different policies affect the crime rate, in this case regimes A versus B, is to embody the policy change in versus and to assume that all model parameters are constant across regimes. This allows the policy effect to be measured by . Equation (10) indicates how a number of assumptions are embedded in the standard approach, in particular the requirement that , that is, that the change of regime does not change the location-specific unobserved utility differential between committing a crime and not doing so. This requirement seems problematic, as it means that the researcher must be willing to assume that the regime change is fully measured by the changes in and Zl,t. Changes in the detection probabilities and penalties for crimes typically come in bundles, and we argue below that there are cases, specifically capital punishment, in which this does not receive adequate attention in the relevant empirical formulations.
MODEL UNCERTAINTY
Our derivation of aggregate crime rates from microfoundations assumed that the researcher had strong prior information about the individual decision process. Put differently, our derivation of an aggregate crime regression
OCR for page 218
Understanding Crime Trends: Workshop Report
was based on certainty about the underlying model of criminal behavior. In this section, we discuss ways to relax this assumption, that is, we consider the case of model uncertainty. In raising this, we emphasize that the problem of inadequate attention to model uncertainty is in no way unique to criminology. Nor do we mean to suggest that criminological studies are unique in the extent to which authors fail to investigate how modifications in baseline models affect inferences.
Characterizing Model Uncertainty
Our reading of the criminology literature suggests several general sources of model uncertainty. The categories we describe have previously been proposed by Brock, Durlauf, and West (2003) for economic growth models and Brock, Durlauf, and West (2007) for business cycle models. These categories are meant to identify general types of model uncertainty that are common in social science analyses. At the same time, our decomposition of model uncertainty is not unique; one can well imagine alternative divisions.
Theory Uncertainty
Social science theories for a given phenomenon are often open-ended (Brock and Durlauf, 2001), which means that one theory does not logically exclude another as having additional explanatory power. Hence there is often no justification for focusing on a subset of plausible explanations in empirical work. Some evidence of why this matters is suggested by Levitt’s (2004) evaluation of sources of the crime decline of the 1990s. Levitt identifies 10 alternative theories of the crime decline, all of which are mutually consistent. Without questioning any of his substantive conclusions, we do note that Levitt is to a large extent forced to evaluate the roles of the different theories based on studies that, typically, do not account for the full range of the competing explanations when measuring the empirical salience of a particular one.
Statistical Instantiation
Models may differ with respect to details of statistical specification that have nothing to do with the underlying social science theories that motivate them, but rather are employed in order to translate these theories into representations that are amenable to data analysis. This is typically so even when the social science theories are themselves expressed mathematically. Differences in these assumptions can lead to different findings.
A good example of how differences in statistical assumptions can
OCR for page 219
Understanding Crime Trends: Workshop Report
affect substantive conclusions is specification of time trends. In the context of the deterrence effects of shall-issue concealed weapons carry laws, different time trend choices have proven to be important. Specifically, Black and Nagin (1998) found that the use of quadratic time trends in place of state-specific linear time trends eliminates the evidence of a link between liberalization of concealed weapons laws and crime rates found in Lott and Mustard (1997). Lott’s rejoinder (1998) argues that it is hard to identify the effects of a policy change (in this case, concealed weapons legality) because a quadratic trend will mask it; intuitively, if crime is rising before a law is passed and decreases thereafter, this will be approximated by the quadratic trend.3 Lott’s intuition may be reasonable, but his argument is question begging, as it applies in both directions. If crime follows an exogenously determined quadratic trend over some time interval and rising crime levels lead to a change in legislation, then Lott’s approach will spuriously identify a causal effect from the legislation. This is true even if state-specific trends are employed.
From the perspective of model uncertainty, Black and Nagin and Lott are working with different statistical instantiations of unexplained temporal heterogeneity. Under the Black and Nagin specification, there may be, as Lott argues, substantial collinearity between the variable used to measure temporal heterogeneity and the variables used to measure the effects of shall-issue concealed weapons legislation. This multicollinearity does not invalidate the Black and Nagin model on logical grounds. In our judgment, the differences between Black and Nagin and Lott on this issue reflect the absence of good explanations for much of the temporal evolution of crime rates. Neither a linear specification nor a quadratic specification (or for that matter, more flexible splines or alternative semiparametric methods) instantiate substantive ideas about the crime process. Rather, they constitute efforts to purge the data so that the residual components may be analyzed.
Trend specification also matters in the analysis of unemployment rates and crime. Greenberg (2001) criticizes Cantor and Land (1985) for modeling trends using deterministic rather than unit root methods. Again, social science theory does not dictate a preference for one type of trend over another. While both Greenberg and Cantor suggest justifications in favor of their trend specifications that derive from individual behavioral determinants, neither of them demonstrates a one-to-one or even precise mapping from these determinants to their statistical modeling assumptions.
Other examples of this type of model uncertainty include assumptions about additivity, linearity, and the use of logarithms versus levels.
3
This argument is further developed in Plassmann and Whitley (2003).
OCR for page 220
Understanding Crime Trends: Workshop Report
Parameter Heterogeneity
A third type of model uncertainty concerns parameter heterogeneity. Researchers often disagree on whether or not observations are simply draws from a common data-generating process, so that any heterogeneity in the observations derives from differences in values of some set of observable control variables and different realizations of the model errors. Social science theory typically does not impose that parameters are constant across observations. For example, the argument that there is a deterrent effect from a given penalty does not imply that the effect is independent of the geographical unit in which the penalty is present. Parameter heterogeneity may be linked to deep questions about the interpretation of statistical models; see Brock and Durlauf (2001) for a discussion of parameter heterogeneity and the concept of exchangeability of observations. Exchangeability, roughly speaking, captures the idea that observations, such as state-specific crime rates, may be treated as draws from a common statistical process.
One example of sensitivity of empirical claims to assumptions about parameter heterogeneity is again found in the controversy between Black and Nagin and Mustard and Lott. Black and Nagin found that evidence of crime reductions associated with shall-issue laws are sensitive to the presence of Florida in the dataset. They found that eliminating data from Florida eliminated the evidentiary support for a handgun-crime link from some of the Lott and Mustard specifications.
Another example appears in the capital punishment literature. Donohue and Wolfers (2005) challenge findings of Dezhbakhsh, Rubin, and Shepherd (2003) on the grounds that the findings are not robust to the exclusion of California and Texas. As argued by Cohen-Cole et al. (2008), this disagreement may be understood as a disagreement about parameter homogeneity.
Model Averaging
How can the dependence of empirical claims on model specification be constructively addressed? We describe a strategy based on model averaging; ideas associated with model averaging appear to originate in Leamer (1978). They have become prominent in the past decade in statistics; a valuable conceptual argument is made in Draper (1995), and the development of formal methods has been greatly advanced by Raftery (e.g., Raftery, Madigan, and Hoeting, 1997). We proceed using Bayesian language for expositional convenience, although the analysis can be done using frequentist estimators.
For a given exercise, suppose that the objective of the researcher is to construct a conditional density of crime rates ρl,t+1 based on data Dt and
OCR for page 221
Understanding Crime Trends: Workshop Report
model m, that is, Pr(ρl,t+1|Dt,m). Many disagreements about substantive empirical questions, such as forecasts or the effects of alternative policies, derive from disagreements about the choice of model m. This is, of course, why model selection plays such a significant role in empirical work. From the perspective of some empirical questions, it is not obvious that this is the appropriate role for model choice. If the goal of an exercise is to compare policies, the model choice is a nuisance parameter. Similarly, if one wants to construct a forecast, then the model itself is not intrinsically interesting.
In order to avoid dependence on a particular model specification, an alternative strategy is to develop conclusions based on a space of candidate models; denote this space as M. Probability statements about a future outcome such as ρ,t+1 can then be constructed conditioning on the entire model space rather than on one of its elements. In other words, one computes the probability density Pr(ρl,t+1|Dt,m), which is the conditional density of the crime rate given the data and a model space. From this perspective, the true model is an unknown that needs to be integrated out of the probability density. Formally,
(11)
Here Pr(m|Dt) denotes the posterior probability that m is the correct model given the data. Conditioning on M means that the analyst knows which models comprise M. Intuitively, one constructs probability statements about an outcome, such as a crime rate, based on aggregating the information available across each of the models under consideration. This aggregation places greater weight on models that are more likely, as measured by Pr(m|Dt). The linear structure in equation (11) derives from the law of conditional probability, hence the term averaging.
Model averaging is emerging as a common methodology in economics; its increasing popularity reflects a combination of improved computational capacity and theoretical advances. The approach has been used to study economic growth (Brock, Durlauf, and West, 2003; Doppelhofer, Miller, and Sala-i-Martin, 2004; Fernandez, Ley, and Steel, 2001), finance (Avramov, 2002), forecasting (Garratt et al., 2003), and monetary policy (Brock, Durlauf, and West, 2003). An application to a crime context, the deterrent effect of capital punishment, is Cohen-Cole et al. (2008). While we regard model-averaging methods as very promising, we also emphasize that the methodology is still being developed and a number of outstanding theoretical questions still exist.4 And of course, model averaging still
4
One issue concerning model priors that is worth noting concerns the assignment of priors to similar models. Most of the model-averaging literature has employed diffuse priors, that is, all models are assigned equal prior weights. However, it can be the case that some models in
OCR for page 232
Understanding Crime Trends: Workshop Report
deterrence in the sample studied by Dezhbakhsh, Rubin, and Shepherd is weak.
Policy-Relevant Calculations
Following our general discussion, the statistical significance of the capital punishment variables in a murder regression does not produce the appropriate information needed to make policy comparisons. This has implications for the way such evidence is employed in death penalty debates. Sunstein and Vermeule (2005) argue that evidence of a deterrent effect can produce a moral case for capital punishment, in that the decision of a government to fail to implement a life-saving policy is equivalent to the decision to implement a policy that costs lives.
Sunstein and Vermeule (2005) develop their argument conditioning on evidence of a deterrence effect. Leaving aside the insouciance with which they treat the empirical literature,7 their argument lacks attention to the appropriate nature of the policy maker’s loss function and the nature of the uncertainty of the empirical evidence.
The Sunstein and Vermeule analysis treats the expected number of lives saved as the variable of interest to the policy maker; in Dezhbakhsh, Rubin, and Shepherd, this value is a function of the estimated parameter βE in (20). The expected number of lives saved is not necessarily sufficient in describing a policy maker’s utility function, even if this function is a monotonically increasing function of the number of lives saved. As such, their attention to this figure is analogous to making a utilitarian as opposed to a welfarist calculation (see Sen, 1979). While Sunstein and Vermeule would presumably respond that they are assuming that the precision associated with estimates of the expected number of lives saved is high, precision needs to be defined with respect to the policy maker’s utility function. It is not an independent object.
The sensitivity of deterrence evidence to model choice, as demonstrated by Donohue and Wolfers and extended in Cohen-Cole et al. (2008), raises the issues we have discussed with respect to decision making under ambiguity and the evaluation of policies when one does not wish to base them on a choice of model priors. Without a justification of the choice of priors, there is no expected deterrence effect on which Sunstein and Vermeule can even rely. Our impression of the philosophy literature is that
7
At the same time they also state that
“The foundation of our argument is a large and growing body of evidence that capital punishment may well have a deterrent effect, possibly a quite powerful one…. The particular numbers do not much matter” (p. 706).
OCR for page 233
Understanding Crime Trends: Workshop Report
the issue of policy evaluation under ambiguity has generally not been discussed, although Gaus (2006) makes an interesting argument in favor of following principles rather than expected-effect calculations when assessing policies, the effects of which are associated with substantial uncertainty.
To be clear, none of this means that Sunstein and Vermeule (2005) are incorrect in their conclusions about the ethical implications of a certain deterrent effect for a policy maker or that the death penalty is either moral or immoral per se. Rather, our claim is that the policy implications of the uncertainty associated with deterrence effects cannot be assessed outside of the policy maker’s preferences.
Right-to-Carry Laws and Crime: Firearms and Violence Revisited
Our third example is the controversy over the effects of shall-issue concealed weapons laws in the National Academies report Firearms and Violence (National Research Council, 2005). This report concluded (pp. 150-151):
with the current evidence it is not possible to determine that there is a causal link between the right-to-carry laws and crime rates. It is also the committee’s view that additional analysis along the lines of the current literature is unlikely to yield results that will persuasively demonstrate a causal link between right-to-carry laws and crime rates (unless substantial numbers of states were to adopt or repeal right-to-carry laws), because of the sensitivity of the results to model specification.
Committee member James Q. Wilson dissented from this part of the study, on the grounds that the sensitivity to specification found in the report did not account for the sensibility of different models; in particular, he questioned whether the failure of models that excluded socioeconomic control variables to find deterrent effects was of importance in assessing the deterrent effect. Wilson observes (National Research Council, 2005, p. 270):
Suppose Professor Jones wrote a paper saying that increasing the number of police in a city reduced the crime rate and Professor Smith wrote a rival paper saying that cities with few police officers have low crime rates. Suppose that neither Smith nor Jones used any control variables, such as income, unemployment, population density, or the frequency with which offenders are sent to prison in reaching their conclusions. If such papers were published, they would be rejected out of hand by the committee for the obvious reason that they failed to supply a complete account of the factors that affect the crime rate.
OCR for page 234
Understanding Crime Trends: Workshop Report
The committee’s rejoinder to Wilson argued (National Research Council, 2005, pp. 273-274):
Everyone (including Wilson and the rest of the committee) agrees that control variables matter, but there is disagreement on the correct set. Thus, the facts that there is no way to statistically test for the correct specification and that researchers using reasonable specifications find different answers are highly relevant. Given the existing data and methods, the rest of the committee sees little hope of resolving this fundamental statistical problem.
We believe that this conclusion is too pessimistic. The disagreement between Wilson and the rest of the National Academies committee reflects the absence in the report of an explicit evaluation of how model uncertainty interacts with evidence of shall-issue laws. While the assertion that it is impossible to statistically identify the correct specification of a statistical model is true at some level of generality (although the report is frankly unclear on what is meant by this), this argument is hardly novel; it is known in the philosophy literature as the Duhem-Quine hypothesis (Quine, 1951, is the classic statement) and refers to the idea that all theories are undetermined by available data.
At this level of generality the National Academies committee claim is an uninteresting observation with respect to social science research, since it begs the question of the relative plausibility of assumptions.8 For the dispute at hand, we believe that Wilson is correct in his argument that a model whose specification includes controls suggested by social science theory should receive greater weight than one that does not. Furthermore, these two models are statistically distinguishable. To conclude that one should regard evidence of a deterrent effect as persuasive only if both models produce the same findings makes little sense. The report implicitly suggests that the models without control variables are intrinsically interesting: “No link between right-to-carry laws and changes in crime is apparent in the raw data …; it is only once numerous covariates are included that the … effects … emerge” (p. 150). This remark ignores the classic Simpson’s paradox, in which a bivariate relationship has one direction, whereas a multivariate relationship does not. The standard example of Simpson’s paradox is the positive relationship between admission to the hospital and the probability of death.
8
The report’s suggestion that randomized experiments represent the gold standard for research ignores the assumptions required for their conduct—integrity of the researcher, accuracy of data collection, etc. An advocate of randomized experiments would presumably dismiss concerns about such factors as implausible—but this is precisely our point.
OCR for page 235
Understanding Crime Trends: Workshop Report
Model averaging provides a natural way of integrating the information across the alternative specifications considered in the National Academies report. As we see it, the committee could have addressed the sensitivity of shall-issue deterrence effects by constructing a set of specifications that included those found in the literature as well as others that are formed by combining the assumptions underlying these models. Intuitively, one thinks of the assumptions that differentiate models as the axes of the model space, and one fills the model space out with those combinations of assumptions that are coherent with one another. Averaging over this space would have integrated the information in the different models and indicated whether evidence of a shall-issue deterrent effect is present when one conditions on a model space rather than a particular model.
One answer to our advocacy of model averaging as a tool to address model uncertainty of the type facing the National Academies committee is that a given body of empirical studies captures only a small fraction of the universe of potential models (and indeed might represent a measure 0 set). This is certainly a tenable position. But if this position is taken, then it would be irrelevant whether a given body of studies produced similar or conflicting results. If it is then claimed that the degree of consistency in results across models contained in a subspace is informative about the results that would be ascertained were the model space expanded, then it is difficult to see why the relative prior plausibility and relative evidentiary support within an initial model space are not informative as well.
A second answer to the use of model averaging might rely on the absence of a principled basis for assigning prior model probabilities. We are certainly sympathetic to this view. But if this position is taken, then the implications of the body of model-specific findings of an effect of shall-issue laws to policy need to be explicitly considered. It is not obvious, for example, that the fragility that the National Academies report claims to be present in concealed weapons regressions is even an argument against the laws. Suppose that a policy maker possesses minimax preferences with respect to model uncertainty. Fragility of deterrence evidence does not logically lead to rejection of the policy; one needs to know the payoffs under the different models under consideration. The National Academies report seems to take the position that, in absence of strong evidence that the laws reduce crime, they should not be implemented. But minimax preferences do not, by themselves, generate this conclusion, which really is based on the presumption that the law should not be implemented unless there is compelling evidence of crime reduction. This line of reasoning can be justified (e.g., Brock, Durlauf, and West, 2003), but it requires context-specific argumentation.
Therefore, a recommendation we make for policy evaluation studies
OCR for page 236
Understanding Crime Trends: Workshop Report
such as Firearms and Violence is that claims about the robustness or fragility of various findings be evaluated with respect to different loss functions, with particular attention to minimax and minimax regret calculations as supplements to the standard Bayesian ones.
SHOULD AGGREGATE CRIME REGRESSIONS BE ABANDONED?
One response to the discussion in this paper would be to search for alternative ways of uncovering aggregate criminological facts. The critiques we have raised are part of the source of interest in so-called natural experiments, in which an exogenous event of some type allows a comparison of aggregate crime outcomes (see Levitt, 1996, for a nice example). In his appendix to the Firearms and Violence study, Horowitz (2005) makes a broad general argument against the use of regression models to elucidate the determinants of crime, specifically in terms of evaluating policy effects.
While his focus is on concealed weapons laws, his claims apply with equal force to other crime contexts. According to Horowitz, “In summary, the problems posed by high-dimensional estimation, misspecified models, and lack of correct knowledge of the correct set of explanatory variables seem insurmountable with observational data” (National Research Council, 2005, p. 308). In contrast, he argues that random assignment of policies could in principle reveal their effects; in particular, he discusses how random assignment can allow for the estimation of average treatment effects (a particular piece of legislation, such as shall-issue concealed weapons laws, is an example of a treatment).
We of course concur that there does not exist an algorithm to infallibly identify the “true” model of crime (or for that matter, other phenomena) when the universe of candidate models is broad enough. However, we do not believe this means that crime regressions cannot be informative about policy. Different models have both different ex ante levels of plausibility and ex post levels of goodness of fit for a given body of observational data. The different concealed weapons regressions with and without socioeconomic controls are not equally ex ante plausible, given the state of social science. And we do not know, given our priors, how the relative goodness of fit of the different models analyzed in the National Academies report would translate into different posterior model probabilities.
Our discussion of the assumptions that underlie the interpretation of aggregate crime regressions may all be interpreted as examples for Horowitz’s arguments about the limitations of regression analysis of crime. We do not claim to have an answer to the question of how to integrate the different types of model uncertainty we have discussed into a single integrated framework, let alone introduce such factors as the extension of
OCR for page 237
Understanding Crime Trends: Workshop Report
the basic crime model to intertemporal decision making. Our disagreement with Horowitz is that we see a role for empirical models in informing policy discussion, even though the researcher is aware of untestable or unappealing assumptions underlying them. The way in which models are used to inform beliefs necessarily requires judgments; this necessity does not mean that the models are uninformative. A researcher brings a body of social science and statistical knowledge to bear in the assessment of empirical results; this knowledge matters in assessing the dependence of a result on an assumption. Put differently, not all assumptions are equally arbitrary.
The need for assumptions is not unique to regression analysis with observational data; all empirical work is theory-laden (to use Quine’s phrase). An experiment of the type proposed by Horowitz with respect to shall-issue weapons permit laws—randomized legalization across states—would, if one is to use the findings to inform policy makers, require assumptions about (1) the degree to which potential criminals can alter the locations in which crimes are committed, (2) the nature of migration by potential criminals across state boundaries both before the experiment and in response to it, (3) the effect on the current crime choices of potential criminals of the knowledge that an experiment that may affect future laws in their state of residence is being conducted, etc. Also, the translation of findings from such an experiment into a recommendation for those states that did not implement the policy requires exchangeability assumptions on the states. Does one assume that the deterrent effect of the law is identical across states? If state-level deterrent effects are heterogeneous, how is this heterogeneity to be modeled—via random effects, varying coefficients, or some other method?9 Randomized experiments cannot avoid the need for judgments; as described in detail in Heckman (2000, 2005), judgment is intrinsic to social scientific inquiry.
Overall, we do not see good reasons to regard natural experiments as superior to regressions with observational data in terms of their relative utility as means of understanding crime.10 It is straightforward to construct examples in which one methodology can provide insights that the other does not. Each has a contribution to make in criminological research.
9
Abbring and Heckman (2007) provide a comprehensive overview of the assumptions required in developing estimates of treatment effects that account for considerations of the type hinted at in our discussion.
10
See Heckman (2005) and Manski (2007) for discussion of the limitations of experiments; Heckman and Navarro (2004) compare the strengths and weaknesses of different empirical strategies for uncovering the determinants of individual choice.
OCR for page 238
Understanding Crime Trends: Workshop Report
CONCLUSION
In this chapter, we have described some issues we regard as important in the econometric study of crime: microfoundations, aggregation, counterfactual analysis, and policy evaluation. We have tried to make clear the various assumptions that must be maintained to interpret aggregate crime regressions with respect to individual behavior and have emphasized how standard uses of these regressions to evaluate policy presuppose a number of assumptions. In light of disagreements about these assumptions, which ultimately underlie claims of fragility or robustness of an empirical result, we have outlined some ways of using model-averaging methods and statistical decision theory to make progress. Throughout, we have emphasized the role of judgment in empirical work, for which no algorithm exists.
ACKNOWLEDGMENTS
We thank the National Science Foundation and University of Wisconsin Graduate School for financial support. Arthur Goldberger and Justin Wolfers provided immensely helpful comments on a previous draft.
REFERENCES
Abbring, Jaap H., and James J. Heckman. (2007). Econometric evaluation of social programs, part III: Distributional treatment effects, dynamic treatment effects, dynamic discrete choice, and general equilibrium policy evaluation. In J. Heckman and E. Leamer (Eds.), Handbook of econometrics, volume 6. Amsterdam: Elsevier.
Aldrich, John H., and Forrest D. Nelson. (1984). Linear probability, logit, and probit models. Beverly Hills, CA: Sage.
Avramov, Doron. (2002). Stock return predictability and model uncertainty. Journal of Finance, 64, 423-458.
Ayres, Ian, and John Donohue, III. (2003). Shooting down the “more guns, less crime” hypothesis. Stanford Law Review, 55, 1193-1312.
Becker, Gary S. (1968). Crime and punishment: An economic analysis. Journal of Political Economy, 78(2), 169-217.
Becker, Gary S. (1993). Nobel lecture: The economic way of looking at behavior. Journal of Political Economy, 101(3), 385-409.
Bernard, Andrew B., and Steven N. Durlauf. (1995). Convergence in international output. Journal of Applied Econometrics, 10(2), 97-108.
Bernard, Andrew B., and Steven N. Durlauf. (1996). Interpreting tests of the convergence hypothesis. Journal of Econometrics, 71(1-2), 161-173.
Berry, Steven, James Levinsohn, and Ariel Pakes. (1995). Automobile prices in market equilibrium. Econometrica, 63(4), 841-890.
Black, Dan A., and Daniel S. Nagin. (1998). Do right-to-carry laws deter violent crimes? Journal of Legal Studies, 27(1), 209-219.
Blundell, Richard, and Thomas Stoker. (2005). Heterogeneity and aggregation. Journal of Economic Literature, 43(2), 347-391.
OCR for page 239
Understanding Crime Trends: Workshop Report
Brock, William A., and Steven Durlauf. (2001). Growth economics and reality. World Bank Economic Review, 15,229-272.
Brock, William A., Steven Durlauf, and Kenneth West. (2003). Policy analysis in uncertain economic environments (with discussion). Brookings Papers on Economic Activity, 1, 235-322.
Brock, William A., Steven Durlauf, and Kenneth West. (2007). Model uncertainty and policy evaluation: Some theory and empirics. Journal of Econometrics, 136(2), 629-664.
Brock, William A., Steven Durlauf, James Nason, and Giacomo Rondina. (2007). Simple versus optimal rules as guides to policy. Journal of Monetary Economics, 54(5), 1372-1396.
Cantor, David, and Kenneth C. Land. (1985). Unemployment and crime rates in the post-World War II United States: A theoretical and empirical investigation. American Sociological Review, 50, 317-332.
Cohen-Cole, Ethan, Steven Durlauf, Jeffrey Fagan, and Daniel Nagin. (2008). Model uncertainty and the deterrent effect of capital punishment. American Law and Economics Review, forthcoming.
Dezhbakhsh, Hashem, Paul Rubin, and Joanna M. Shepherd. (2003). Does capital punishment have a deterrent effect? New evidence from post-moratorium panel data. American Law and Economics Review, 5(2), 344-376.
Donohue, John, and Steven Levitt. (2001). The impact of legalized abortion on crime. Quarterly Journal of Economics, 116(2), 379-420.
Donohue, John, and Justin Wolfers. (2005). Uses and abuses of empirical evidence in the death penalty debate. Stanford Law Review, 58(3),791-846.
Doppelhofer, Gernot, Ronald I. Miller, and Xavier Sala-i-Martin. (2004). Determinants of long-term growth: A Bayesian averaging of classical estimates (BACE) approach. American Economic Review, 94(4), 813-835.
Draper, David. (1995). Assessment and propagation of model uncertainty. Journal of the Royal Statistical Society, Series B, 57, 45-70.
Durlauf, Steven, Salvador Navarro, and David Rivers. (2008). Notes on the econometrics of deterrence. Unpublished working paper, Department of Economics, University of Wisconsin, Madison.
Ehrlich, Isaac. (1972). The deterrent effect of criminal law enforcement. Journal of Legal Studies, 1(2), 259-276.
Ehrlich, Isaac. (1973). Participation in illegal activities: A theoretical and empirical investigation. Journal of Political Economy, 81(3), 521-565.
Fernandez, Carmen, Eduardo Ley, and Mark Steel. (2001). Model uncertainty in cross-country growth regressions. Journal of Applied Econometrics, 16(5), 563-576.
Garratt, Anthony, Kevin Lee, M. Hashem Pesaran, and Yongcheol Shin. (2003). Forecasting uncertainties in macroeconometric modelling: An application to the UK economy. Journal of the American Statistical Association, 98(464), 829-838.
Gaus, Gerald F. (2006). Social complexity and evolved moral principles. Unpublished paper, University of Arizona.
Gelman, Andrew, James S. Liebman, Valerie West, and Alexander Kiss. (2004). A broken system: The persistent patterns of reversals of death sentences in the United States. Journal of Empirical Legal Studies, 1(2), 209-261.
Goldberger, Arthur S. (1991). A course in econometrics. Cambridge, MA: Harvard University Press.
Greenberg, David. (2001). Time series analysis of crime rates. Journal of Quantitative Criminology, 17(4), 291-327.
Hansen, Lars Peter, and Thomas J. Sargent. (2007). Robustness. Princeton, NJ: Princeton University Press.
OCR for page 240
Understanding Crime Trends: Workshop Report
Heckman, James L. (2000). Causal parameters and policy analysis in economics: A twentieth century retrospective. Quarterly Journal of Economics, 115(1), 45-97.
Heckman, James L. (2005). The scientific model of causality. Sociological Methodology, 35, 1-98.
Heckman, James L., and Salvador Navarro. (2004). Using matching, instrumental variables, and control functions to estimate economic choice models. Review of Economics and Statistics, 86(1), 30-57.
Heckman, James L., and Salvador Navarro. (2007). Dynamic discrete choices and dynamic treatment effects. Journal of Econometrics, 136(2), 341-396.
Horowitz, Joel L. (2005). Statistical issues in the evaluation of right-to-carry laws. In National Research Council, Firearms and violence: A critical review (pp. 299-308), Committee to Improve Research Information and Data on Firearms, Charles F. Wellford, John V. Pepper, and Carol V. Petrie (Eds.), Committee on Law and Justice, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
LaFree, Gary. (2005). Evidence for elite convergence in cross-national homicide victimization trends, 1956 to 2000. The Sociological Quarterly, 46, 191-211.
Leamer, Edward E. (1978). Specification searches. New York: John Wiley & Sons.
Levitt, Steven D. (1996). The effect of prison population size on crime rates: Evidence from prison overcrowding litigation. Quarterly Journal of Economics, 111, 319-351.
Levitt, Steven D. (2004). Understanding why crime fell in the 1990s: Four factors that explain the decline and six that do not. Journal of Economic Perspectives, 18(1), 163-190.
Lott, John R. (1998). The concealed-handgun debate. Journal of Legal Studies, 27(1), 221-291.
Lott, John R., and David B. Mustard. (1997). Crime, deterrence, and right-to-carry concealed handguns. Journal of Legal Studies, 26(1), 1-68.
Mankiw, N. Gregory, David Romer, and David N. Weil. (1992). A contribution to the empirics of economic growth. Quarterly Journal of Economics, 107(2), 407-437.
Manski, Charles F. (2005). Statistical treatment rules for heterogeneous populations. Econometrica, 72, 1221-1246.
Manski, Charles F. (2006). Minimax-regret treatment choice with missing outcome data. Journal of Econometrics, 139(1), 105-115.
Manski, Charles F. (2007). Identification for prediction and decision. Cambridge, MA: Harvard University Press.
Marschak, Jacob. (1953). Economic measurements for policy and prediction. In William C. Hood and Tjalling C. Koopmans (Eds.), Studies in econometric method. New York: John Wiley & Sons.
Matzkin, Rosa L. (1992). Nonparametric and distribution-free estimation of the binary threshold crossing and the binary choice models. Econometrica, 60(2), 239-270.
National Research Council. (2005). Firearms and violence: A critical review. Committee toImprove Research Information and Data on Firearms, Charles F. Wellford, John V. Pepper, and Carol V. Petrie (Eds.), Committee on Law and Justice, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
Navarro, Salvador. (2007). Control functions. In Lawrence E. Blume and Steven Durlauf (Eds.), New Palgrave dictionary of economics (revised ed.). London: Palgrave-Macmillan.
O’Brien, Robert M. (1999). Measuring the convergence/divergence of “serious crime” arrest rates for males and females: 1960-1995. Journal of Quantitative Criminology, 15(1), 97-114.
Plassmann, Florenz, and John Whitley. (2003). Confirming “more guns, less crime.” Stanford Law Review, 55, 1313-1369.
Quine, Willard Van Orman. (1951). Two dogmas of empiricism. Philosophical Review, 60, 20-43.
OCR for page 241
Understanding Crime Trends: Workshop Report
Raftery, Adrian E., David Madigan, and Jennifer A. Hoeting. (1997). Bayesian model averaging for linear regression models. Journal of the American Statistical Association, 92(437), 179-191.
Savage, Leonard J. (1951). The theory of statistical decision. Journal of the American Statistical Association, 46, 55-67.
Sen, Amartya. (1979). Utilitarianism and welfarism, Journal of Philosophy, 76(9), 463-489.
Stoker, Thomas M. (1993). Empirical approaches to the problem of aggregation over individuals. Journal of Economic Literature, 31(4), 1827-1874.
Sunstein, Cass R., and Adrian Vermeule. (2005). Is capital punishment morally required? Acts, omissions and life-life tradeoffs. Stanford Law Review, 58(3), 703-751.
Wald, Abraham. (1950). Statistical decision functions. New York: John Wiley & Sons.
OCR for page 242
Understanding Crime Trends: Workshop Report
This page intentionally left blank.