On the Use of Aggregate Crime Regressions in Policy Evaluation

*Steven N. Durlauf, Salvador Navarro, and David A. Rivers*

Despite recent efforts to employ microeconomic data and natural experiments, aggregate crime regressions continue to play a significant role in criminological analyses. One use of these regressions is predictive, as illustrated by the papers in this volume that employ aggregate crime trends regressions—Baumer (Chapter 5) and Pepper (Chapter 6). A second use involves policy evaluation: Prominent and controversial cases include the deterrent effect of shall-issue concealed weapons legislation (e.g., Ayres and Donohue, 2003; Black and Nagin, 1998; Lott, 1998; Lott and Mustard, 1997; Plassmann and Whitley, 2003) and the deterrent effect of capital punishment (e.g., Dezhbakhsh, Rubin, and Shepherd, 2003; Donohue and Wolfers, 2005). These uses are interrelated, as is evident from the effort to evaluate how changes in criminal justice policies explain the great reduction of crime in the 1990s.

The goal of this chapter is to examine the construction and interpretation of aggregate crime regressions. Specifically, we employ contemporary economic and econometric reasoning to understand how aggregate crime regressions may be appropriately used to inform positive and normative questions. While by no means comprehensive, we hope our discussion will prove useful in highlighting some of the limitations of the use of these regressions and in particular will indicate how empirical findings may be misinterpreted when careful attention is not given to the link between the aggregate data and individual behavior.^{1}

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.

Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 211

7
On the Use of Aggregate Crime
Regressions in Policy Evaluation
Steen N. Durlauf, Salador Naarro, and Daid A. Riers
Despite recent efforts to employ microeconomic data and natural
experiments, aggregate crime regressions continue to play a significant role
in criminological analyses. One use of these regressions is predictive, as
illustrated by the papers in this volume that employ aggregate crime trends
regressions—Baumer (Chapter 5) and Pepper (Chapter 6). A second use
involves policy evaluation: Prominent and controversial cases include the
deterrent effect of shall-issue concealed weapons legislation (e.g., Ayres and
Donohue, 2003; Black and Nagin, 1998; Lott, 1998; Lott and Mustard,
1997; Plassmann and Whitley, 2003) and the deterrent effect of capital
punishment (e.g., Dezhbakhsh, Rubin, and Shepherd, 2003; Donohue and
Wolfers, 2005). These uses are interrelated, as is evident from the effort to
evaluate how changes in criminal justice policies explain the great reduction
of crime in the 1990s.
The goal of this chapter is to examine the construction and interpreta-
tion of aggregate crime regressions. Specifically, we employ contemporary
economic and econometric reasoning to understand how aggregate crime
regressions may be appropriately used to inform positive and normative
questions. While by no means comprehensive, we hope our discussion
will prove useful in highlighting some of the limitations of the use of these
regressions and in particular will indicate how empirical findings may be
misinterpreted when careful attention is not given to the link between the
aggregate data and individual behavior.1
1 The interpretation of aggregate data continues to be one of the most difficult questions in
social science; Stoker (1993) and Blundell and Stoker (2005) provide valuable overviews.

OCR for page 211

UNDERSTANDING CRIME TRENDS
The chapter is organized as follows. We begin by describing a stan-
dard choice-based model of crime. We then discuss how this individual-
level model can be aggregated to produce crime regressions of the type
found in the literature. In the next three sections we discuss the analysis
of counterfactuals, issues of model uncertainty in crime regressions, and
the relationship between statistical models and policy evaluation. We then
apply our general arguments to areas in the empirical criminology litera-
ture: the convergence of crime rates, capital punishment, and shall-issue
concealed weapons laws. The next section discusses whether the limitations
that exist in using crime regressions mean that they should be replaced by
quasi-experimental methods, and a final section concludes the chapter. Our
discussion is conceptual; Durlauf, Navarro, and Rivers (2008) provide a
more systematic treatment of many of the issues we raise as well as an
empirical application.
CRIME AS A CHOICE
From the vantage point of economics, the fundamental idea underlying
the analysis of crime is that each criminal act constitutes a purposeful
choice on the part of the criminal. In turn, this means that the development
of a theory of the aggregate crime rate should be explicitly understood as
deriving from the aggregation of individual decisions. The basic logic of
the economic approach to crime was originally developed by Gary Becker
(1968) and extended by Isaac Ehrlich (1972, 1973). This logic underlies the
renaissance of crime research in economics, exemplified in the work of, for
example, Levitt (1996) and Donohue and Levitt (2001).
In constructing a formal model, the idea that crime is purposeful means
that an observed criminal act is understood as the outcome of a decision
problem in which a criminal maximizes an expected utility function sub-
ject to whatever constraints he faces. The utility function is not a primitive
assumption about behavior (i.e., no economist thinks that agents carry
explicit representations of utility functions in their heads); rather, it is a
mathematical representation of an individual’s preferences, one that consti-
tutes a rank ordering across the potential actions the individual may take.
The choice-theoretic conception does not, by itself, have any implica-
tions for the process by which agents make these decisions, although cer-
tain behavioral restrictions are standard for economists. For example, to
say that the commission of a crime is a purposeful act says nothing about
how an individual assesses the various probabilities that are relevant to the
choice, such as the conditional probability of being caught given that the
crime is committed. That said, the economic analyses typically assume that
an individual’s subjective beliefs—that is, the probabilities that inform his
decision—are rational in the sense that they correspond to the probabili-

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
ties generated by the optimal use of the individual’s available information.
While the relaxation of this notion of rationality has been a major theme in
recent economic research (behavioral economics is now an established field
of the discipline), it has not generally been a central focus in crime research,
at least as conducted by economists. But we emphasize that the choice-
based approach does not require rationality as conventionally understood.
As Becker (1993, p. 386) has written: “The analysis assumes that individu-
als maximize welfare as they conceive it, whether they be selfish, altruistic,
loyal, spiteful, or masochistic. Their behavior is forward looking, and it is
also assumed to be consistent over time. In particular they try as best they
can to anticipate the consequences of their actions.”
To see how crime choice may be formally described, we follow the
standard binary choice model of economics. We consider the decision
problem of individuals indexed by i each of whom decides at each period
t whether or not to commit a crime. Individuals live in locations l, and it
is assumed that a person commits crimes only within the location in which
he lives. Individual behaviors are coded as wi,t = 1 if a crime is committed,
0 otherwise. A common form for the expected utility associated with the
()
choice ui,t ω i,t is
() () ()
ui, t ω i, t = Zl , t βω i, t + Xi, t γω i, t + ξl , t ω i, t + ε i, t ω i, t . (1)
In this expression, Zl,t denotes a set of observable (to the modeler)
location-specific characteristics, and Xi,t denotes a vector of observable
individual-specific characteristics. The multiplication of the terms Zl,t b
and Xi,t g by wi,t capture the idea that the utility effect of these variables
depends on whether the crime is committed. For example, the effect of a
particular set of punishments on an individual’s utility will differ according
() ()
to whether or not he commits a crime. The terms ξl ,t ω i,t and ε i,t ω i,t
denote unobservable (to the modeler) location-specific and individual-spe-
cific utility terms. These are functions of wi,t because these effects also
depend on whether a crime was committed. From the perspective of a
modeler, an individual’s sense of guilt is unobservable, and may be thought
of as a utility consequence that occurs if he commits a crime. Similarly, the
quality of the police force in a location is not observable (even if empirical
proxies exist) and will affect utility only if a crime is committed, in this case
via the effect on the likelihood of apprehension and punishment.
The assumption of linearity of the utility function, while common
in binary choice analysis, represents a statistical simplification and does
not derive from choice-based reasoning per se. It is possible to consider
nonparametric forms of the utility function (see Matzkin, 1992). We focus
on the linear case both because it is the empirical standard in much of

OCR for page 211

UNDERSTANDING CRIME TRENDS
social science and because it is not clear that more general forms will be
particularly informative for the issues we wish to address. Some forms of
nonlinearity may be trivially introduced, such as including the products of
elements of any initial choice of Xi,t as additional observables.
The distinction between observable and unobservable variables is fun-
damental to the relationship between choice-based theories of crime and
their embodiment in a statistical framework. We assume that the indi-
vidual and location-specific unobservables are independent of each other
both contemporaneously and across time. We further assume that the
individual-specific errors are independent of both the individual-specific and
location-specific observables. We do not assume that the location-specific
unobservables are independent of the location-specific observables; there
is no good theoretical reason why they should be so and, unlike the other
independence assumptions, whether it holds or not is important in the
interpretation of aggregate regressions.
Under our specification, the net expected utility from committing a
crime is
ν i, t = Zl , t β + Xi, t γ + ξl , t (1) − ξl , t (0) + ε i, t (1) − ε i, t (0), (2)
and the choice-based perspective amounts to saying that a person chooses to
commit a crime if the net utility is positive, that is, wi,t = 1, if and only if
Zl ,t β + Xi,t γ + ξl ,t (1) − ξl ,t (0) > ε i,t (0) − ε i,t (1) . (3)
Inequality (3) is useful as it provides a way of assigning probabilities to
crime choices. Conditional on Xi,t , Zl,t , and ξl ,t (1) − ξl ,t (0) , the individual
choices are stochastic; the distribution function of ε i,t (0) − ε i,t (1) , which
we denote by Gi,t , determines the probability that a crime is committed.
Formally,
( ) ( )
Pr ω i, t = 1 Zl , t , Xi, t , ξl , t (1) − ξl , t (0) = Gi, t Zl , t β + Xi, t γ + ξl , t (1) − ξl , t (0) . (4)
This conditional probability structure captures the microfoundations
of the economic model we wish to study. This formulation is in fact a rela-
tively simple behavioral model, in that we ignore issues such as (1) selection
into and out of the population generated by the dynamics of incarcera-
tion and (2) those aspects of a crime decision at t in which a choice is a
single component in a sequence of decisions that collectively determine an
individual’s utility; that is, a more general preference specification is one
in which agents make decisions to maximize a weighted average of current
and future utility, accounting for the intertemporal effects of their deci-
sions in each period. While the introduction of dynamic considerations

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
into the choice problem raises numerous issues, such as state dependence,
heterogeneity, and dynamic selection, these can in principle be dealt with
using the analysis of Heckman and Navarro (2007), albeit at the expense
of considerable complication of the analysis.
AGGREGATION
How do the conditional crime probabilities for individuals described
by (4) aggregate within a location? Let rl,t denote the realized crime rate
in locality l at time t. Notice that we define the crime rate as the percent-
age of individuals committing crimes, not the number of crimes per se, so
we are ignoring multiple acts by a single criminal. Given our assumptions,
for the location-specific choice model (4), if individuals are constrained to
commit crimes in the location of residence, then the aggregate crime rate in
a locality is determined by integrating over the observable individual-spe-
cific heterogeneity in the location’s population. Let FX denote the empirical
l ,t
distribution function of Xi,t within l. The expected crime rate in a location
at a given time is
( ) ( )
E ρl , t Zl , t , FX , ξl , t (1) − ξl , t (0) = ∫ Gi, t Zl , t β + Xγ + ξl , t (1) − ξl , t (0) dFX (5)
l ,t l ,t
In order to convert this aggregate relationship into a linear regres-
sion form, it is necessary to further restrict the distribution function Gi,t.
Suppose that the associated probability densities dGi,t are uniform; a uni-
form density produces what is known as a linear probability model for the
individual choices. In this case, the crime rate in locality l at time t obeys
ρl ,t = Zl ,t β + X l ,t γ + ξl ,t (1) − ξl ,t (0) + θl ,t , (6)
where is the empirical mean of w ithin a nd
X l ,t X i,t l
( )
θl ,t = ρl ,t − E ρl ,t Zl ,t , FX , ξl ,t (1) − ξl ,t (0) captures the difference between
l ,t
the realized and expected crime rate in a locality. This is the model typically
employed in aggregate crime regressions.
Our construction of equation (6) from choice-based foundations illus-
trates how standard aggregate crime regressions require a number of statis-
tical assumptions if they are to be interpreted as aggregations of individual
behavior. The assumption of a uniform density for the individual specific
heterogeneity is of concern; in order to ensure that the probabilities of each
choice are bounded between 0 and 1, the support of the uniform density
may need to be agent-specific.2 Unfortunately, other random utility speci-
2 See Aldrich and Nelson (1984, Chapter 1) for an accessible discussion of the problems of
the linear probability model.

OCR for page 211

UNDERSTANDING CRIME TRENDS
fications do not aggregate in a straightforward manner. To illustrate the
()
problem, note that if one assumes that ε i,t ω i,t has a type-I extreme value
distribution, which is the implicit assumption in the logit binary choice
( )
Pr ω = 1 Z , X , ξ (1) − ξ (0)
i ,t i ,t l ,t i ,t l ,t l ,t
model, then log will be linear in
( )
1 − Pr ω = 1 Z , X , ξ (1) − ξ (0)
i ,t i ,t l ,t i ,t l ,t l ,t
the various payoff components but will not produce a closed form solution
for the aggregate crime rate. Methods are available to allow for analysis
of aggregate data under logit type assumptions (see Berry, Levinsohn, and
Pakes, 1995) but have not been applied, as far as we know, to the crime
context.
On its own terms, our development of a linear crime regression indi-
cates how aggregation affects the consistency of particular estimators. While
we have assumed that the individual-specific unobserved and observed
determinants of crime choices are independent, we have not made an
()
analogous assumption on the location-specific unobservables ξl ,t ω i,t . In
the aggregate regression, these may be correlated with either the aggregate
observables that appear in the utility function Zl,t or those variables that
appear as a consequence of aggregation X l ,t . From the perspective of
theorizing about individual behavior, there is no reason why the regression
residual ξl ,t (1) − ξl ,t (0) + θl ,t should be orthogonal to any of the regressors
in equation (6). By implication, this means that all the variables in equation
(6) should be instrumented. Hence in our judgment the focus on instru-
menting endogenous regressors that one finds in empirical crime analyses
is often insufficient, in that, while this strategy addresses endogeneity, it
does not address unobserved location-specific heterogeneity. Notice that
if individual-level data were available, this problem would not arise, since
one would normally allow for location-specific, time-specific, and location-
time-specific fixed effects for a panel.
COUNTERFACTUAL ANALYSIS
How can an aggregate crime regression be used to evaluate counter-
factuals such as a change in policy? Given our choice-theoretic framework,
a counterfactual analysis may be understood as a comparison of choices
under alternative policy regimes A and B. The net utility to the commission
of a crime will depend on the regime, so that
ν iAt = ZlAt β A + XiAt γ A + ξlAt (1) − ξlAt (0) + ε iAt (1) − ε iAt (0) (7)
, , , , , , ,
and

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
ν iBt = ZlBt β B + XiBt γ B + ξlBt (1) − ξlBt (0) + ε iBt (1) − ε iBt (0) (8)
, , , , , , ,
respectively. The net utility to individual i of committing a crime equals
ν i ,t =
ZlAt β A + XiAt γ A + ξlAt (1) − ξlAt (0) + ε iAt (1) − ε iAt (0) +
, , , , , ,
( ) ( )
Dl ,t ZlBt β B − ZlAt β A + Dl ,t XiBt γ B − XiAt γ A + (9)
, , , ,
( ))
(
Dl ,t ξlBt (1) − ξlBt (0) − ξlAt (1) − ξlAt (0) +
, , , ,
D (ε (0 ))) .
(0 ) − ( ε
(1) − εiBt (1) − εiAt
B A
l ,t i ,t , i ,t ,
where Dl,t = 1 if regime B applies to locality l at t; 0 otherwise. The analo-
gous linear aggregate crime rate regression is
ρl , t =
( )
+ Dl , t ZlBt β B − ZlAt β A + Dl , t X l , t γ B − X l , t γ A +
A B A
(10)
ZlAt β A + X l,t γ A
, , ,
( )
( )
ξlAt (1) − ξlAt (0) + θlAt + Dl , t ξlBt (1) − ξlBt (0) − ξlAt (1) − ξlAt (0) + θlBt − θlAt ..
, , , , , , , , ,
The standard approach measuring how different policies affect the
crime rate, in this case regimes A versus B, is to embody the policy change
in ZlAt versus ZlBt and to assume that all model parameters are con-
, ,
stant across regimes. This allows the policy effect to be measured by
( )
ZlBt − ZlAt β . Equation (10) indicates how a number of assumptions are
, ,
embedded in the standard approach, in particular the requirement that
( )
ξlBt (1) − ξlBt (0) − ξlAt (1) − ξlAt (0) = 0 , that is, that the change of regime does
, , , ,
not change the location-specific unobserved utility differential between
committing a crime and not doing so. This requirement seems problematic,
as it means that the researcher must be willing to assume that the regime
change is fully measured by the changes in X l ,t and Zl,t. Changes in the
detection probabilities and penalties for crimes typically come in bundles,
and we argue below that there are cases, specifically capital punishment,
in which this does not receive adequate attention in the relevant empirical
formulations.
MODEL UNCERTAINTY
Our derivation of aggregate crime rates from microfoundations assumed
that the researcher had strong prior information about the individual deci-
sion process. Put differently, our derivation of an aggregate crime regression

OCR for page 211

UNDERSTANDING CRIME TRENDS
was based on certainty about the underlying model of criminal behavior.
In this section, we discuss ways to relax this assumption, that is, we con-
sider the case of model uncertainty. In raising this, we emphasize that the
problem of inadequate attention to model uncertainty is in no way unique
to criminology. Nor do we mean to suggest that criminological studies are
unique in the extent to which authors fail to investigate how modifications
in baseline models affect inferences.
Characterizing Model Uncertainty
Our reading of the criminology literature suggests several general
sources of model uncertainty. The categories we describe have previously
been proposed by Brock, Durlauf, and West (2003) for economic growth
models and Brock, Durlauf, and West (2007) for business cycle models.
These categories are meant to identify general types of model uncertainty
that are common in social science analyses. At the same time, our decompo-
sition of model uncertainty is not unique; one can well imagine alternative
divisions.
Theory Uncertainty
Social science theories for a given phenomenon are often open-ended
(Brock and Durlauf, 2001), which means that one theory does not logically
exclude another as having additional explanatory power. Hence there is
often no justification for focusing on a subset of plausible explanations in
empirical work. Some evidence of why this matters is suggested by Levitt’s
(2004) evaluation of sources of the crime decline of the 1990s. Levitt iden-
tifies 10 alternative theories of the crime decline, all of which are mutually
consistent. Without questioning any of his substantive conclusions, we do
note that Levitt is to a large extent forced to evaluate the roles of the dif-
ferent theories based on studies that, typically, do not account for the full
range of the competing explanations when measuring the empirical salience
of a particular one.
Statistical Instantiation
Models may differ with respect to details of statistical specification that
have nothing to do with the underlying social science theories that moti-
vate them, but rather are employed in order to translate these theories into
representations that are amenable to data analysis. This is typically so even
when the social science theories are themselves expressed mathematically.
Differences in these assumptions can lead to different findings.
A good example of how differences in statistical assumptions can

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
affect substantive conclusions is specification of time trends. In the context
of the deterrence effects of shall-issue concealed weapons carry laws, dif-
ferent time trend choices have proven to be important. Specifically, Black
and Nagin (1998) found that the use of quadratic time trends in place of
state-specific linear time trends eliminates the evidence of a link between
liberalization of concealed weapons laws and crime rates found in Lott and
Mustard (1997). Lott’s rejoinder (1998) argues that it is hard to identify the
effects of a policy change (in this case, concealed weapons legality) because
a quadratic trend will mask it; intuitively, if crime is rising before a law is
passed and decreases thereafter, this will be approximated by the quadratic
trend.3 Lott’s intuition may be reasonable, but his argument is question
begging, as it applies in both directions. If crime follows an exogenously
determined quadratic trend over some time interval and rising crime levels
lead to a change in legislation, then Lott’s approach will spuriously identify
a causal effect from the legislation. This is true even if state-specific trends
are employed.
From the perspective of model uncertainty, Black and Nagin and Lott
are working with different statistical instantiations of unexplained temporal
heterogeneity. Under the Black and Nagin specification, there may be, as
Lott argues, substantial collinearity between the variable used to measure
temporal heterogeneity and the variables used to measure the effects of
shall-issue concealed weapons legislation. This multicollinearity does not
invalidate the Black and Nagin model on logical grounds. In our judgment,
the differences between Black and Nagin and Lott on this issue reflect the
absence of good explanations for much of the temporal evolution of crime
rates. Neither a linear specification nor a quadratic specification (or for
that matter, more flexible splines or alternative semiparametric methods)
instantiate substantive ideas about the crime process. Rather, they con-
stitute efforts to purge the data so that the residual components may be
analyzed.
Trend specification also matters in the analysis of unemployment rates
and crime. Greenberg (2001) criticizes Cantor and Land (1985) for model-
ing trends using deterministic rather than unit root methods. Again, social
science theory does not dictate a preference for one type of trend over
another. While both Greenberg and Cantor suggest justifications in favor
of their trend specifications that derive from individual behavioral determi-
nants, neither of them demonstrates a one-to-one or even precise mapping
from these determinants to their statistical modeling assumptions.
Other examples of this type of model uncertainty include assumptions
about additivity, linearity, and the use of logarithms versus levels.
3 This argument is further developed in Plassmann and Whitley (2003).

OCR for page 211

0 UNDERSTANDING CRIME TRENDS
Parameter Heterogeneity
A third type of model uncertainty concerns parameter heterogeneity.
Researchers often disagree on whether or not observations are simply draws
from a common data-generating process, so that any heterogeneity in the
observations derives from differences in values of some set of observable
control variables and different realizations of the model errors. Social sci-
ence theory typically does not impose that parameters are constant across
observations. For example, the argument that there is a deterrent effect
from a given penalty does not imply that the effect is independent of the
geographical unit in which the penalty is present. Parameter heterogene-
ity may be linked to deep questions about the interpretation of statistical
models; see Brock and Durlauf (2001) for a discussion of parameter hetero-
geneity and the concept of exchangeability of observations. Exchangeability,
roughly speaking, captures the idea that observations, such as state-specific
crime rates, may be treated as draws from a common statistical process.
One example of sensitivity of empirical claims to assumptions about
parameter heterogeneity is again found in the controversy between Black
and Nagin and Mustard and Lott. Black and Nagin found that evidence
of crime reductions associated with shall-issue laws are sensitive to the
presence of Florida in the dataset. They found that eliminating data from
Florida eliminated the evidentiary support for a handgun-crime link from
some of the Lott and Mustard specifications.
Another example appears in the capital punishment literature.
Donohue and Wolfers (2005) challenge findings of Dezhbakhsh, Rubin,
and Shepherd (2003) on the grounds that the findings are not robust to the
exclusion of California and Texas. As argued by Cohen-Cole et al. (2008),
this disagreement may be understood as a disagreement about parameter
homogeneity.
Model Averaging
How can the dependence of empirical claims on model specification be
constructively addressed? We describe a strategy based on model averag-
ing; ideas associated with model averaging appear to originate in Leamer
(1978). They have become prominent in the past decade in statistics; a valu-
able conceptual argument is made in Draper (1995), and the development
of formal methods has been greatly advanced by Raftery (e.g., Raftery,
Madigan, and Hoeting, 1997). We proceed using Bayesian language for
expositional convenience, although the analysis can be done using frequen-
tist estimators.
For a given exercise, suppose that the objective of the researcher is to
construct a conditional density of crime rates rl,t+ based on data Dt and

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
( )
model m, that is, Pr ρl ,t +1 Dt , m . Many disagreements about substantive
empirical questions, such as forecasts or the effects of alternative policies,
derive from disagreements about the choice of model m. This is, of course,
why model selection plays such a significant role in empirical work. From
the perspective of some empirical questions, it is not obvious that this is the
appropriate role for model choice. If the goal of an exercise is to compare
policies, the model choice is a nuisance parameter. Similarly, if one wants to
construct a forecast, then the model itself is not intrinsically interesting.
In order to avoid dependence on a particular model specification, an
alternative strategy is to develop conclusions based on a space of candidate
models; denote this space as M. Probability statements about a future out-
come such as rl,t+ can then be constructed conditioning on the entire model
space rather than on one of its elements. In other words, one computes the
( )
probability density Pr ρl ,t +1 Dt , M , which is the conditional density of
the crime rate given the data and a model space. From this perspective, the
true model is an unknown that needs to be integrated out of the probability
density. Formally,
( ) ∑ Pr ( ρ )( )
Pr ρl , t +1 Dt , M = Dt , m Pr m Dt .. (11)
l , t +1
m ∈M
( )
Here Pr m Dt denotes the posterior probability that m is the correct
model given the data. Conditioning on M means that the analyst knows
which models comprise M. Intuitively, one constructs probability state-
ments about an outcome, such as a crime rate, based on aggregating the
information available across each of the models under consideration. This
aggregation places greater weight on models that are more likely, as mea-
( )
sured by Pr m Dt . The linear structure in equation (11) derives from the
law of conditional probability, hence the term averaging.
Model averaging is emerging as a common methodology in econom-
ics; its increasing popularity reflects a combination of improved computa-
tional capacity and theoretical advances. The approach has been used to
study economic growth (Brock, Durlauf, and West, 2003; Doppelhofer,
Miller, and Sala-i-Martin, 2004; Fernandez, Ley, and Steel, 2001), finance
(Avramov, 2002), forecasting (Garratt et al., 2003), and monetary policy
(Brock, Durlauf, and West, 2003). An application to a crime context, the
deterrent effect of capital punishment, is Cohen-Cole et al. (2008). While
we regard model-averaging methods as very promising, we also emphasize
that the methodology is still being developed and a number of outstand-
ing theoretical questions still exist.4 And of course, model averaging still
4 One issue concerning model priors that is worth noting concerns the assignment of priors
to similar models. Most of the model-averaging literature has employed diffuse priors, that is,
all models are assigned equal prior weights. However, it can be the case that some models in

OCR for page 211

UNDERSTANDING CRIME TRENDS
deterrence in the sample studied by Dezhbakhsh, Rubin, and Shepherd is
weak.
Policy-Relevant Calculations
Following our general discussion, the statistical significance of the
capital punishment variables in a murder regression does not produce
the appropriate information needed to make policy comparisons. This
has implications for the way such evidence is employed in death penalty
debates. Sunstein and Vermeule (2005) argue that evidence of a deterrent
effect can produce a moral case for capital punishment, in that the decision
of a government to fail to implement a life-saving policy is equivalent to
the decision to implement a policy that costs lives.
Sunstein and Vermeule (2005) develop their argument conditioning on
evidence of a deterrence effect. Leaving aside the insouciance with which
they treat the empirical literature,7 their argument lacks attention to the
appropriate nature of the policy maker’s loss function and the nature of the
uncertainty of the empirical evidence.
The Sunstein and Vermeule analysis treats the expected number of lives
saved as the variable of interest to the policy maker; in Dezhbakhsh, Rubin,
and Shepherd, this value is a function of the estimated parameter bE in (20).
The expected number of lives saved is not necessarily sufficient in describing
a policy maker’s utility function, even if this function is a monotonically
increasing function of the number of lives saved. As such, their attention
to this figure is analogous to making a utilitarian as opposed to a welfarist
calculation (see Sen, 1979). While Sunstein and Vermeule would presum-
ably respond that they are assuming that the precision associated with
estimates of the expected number of lives saved is high, precision needs to
be defined with respect to the policy maker’s utility function. It is not an
independent object.
The sensitivity of deterrence evidence to model choice, as demon-
strated by Donohue and Wolfers and extended in Cohen-Cole et al. (2008),
raises the issues we have discussed with respect to decision making under
ambiguity and the evaluation of policies when one does not wish to base
them on a choice of model priors. Without a justification of the choice
of priors, there is no expected deterrence effect on which Sunstein and
Vermeule can even rely. Our impression of the philosophy literature is that
7 At the same time they also state that
“The foundation of our argument is a large and growing body of evidence that
capital punishment may well have a deterrent effect, possibly a quite powerful
one. . . . The particular numbers do not much matter” (p. 706).

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
the issue of policy evaluation under ambiguity has generally not been dis-
cussed, although Gaus (2006) makes an interesting argument in favor of
following principles rather than expected-effect calculations when assessing
policies, the effects of which are associated with substantial uncertainty.
To be clear, none of this means that Sunstein and Vermeule (2005) are
incorrect in their conclusions about the ethical implications of a certain
deterrent effect for a policy maker or that the death penalty is either moral
or immoral per se. Rather, our claim is that the policy implications of the
uncertainty associated with deterrence effects cannot be assessed outside of
the policy maker’s preferences.
Right-to-Carry Laws and Crime: Firearms and Violence Revisited
Our third example is the controversy over the effects of shall-issue
concealed weapons laws in the National Academies report Firearms
and Violence (National Research Council, 2005). This report concluded
(pp. 150-151):
with the current evidence it is not possible to determine that there is a
causal link between the right-to-carry laws and crime rates. It is also the
committee’s view that additional analysis along the lines of the current
literature is unlikely to yield results that will persuasively demonstrate a
causal link between right-to-carry laws and crime rates (unless substantial
numbers of states were to adopt or repeal right-to-carry laws), because of
the sensitivity of the results to model specification.
Committee member James Q. Wilson dissented from this part of the
study, on the grounds that the sensitivity to specification found in the report
did not account for the sensibility of different models; in particular, he ques-
tioned whether the failure of models that excluded socioeconomic control
variables to find deterrent effects was of importance in assessing the deter-
rent effect. Wilson observes (National Research Council, 2005, p. 270):
Suppose Professor Jones wrote a paper saying that increasing the number
of police in a city reduced the crime rate and Professor Smith wrote a
rival paper saying that cities with few police officers have low crime rates.
Suppose that neither Smith nor Jones used any control variables, such as
income, unemployment, population density, or the frequency with which
offenders are sent to prison in reaching their conclusions. If such papers
were published, they would be rejected out of hand by the committee for
the obvious reason that they failed to supply a complete account of the
factors that affect the crime rate.

OCR for page 211

UNDERSTANDING CRIME TRENDS
The committee’s rejoinder to Wilson argued (National Research Council,
2005, pp. 273-274):
Everyone (including Wilson and the rest of the committee) agrees that
control variables matter, but there is disagreement on the correct set. Thus,
the facts that there is no way to statistically test for the correct specifica-
tion and that researchers using reasonable specifications find different
answers are highly relevant. Given the existing data and methods, the rest
of the committee sees little hope of resolving this fundamental statistical
problem.
We believe that this conclusion is too pessimistic. The disagreement
between Wilson and the rest of the National Academies committee reflects
the absence in the report of an explicit evaluation of how model uncertainty
interacts with evidence of shall-issue laws. While the assertion that it is
impossible to statistically identify the correct specification of a statistical
model is true at some level of generality (although the report is frankly
unclear on what is meant by this), this argument is hardly novel; it is known
in the philosophy literature as the Duhem-Quine hypothesis (Quine, 1951,
is the classic statement) and refers to the idea that all theories are undeter-
mined by available data.
At this level of generality the National Academies committee claim is
an uninteresting observation with respect to social science research, since
it begs the question of the relative plausibility of assumptions.8 For the
dispute at hand, we believe that Wilson is correct in his argument that a
model whose specification includes controls suggested by social science
theory should receive greater weight than one that does not. Furthermore,
these two models are statistically distinguishable. To conclude that one
should regard evidence of a deterrent effect as persuasive only if both
models produce the same findings makes little sense. The report implicitly
suggests that the models without control variables are intrinsically interest-
ing: “No link between right-to-carry laws and changes in crime is appar-
ent in the raw data . . . ; it is only once numerous covariates are included
that the . . . effects . . . emerge” (p. 150). This remark ignores the classic
Simpson’s paradox, in which a bivariate relationship has one direction,
whereas a multivariate relationship does not. The standard example of
Simpson’s paradox is the positive relationship between admission to the
hospital and the probability of death.
8 The report’s suggestion that randomized experiments represent the gold standard for
research ignores the assumptions required for their conduct—integrity of the researcher, accu-
racy of data collection, etc. An advocate of randomized experiments would presumably dismiss
concerns about such factors as implausible—but this is precisely our point.

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
Model averaging provides a natural way of integrating the information
across the alternative specifications considered in the National Academies
report. As we see it, the committee could have addressed the sensitivity
of shall-issue deterrence effects by constructing a set of specifications that
included those found in the literature as well as others that are formed by
combining the assumptions underlying these models. Intuitively, one thinks
of the assumptions that differentiate models as the axes of the model space,
and one fills the model space out with those combinations of assumptions
that are coherent with one another. Averaging over this space would have
integrated the information in the different models and indicated whether
evidence of a shall-issue deterrent effect is present when one conditions on
a model space rather than a particular model.
One answer to our advocacy of model averaging as a tool to address
model uncertainty of the type facing the National Academies committee
is that a given body of empirical studies captures only a small fraction of
the universe of potential models (and indeed might represent a measure 0
set). This is certainly a tenable position. But if this position is taken, then
it would be irrelevant whether a given body of studies produced similar
or conflicting results. If it is then claimed that the degree of consistency
in results across models contained in a subspace is informative about the
results that would be ascertained were the model space expanded, then it
is difficult to see why the relative prior plausibility and relative evidentiary
support within an initial model space are not informative as well.
A second answer to the use of model averaging might rely on the
absence of a principled basis for assigning prior model probabilities. We
are certainly sympathetic to this view. But if this position is taken, then the
implications of the body of model-specific findings of an effect of shall-
issue laws to policy need to be explicitly considered. It is not obvious, for
example, that the fragility that the National Academies report claims to
be present in concealed weapons regressions is even an argument against
the laws. Suppose that a policy maker possesses minimax preferences with
respect to model uncertainty. Fragility of deterrence evidence does not
logically lead to rejection of the policy; one needs to know the payoffs
under the different models under consideration. The National Academies
report seems to take the position that, in absence of strong evidence that
the laws reduce crime, they should not be implemented. But minimax
preferences do not, by themselves, generate this conclusion, which really is
based on the presumption that the law should not be implemented unless
there is compelling evidence of crime reduction. This line of reasoning can
be justified (e.g., Brock, Durlauf, and West, 2003), but it requires context-
specific argumentation.
Therefore, a recommendation we make for policy evaluation studies

OCR for page 211

UNDERSTANDING CRIME TRENDS
such as Firearms and Violence is that claims about the robustness or
fragility of various findings be evaluated with respect to different loss func-
tions, with particular attention to minimax and minimax regret calculations
as supplements to the standard Bayesian ones.
SHOULD AGGREGATE CRIME REGRESSIONS BE ABANDONED?
One response to the discussion in this paper would be to search
for alternative ways of uncovering aggregate criminological facts. The
critiques we have raised are part of the source of interest in so-called
natural experiments, in which an exogenous event of some type allows
a comparison of aggregate crime outcomes (see Levitt, 1996, for a nice
example). In his appendix to the Firearms and Violence study, Horowitz
(2005) makes a broad general argument against the use of regression
models to elucidate the determinants of crime, specifically in terms of
evaluating policy effects.
While his focus is on concealed weapons laws, his claims apply with
equal force to other crime contexts. According to Horowitz, “In summary,
the problems posed by high-dimensional estimation, misspecified models,
and lack of correct knowledge of the correct set of explanatory variables
seem insurmountable with observational data” (National Research Council,
2005, p. 308). In contrast, he argues that random assignment of policies
could in principle reveal their effects; in particular, he discusses how random
assignment can allow for the estimation of average treatment effects (a par-
ticular piece of legislation, such as shall-issue concealed weapons laws, is
an example of a treatment).
We of course concur that there does not exist an algorithm to infallibly
identify the “true” model of crime (or for that matter, other phenomena)
when the universe of candidate models is broad enough. However, we do
not believe this means that crime regressions cannot be informative about
policy. Different models have both different ex ante levels of plausibility
and ex post levels of goodness of fit for a given body of observational
data. The different concealed weapons regressions with and without socio-
economic controls are not equally ex ante plausible, given the state of social
science. And we do not know, given our priors, how the relative goodness
of fit of the different models analyzed in the National Academies report
would translate into different posterior model probabilities.
Our discussion of the assumptions that underlie the interpretation
of aggregate crime regressions may all be interpreted as examples for
Horowitz’s arguments about the limitations of regression analysis of crime.
We do not claim to have an answer to the question of how to integrate
the different types of model uncertainty we have discussed into a single
integrated framework, let alone introduce such factors as the extension of

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
the basic crime model to intertemporal decision making. Our disagreement
with Horowitz is that we see a role for empirical models in informing policy
discussion, even though the researcher is aware of untestable or unappeal-
ing assumptions underlying them. The way in which models are used to
inform beliefs necessarily requires judgments; this necessity does not mean
that the models are uninformative. A researcher brings a body of social
science and statistical knowledge to bear in the assessment of empirical
results; this knowledge matters in assessing the dependence of a result on
an assumption. Put differently, not all assumptions are equally arbitrary.
The need for assumptions is not unique to regression analysis with obser-
vational data; all empirical work is theory-laden (to use Quine’s phrase). An
experiment of the type proposed by Horowitz with respect to shall-issue
weapons permit laws—randomized legalization across states—would, if one
is to use the findings to inform policy makers, require assumptions about
(1) the degree to which potential criminals can alter the locations in which
crimes are committed, (2) the nature of migration by potential criminals
across state boundaries both before the experiment and in response to it,
(3) the effect on the current crime choices of potential criminals of the
knowledge that an experiment that may affect future laws in their state of
residence is being conducted, etc. Also, the translation of findings from such
an experiment into a recommendation for those states that did not imple-
ment the policy requires exchangeability assumptions on the states. Does
one assume that the deterrent effect of the law is identical across states?
If state-level deterrent effects are heterogeneous, how is this heterogeneity
to be modeled—via random effects, varying coefficients, or some other
method?9 Randomized experiments cannot avoid the need for judgments;
as described in detail in Heckman (2000, 2005), judgment is intrinsic to
social scientific inquiry.
Overall, we do not see good reasons to regard natural experiments as
superior to regressions with observational data in terms of their relative
utility as means of understanding crime.10 It is straightforward to construct
examples in which one methodology can provide insights that the other
does not. Each has a contribution to make in criminological research.
9 Abbring and Heckman (2007) provide a comprehensive overview of the assumptions
required in developing estimates of treatment effects that account for considerations of the
type hinted at in our discussion.
10 See Heckman (2005) and Manski (2007) for discussion of the limitations of experiments;
Heckman and Navarro (2004) compare the strengths and weaknesses of different empirical
strategies for uncovering the determinants of individual choice.

OCR for page 211

UNDERSTANDING CRIME TRENDS
CONCLUSION
In this chapter, we have described some issues we regard as important
in the econometric study of crime: microfoundations, aggregation, counter-
factual analysis, and policy evaluation. We have tried to make clear the
various assumptions that must be maintained to interpret aggregate crime
regressions with respect to individual behavior and have emphasized how
standard uses of these regressions to evaluate policy presuppose a number
of assumptions. In light of disagreements about these assumptions, which
ultimately underlie claims of fragility or robustness of an empirical result,
we have outlined some ways of using model-averaging methods and statisti-
cal decision theory to make progress. Throughout, we have emphasized the
role of judgment in empirical work, for which no algorithm exists.
ACKNOWLEDGMENTS
We thank the National Science Foundation and University of Wisconsin
Graduate School for financial support. Arthur Goldberger and Justin
Wolfers provided immensely helpful comments on a previous draft.
REFERENCES
Abbring, Jaap H., and James J. Heckman. (2007). Econometric evaluation of social programs,
part III: Distributional treatment effects, dynamic treatment effects, dynamic discrete
choice, and general equilibrium policy evaluation. In J. Heckman and E. Leamer (Eds.),
Handbook of econometrics, olume . Amsterdam: Elsevier.
Aldrich, John H., and Forrest D. Nelson. (1984). Linear probability, logit, and probit models.
Beverly Hills, CA: Sage.
Avramov, Doron. (2002). Stock return predictability and model uncertainty. Journal of
Finance, , 423-458.
Ayres, Ian, and John Donohue, III. (2003). Shooting down the “more guns, less crime”
hypothesis. Stanford Law Reiew, , 1193-1312.
Becker, Gary S. (1968). Crime and punishment: An economic analysis. Journal of Political
Economy, (2), 169-217.
Becker, Gary S. (1993). Nobel lecture: The economic way of looking at behavior. Journal of
Political Economy, 0(3), 385-409.
Bernard, Andrew B., and Steven N. Durlauf. (1995). Convergence in international output.
Journal of Applied Econometrics, 0(2), 97-108.
Bernard, Andrew B., and Steven N. Durlauf. (1996). Interpreting tests of the convergence
hypothesis. Journal of Econometrics, (1-2), 161-173.
Berry, Steven, James Levinsohn, and Ariel Pakes. (1995). Automobile prices in market equi-
librium. Econometrica, (4), 841-890.
Black, Dan A., and Daniel S. Nagin. (1998). Do right-to-carry laws deter violent crimes?
Journal of Legal Studies, (1), 209-219.
Blundell, Richard, and Thomas Stoker. (2005). Heterogeneity and aggregation. Journal of
Economic Literature, (2), 347-391.

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
Brock, William A., and Steven Durlauf. (2001). Growth economics and reality. World Bank
Economic Reiew, , 229-272.
Brock, William A., Steven Durlauf, and Kenneth West. (2003). Policy analysis in uncertain
economic environments (with discussion). Brookings Papers on Economic Actiity, ,
235-322.
Brock, William A., Steven Durlauf, and Kenneth West. (2007). Model uncertainty and policy
evaluation: Some theory and empirics. Journal of Econometrics, (2), 629-664.
Brock, William A., Steven Durlauf, James Nason, and Giacomo Rondina. (2007). Simple versus
optimal rules as guides to policy. Journal of Monetary Economics, (5), 1372-1396.
Cantor, David, and Kenneth C. Land. (1985). Unemployment and crime rates in the post-
World War II United States: A theoretical and empirical investigation. American Socio-
logical Reiew, 0, 317-332.
Cohen-Cole, Ethan, Steven Durlauf, Jeffrey Fagan, and Daniel Nagin. (2008). Model uncer-
tainty and the deterrent effect of capital punishment. American Law and Economics
Reiew, forthcoming.
Dezhbakhsh, Hashem, Paul Rubin, and Joanna M. Shepherd. (2003). Does capital punishment
have a deterrent effect? New evidence from post-moratorium panel data. American Law
and Economics Reiew, (2), 344-376.
Donohue, John, and Steven Levitt. (2001). The impact of legalized abortion on crime.
Quarterly Journal of Economics, (2), 379-420.
Donohue, John, and Justin Wolfers. (2005). Uses and abuses of empirical evidence in the death
penalty debate. Stanford Law Reiew, (3), 791-846.
Doppelhofer, Gernot, Ronald I. Miller, and Xavier Sala-i-Martin. (2004). Determinants
of long-term growth: A Bayesian averaging of classical estimates (BACE) approach.
American Economic Reiew, (4), 813-835.
Draper, David. (1995). Assessment and propagation of model uncertainty. Journal of the Royal
Statistical Society, Series B, , 45-70.
Durlauf, Steven, Salvador Navarro, and David Rivers. (2008). Notes on the econometrics
of deterrence. Unpublished working paper, Department of Economics, University of
Wisconsin, Madison.
Ehrlich, Isaac. (1972). The deterrent effect of criminal law enforcement. Journal of Legal
Studies, (2), 259-276.
Ehrlich, Isaac. (1973). Participation in illegal activities: A theoretical and empirical investiga-
tion. Journal of Political Economy, (3), 521-565.
Fernandez, Carmen, Eduardo Ley, and Mark Steel. (2001). Model uncertainty in cross-country
growth regressions. Journal of Applied Econometrics, (5), 563-576.
Garratt, Anthony, Kevin Lee, M. Hashem Pesaran, and Yongcheol Shin. (2003). Forecasting
uncertainties in macroeconometric modelling: An application to the UK economy. Journal
of the American Statistical Association, (464), 829-838.
Gaus, Gerald F. (2006). Social complexity and eoled moral principles. Unpublished paper,
University of Arizona.
Gelman, Andrew, James S. Liebman, Valerie West, and Alexander Kiss. (2004). A broken sys-
tem: The persistent patterns of reversals of death sentences in the United States. Journal
of Empirical Legal Studies, (2), 209-261.
Goldberger, Arthur S. (1991). A course in econometrics. Cambridge, MA: Harvard University
Press.
Greenberg, David. (2001). Time series analysis of crime rates. Journal of Quantitatie Crimi-
nology, (4), 291-327.
Hansen, Lars Peter, and Thomas J. Sargent. (2007). Robustness. Princeton, NJ: Princeton
University Press.

OCR for page 211

0 UNDERSTANDING CRIME TRENDS
Heckman, James L. (2000). Causal parameters and policy analysis in economics: A twentieth
century retrospective. Quarterly Journal of Economics, (1), 45-97.
Heckman, James L. (2005). The scientific model of causality. Sociological Methodology, ,
1-98.
Heckman, James L., and Salvador Navarro. (2004). Using matching, instrumental variables,
and control functions to estimate economic choice models. Reiew of Economics and
Statistics, (1), 30-57.
Heckman, James L., and Salvador Navarro. (2007). Dynamic discrete choices and dynamic
treatment effects. Journal of Econometrics, (2), 341-396.
Horowitz, Joel L. (2005). Statistical issues in the evaluation of right-to-carry laws. In National
Research Council, Firearms and iolence: A critical reiew (pp. 299-308), Committee
to Improve Research Information and Data on Firearms, Charles F. Wellford, John V.
Pepper, and Carol V. Petrie (Eds.), Committee on Law and Justice, Division of Behavioral
and Social Sciences and Education. Washington, DC: The National Academies Press.
LaFree, Gary. (2005). Evidence for elite convergence in cross-national homicide victimization
trends, 1956 to 2000. The Sociological Quarterly, , 191-211.
Leamer, Edward E. (1978). Specification searches. New York: John Wiley & Sons.
Levitt, Steven D. (1996). The effect of prison population size on crime rates: Evidence from
prison overcrowding litigation. Quarterly Journal of Economics, , 319-351.
Levitt, Steven D. (2004). Understanding why crime fell in the 1990s: Four factors that explain
the decline and six that do not. Journal of Economic Perspecties, (1), 163-190.
Lott, John R. (1998). The concealed-handgun debate. Journal of Legal Studies, (1),
221-291.
Lott, John R., and David B. Mustard. (1997). Crime, deterrence, and right-to-carry concealed
handguns. Journal of Legal Studies, (1), 1-68.
Mankiw, N. Gregory, David Romer, and David N. Weil. (1992). A contribution to the empirics
of economic growth. Quarterly Journal of Economics, 0(2), 407-437.
Manski, Charles F. (2005). Statistical treatment rules for heterogeneous populations. Econo-
metrica, , 1221-1246.
Manski, Charles F. (2006). Minimax-regret treatment choice with missing outcome data.
Journal of Econometrics, (1), 105-115.
Manski, Charles F. (2007). Identification for prediction and decision. Cambridge, MA:
Harvard University Press.
Marschak, Jacob. (1953). Economic measurements for policy and prediction. In William C.
Hood and Tjalling C. Koopmans (Eds.), Studies in econometric method. New York:
John Wiley & Sons.
Matzkin, Rosa L. (1992). Nonparametric and distribution-free estimation of the binary thresh-
old crossing and the binary choice models. Econometrica, 0(2), 239-270.
National Research Council. (2005). Firearms and iolence: A critical reiew. Committee
to Improve Research Information and Data on Firearms, Charles F. Wellford, John V.
Pepper, and Carol V. Petrie (Eds.), Committee on Law and Justice, Division of Behavioral
and Social Sciences and Education. Washington, DC: The National Academies Press.
Navarro, Salvador. (2007). Control functions. In Lawrence E. Blume and Steven Durlauf (Eds.),
New Palgrae dictionary of economics (revised ed.). London: Palgrave-Macmillan.
O’Brien, Robert M. (1999). Measuring the convergence/divergence of “serious crime” arrest
rates for males and females: 1960-1995. Journal of Quantitatie Criminology, (1),
97-114.
Plassmann, Florenz, and John Whitley. (2003). Confirming “more guns, less crime.” Stanford
Law Reiew, , 1313-1369.
Quine, Willard Van Orman. (1951). Two dogmas of empiricism. Philosophical Reiew, 0,
20-43.

OCR for page 211

ON THE USE OF AGGREGATE CRIME REGRESSIONS
Raftery, Adrian E., David Madigan, and Jennifer A. Hoeting. (1997). Bayesian model averag-
ing for linear regression models. Journal of the American Statistical Association, (437),
179-191.
Savage, Leonard J. (1951). The theory of statistical decision. Journal of the American Statisti-
cal Association, , 55-67.
Sen, Amartya. (1979). Utilitarianism and welfarism, Journal of Philosophy, (9), 463-489.
Stoker, Thomas M. (1993). Empirical approaches to the problem of aggregation over indi-
viduals. Journal of Economic Literature, (4), 1827-1874.
Sunstein, Cass R., and Adrian Vermeule. (2005). Is capital punishment morally required? Acts,
omissions and life-life tradeoffs. Stanford Law Reiew, (3), 703-751.
Wald, Abraham. (1950). Statistical decision functions. New York: John Wiley & Sons.

OCR for page 211