Committee Response to Wilson’s Dissent
This response addresses Professor Wilson’s dissent from one aspect of the committee report. It is important to stress at the outset that his dissent focuses on one part of one chapter of the report. Except for the effects of right-to-carry laws on homicide, the entire committee is in agreement on the material in Chapter 6 and the report overall. In particular, the committee, including Wilson, found that “it is impossible to draw strong conclusions from the existing literature on the causal impact” of right-to-carry laws on violent and property crime in general and rape, aggravated assault, auto theft, burglary, and larceny in particular.
The only substantive issue on which the committee differed is whether the existing research supports the conclusion that right-to-carry laws substantially reduce murder. The report suggests that the scientific evidence is inconclusive. Wilson disagreed, arguing that virtually every estimate shows a substantial and statistically significant negative effect of right-to-carry laws on murder.
While it is true that most of the reported estimates are negative, several are positive and many are statistically insignificant. In addition, when we use Lott’s trend model but restrict the out years to five years or less (Table 6-7), the trends for murder become positive and those for other crimes remain negative. Therefore, the key question is how to reconcile the contrary findings or, conversely, how to explain why these particular positive, or negative, findings should be dismissed. Three sets of results discussed more fully in Chapter 6 provide support for the committee’s conclusion: Published studies, the committee’s analysis of control variables, and the committee’s analysis extending the time period.
1. Published studies. There is no question that the empirical results on the effects of right-to-carry laws on murder (and other crimes) are sensitive to seemingly small variations in data and specification. Indeed, Wilson agrees that a few studies find positive effects of right-to-carry laws on murder. We cite four studies in Tables 6-3 and 6-4: Ayres and Donohue, Black and Nagin, Moody, and Plassmann and Tideman (cited in Chapter 6). There are almost certainly others not reported in these tables.
The rest of the committee and Wilson agree that fragility does not prove that the results of any specific paper are incorrect. However, some of the published results must be incorrect because they are inconsistent with one another. The important question, therefore, is whether the correct results can be identified. The rest of the committee thinks that they cannot. Contrary to Wilson’s claim, the committee did assess the existing body of empirical literature on right-to-carry laws (see the section beginning on page 127 and Tables 6-3 and 6-4). As described in the report, all of the empirical research on right-to-carry laws relies on the same conceptual and methodological ideas (page 121). Relative to the basic models estimated by Lott, some researchers used data from more counties and some from fewer; some used hybrid linear models while others used nonlinear specifications; some provide state-specific estimates while most provide a single national estimate; some added control variables while others used relatively parsimonious specifications; and so forth. All of the studies described in the literature review made plausible cases for their choices of models and data. Wilson seems to argue that a careful evaluation of the literature would reveal which paper or papers obtained correct results, but he does not suggest the evaluation criteria. The rest of the committee does not think that application of any scientific criteria to existing papers would identify the effects of right-to-carry laws on crime.
2. Committee control variable analysis. Chapter 6 shows that when the trend and dummy variable models do not include demographic and socioeconomic covariates (but do include year and county dummy variables) the estimates are relatively small, positive in one case (Table 6-6, Row 3), and statistically insignificant in all cases. Contrary to Wilson’s assertion, the chapter does not claim that this or any other specification is correct. Rather, this finding simply reveals that “detecting the effect, if any, of right-to-carry laws requires controlling for appropriate confounding variables.” In light of the fragility revealed in the literature, the fundamental issue is which set of covariates is sufficient to identify the effects of right-to-carry laws on homicide and other crimes. The importance of controlling for the correct set of covariates is well known. In fact, much of the debate between Lott and his statistically oriented critics focuses on determining the correct set of control variables. Everyone (including Wilson and the rest of the committee) agrees that control variables matter, but there is disagree-
ment on the correct set. Thus, the facts that there is no way to statistically test for the correct specification and that researchers using reasonable specifications find different answers are highly relevant. Given the existing data and methods, the rest of the committee sees little hope of resolving this fundamental statistical problem.
Furthermore, the example of the relationship between crime rates and policing in the dissent raises another problem. The usual way one proceeds in research is to estimate the relationship between two variables and if a significant relationship is found controls are introduced to test the relationship. As the dissent notes, these controls are selected based on reasonable theories and research. In this case, the bivariate relationship (between right to carry laws and crime) is small, positive in one case, and insignificant in all. This is not like the hypothesized conflicting bivariate findings in Wilson’s police example. Thus the selection of controls in the analysis of right-to-carry laws is as difficult as the committee contends
3. Committee trend model analysis. Wilson states that the trend model analysis in Table 6-7 estimates the effects of right-to-carry laws on a yearly basis, rather than a single trend.1 This is incorrect. The estimates reported in Table 6-7 are found using Lott’s trend model with restrictions on the number of postadoption years used in the analysis. If the model is correctly specified, this restriction should be inconsequential. However, we find substantial differences, especially for murder. In fact, when we restrict the number of postadoption years to five or fewer, the estimates switch from negative to positive. Thus, Model 6.2 appears to be misspecified. Moreover, despite Wilson’s assertion, these types of sensitivity test are commonly used in peer-reviewed journals and are suggested by Rosenbaum (2001) as a way to assess the robustness of an empirical model. Of course, results like those reported in Chapter 6 might often lead a paper to be rejected from a peer-reviewed journal.
Wilson further suggests that Lott’s findings may depend on the crime rate trends that changed dramatically over the course of the 1990s. All of the studies in this literature, however, attempt to control for trends in crime, and thus purport to reveal a time invariant effect of right-to-carry laws. If the effects vary by time, all of the existing models are misspecified.
In sum, we are encouraged that Professor Wilson agrees with the rest of the committee except for the specific conclusion regarding the effects of right-to-carry laws on murder. On this point, we find his arguments to be unconvincing and his summary of some parts of the chapter inaccurate. In our view the evidence on homicide is not noticeably different from that on other crimes evaluated in this literature and cannot be easily separated. If
Contrary to Wilson’s claim, the results in Table 6-7 all rely on models with covariates.
the effects of right-to-carry laws on violent and property crimes are ambiguous, as argued in Chapter 6, we see no reason why the same is not true of homicide. Professor Wilson may be correct on this matter—it is theoretically possible—but we maintain that the scientific evidence does not support his position.
Rosenbaum, P.R. 2001 Replicating effects and biases. American Statistician 55(3):223-227.