The Volokh Conspiracy -

John Lott Responds

to some posts (linked to at the end of this one) that criticize his work in light of the National Academy of Science report on gun control laws:

Last month, the National Academy of Sciences issued a 328-page report on gun control laws. The big news that has been ignored on all the blog sites is that the academy's panel couldn't identify any benefits of the decades-long effort to reduce crime and injury by restricting gun ownership. The only conclusion it could draw was: Let's study the question some more.

The panel has left us with two choices: Either academia and the government have wasted tens of millions of dollars and countless man-hours on useless research (and the panel would like us to spend more in the same worthless pursuit), or the National Academy is so completely unable to separate politics from its analyses that it simply can't accept the results for what they are.

Based on 253 journal articles, 99 books, 43 government publications, and some of its own empirical work, the panel couldn't identify a single gun control regulation that reduced violent crime, suicide or accidents.

From the assault weapons ban to the Brady Act to one-gun-a-month restrictions to gun locks, nothing worked. (Something that I have been the first person to investigate empirically for many of these laws, and I also had been unable to find evidence that they reduced violent crime.)

The study was not the work of gun-control opponents. The panel was set up during the Clinton administration, and of its members whose views on guns were publicly known before their appointments all but one had favored gun control. Something that I wrote up about the panel three years ago is still relevant.

While the panel dealt with a broad range of gun control issues, only one issue has received attention on different blogs: right-to-carry laws. In fact, the panel apparently originated with the desire from some to respond to the debate on that issue and to respond specifically to my research that concludes that allowing law abiding citizens to carry concealed weapons reduces crime. I originally overheard Phil Cook and Dan Nagin discussing the need for a panel to "deal with" me in the same way that an earlier panel had "dealt with Isaac" Ehrlich's work showing that the death penalty deterred murder. They agreed and Nagin said that he would talk to Al Blumstein about setting up such a panel. Needless to say, that is what ended up happening.

1) James Q. Wilson's very unusual dissent is very interesting (only two out of the last 236 reports over the last 10 years have carried a dissent). Wilson states that all the research provided "confirmation of the findings that shall-issue laws drive down the murder rate . . . " Wilson has been on four of these panels and never previously thought that it was necessary to write a dissent, including the previous panel that attacked Isaac Ehrlich's work showing that the death penalty represented a deterrent.

Wilson said that that panel's conclusion raises concerns given that "virtually every reanalysis done by the committee" confirmed right-to-carry laws reduced crime. He found the committee's only results that didn't confirm the drop in crime "quite puzzling." They accounted for "no control variables" - nothing on any of the social, demographic, and public policies that might affect crime. Furthermore, he didn't understand how evidence that was not publishabled in a peer-reviewed journal would be given such weight.

The non-results are basically due to dropping all the control variables (particularly the arrest rate which is not defined when the crime rate is zero). When that happens a lot of observations with zero crime rates are introduced. The problem with using OLS when you have all these zero crime rates is that if a crime rate is already zero, no matter how good the law is, it can't lower the crime rate any further. There is thus a positive bias in these results. Plassmann's two papers (his piece in the Journal of Law and Economics with Nic Tideman and his paper with Whitley in the Stanford Law Review) show how you can address this as a count data problem. Although his research consistently shows statistically significant results that shall issue laws reduce crime, the National Academy report ignores the research.

The panel's discussion of Duggan's results focuses on the regressions without any control variables and that use the OLS estimates when they have a large number of zero values for the crime rates.

2) As an interesting aside, there are a number of factual mistakes in the NAS report and those mistakes work against my findings. For example, Figure 6.1 makes a mistake where it shows the increase in violent crime of 7 percent in year one, when the amount is 5 percent (7-2, where 2 is from the trend). (Of course, the overall problem with the hybrid approach is discussed below.) There are significant drops in crime in Table 6-3 that are statistically significant, but they are not properly marked to indicate that is so. Even something trivial as the number of states currently with right-to-carry laws is wrong, 36 (not 34) (and if Minnesota is included the number is 37).

3) Last year there was a debate over the use of clustering between Ayres and Donohue and me, but the statements of the NAS panel corresponds extremely closely to what was written in my original paper with David Mustard.

4) p. 127: "We focus on the conflicting results . . ." No attempt is made to give readers an idea of the frequency or importance of unusual results. Take the results in Table 6-3. For Plassmann and Whitley, the panel doesn't mention that Plassmann and Whitley say that there are "major problems" with the particular regressions that the panel decides to report and more importantly that the effects in those regressions are biased towards zero (see point 2 above). For Moody's results, they show only two specifications of all the results that he reports and don't mention that the one weird result that he got was from a specification that he flagged as problematic and not controlling for other factors.

Even with the very selective sample of regressions that they pick, there is not one statistically significant bad effect of right-to-carry laws on murder. Only one case for robbery and that is one problematic specification from Ayres and Donohue.

5) Hybrid model. The so-called hybrid model used by Ayres and Donohue finds that the law dummy variable is positive while the trend variable indicates that crime rates decline over time. While Plassmann and Whitley do a good job explaining why the "hybrid" model produces misleading results and the panel never discusses their critique (looking at the crime rates on a year by year basis show no initial increase in crime), it still would have been useful for the panel to at least say whether the "hybrid" results produced a statistically significant temporary bad effect. The problem with determining statistical significance is that when both the dummy and trend variables are on at the same time, we are concerned about the net effect not just the dummy variable by itself as Ayres and Donohue argue. The answer for all those results in the panel's Table 6-4 is "no."

6) Reset tests. Professor Horowitz's discussion of the reset tests seem too strong since I provided the panel with the reset tests done for a wide range of estimates. Even accepting that the Reset test is appropriate (and no one else on the panel also uses this test in their work), there are many estimates where the results pass this test and he should thus conclude that those indicate a drop in violent crime.

7) Using too many control variables. Bartley and Cohen and I report all possible combinations of the control variables and show a great deal of consistency in the results. The only difference between these and those discussed in the NAS report is that these regressions included the arrest rate because of the zero crime rate problem.

8) Process. While the NAS is in name an academic organization, the process was hardly an academic one. Members of the panel were forbidden to talk to me about the issues being examined by the panel. Despite promises to get my input on the panels' review as it went forward, that never occurred. In particular, Charles Wellford promised me that I would be able to look at the tables and figures in the report. If I had been involved, I could have helped catch some of their mistakes. When the report was finally released to the public, I was promised that I would get a copy at the beginning of the presentation and that I would be allowed to ask questions. I was told that they preferred that I not attend the presentation, but there would be no problem with me asking questions. Instead even though the presentation ended a half hour earlier than scheduled because there were supposedly no more questions, my questions were never asked. (I had one main question: Professor Wellford mentions all the research that has been done on right-to-carry laws, but if he is correct that right-to-carry laws are just as likely to increase as decrease crime, can he point to one refereed journal article that claims to find a bad effect from the law?) Despite promises to the contrary, I did not receive a copy of the study until well into the afternoon and then only after a reporter from USA Today sent me a copy.

Minor notes: Despite claims to the contrary, I responded to the Ayres and Donohue study in January of 2004. (Simultaneously, it goes unnoticed that Ayres and Donohue themselves ignored virtually all of Plassmann and Whitley's points.)

In commenting on the report, others have raised additional issues that the NAS study did not find relevant. As to the claims raised again in these posts reguarding Jim Lindgren's investigation of the "phantom survey," many are apparently unaware that David Gross, David Mustard, and I have said that Lindgren has grossly mischaracterized what we said to him. For comments by Gross and Mustard, please see statements 3 and 4 in this link.

For a general response to the charges on the survey and other issues you raise see this link. False claims have been made with regard to these issues and the pseudonym.

Claims have also been made by Jim Lindgren regarding the demographic control variables, but he fails to note that it is only for the state level regressions and not the county level regressions where some of the significant results are affected. Given all the combinations of control variables that have been examined, even in that case, one wants some theory for why you selectively include what appears to be a weird combination of demographic controls. I think that Lindgren is a biased observer. He was upset after a critical piece that I published on his work in 2003 and his attacks started shortly after that. Further his attacks are untrue.

Final comments.

It is hard to look through the NAS panel's tables on right-to-carry laws and not find overwhelming evidence that right-to-carry laws reduce violent crime. The results that don't are based upon the inclusion of zero values noted in point 1 above. Overall, the panel's own evidence from the latest data up through 2000 shows significant benefits and no costs from these laws.

My impression is that Gary Kleck also has a very similar reaction to the panels' findings regarding surveys on self defense.

I express no opinion on this debate, but I thought I'd pass along John Lott's reactions.

	[Stuart Benjamin, December 17, 2004 at 8:20pm] Trackbacks Bad News for John Lott: Back in 1997, John Lott wrote an article in the Journal of Legal Studies putting forward data that seemed to show that right-to-carry (gun) laws reduced violent crimes. That article, and the subsequent book "More Guns Less Crime" influenced me and many others -- including state legislatures that passed right-to-carry laws based in significant part on the assertion that this would reduce crime. I don't have any particular precommitments regarding guns (and I shot many a gun while hunting as a youth), so I am guided by the empirical evidence. If more guns produces net benefits to society, then let's have more guns; if it doesn't, then let's not. Anyway, Lott's thesis has come under attack from a number of quarters. His bizarre behavior -- like writing emails as "Mary Rosh," a student who said "he was the best professor I ever had," etc. -- did not help his credibility, but it did not impugn his data. (He ultimately admitted he was Rosh.) His apparent fabrication of a study on which he relied was more serious, since it was part of his factual underpinnings, but the study was not central to his thesis. (By the way, co-blogger Jim Lindgren did excellent work ferreting out the details about the phantom survey.) The key question was, and is, whether his data are correct that right-to-carry laws reduce crime. His core thesis, though, was called into doubt by a number of researchers, most prominently in a study (and reply, both complete with data sets) written by Ian Ayres and John Donohue, two top empirical economists. They concluded that the data did not support Lott's assertions regarding right-to-carry laws and crime. Lott helped to write and then withdrew his name from a response to Ayres and Donohue. He responded in other venues to them, but did not respond to some of their key assertions. Perhaps he was waiting/hoping for vindication from the closest thing to a gold standard in academic review -- a report on the issue from the National Research Council. That report has been years in the making, and features some of the top researchers in the country. Well, the report has been issued, it contains bad news for Lott: It concludes that "There is no credible evidence that 'right-to-carry' laws, which allow qualified adults to carry concealed handguns, either decrease or increase violent crime." They discuss Lott's research at some length and find it wanting. Note that they do not say that right-to-carry laws increase crime. That may be a silver lining for those opposed to gun control who believe that in the absence of evidence of a benefit states should allow people to carry guns, but it doesn't help Lott very much: He staked his reputation on his claim that the data showed a decrease. So much for his reputation. Related Posts (on one page): John Lott Responds John Lott and the National Research Council's Report.-- Bad News for John Lott: [Jim Lindgren, December 27, 2004 at 1:02pm] Trackbacks John Lott and the National Research Council's Report.-- I was away when the story broke about the National Research Council releasing its report on the effects of gun laws on violence, though I saw Stuart Benjamin's nice post on it when I returned. The focus of Stuart's comments were the report's criticisms of John Lott's work, though he also mentioned my role in investigating whether a 1997 study that Lott says he did was ever actually done. As those of you who have been following this might remember, I thought that Ayres and Donohue did an excellent job of data analysis on the issue of the effect of carry laws on crime rates. I read through much of the report, including particularly chapter 6 that is critical of Lott's work, the dissent to part of that chapter by James Q. Wilson, and the majority response to that dissent. From the portions that I have read, I found the report sober, impressive, and fair, though there are substantial parts of this literature that I am unfamiliar with. As to Lott's work, I actually thought that the Council's report was too generous to his research in spots. In particular, I thought that it failed to point out just how much Lott's results are driven by poorly executed demographic controls, a point that Ayres and Donohue make effectively in their Stanford exchange. While the Council's report raises a lot of questions about Lott's use of control variables in general, particularly in its Appendix D, the Report does not seem to focus on the degree to these questionable demographic controls determine some of Lott's results. As usual, Tim Lambert has the most thorough coverage on the report. Related Posts (on one page): John Lott Responds John Lott and the National Research Council's Report.-- Bad News for John Lott: [Eugene Volokh, January 13, 2005 at 1:34pm] Trackbacks John Lott Responds to some posts (linked to at the end of this one) that criticize his work in light of the National Academy of Science report on gun control laws: Last month, the National Academy of Sciences issued a 328-page report on gun control laws. The big news that has been ignored on all the blog sites is that the academy's panel couldn't identify any benefits of the decades-long effort to reduce crime and injury by restricting gun ownership. The only conclusion it could draw was: Let's study the question some more. The panel has left us with two choices: Either academia and the government have wasted tens of millions of dollars and countless man-hours on useless research (and the panel would like us to spend more in the same worthless pursuit), or the National Academy is so completely unable to separate politics from its analyses that it simply can't accept the results for what they are. Based on 253 journal articles, 99 books, 43 government publications, and some of its own empirical work, the panel couldn't identify a single gun control regulation that reduced violent crime, suicide or accidents. From the assault weapons ban to the Brady Act to one-gun-a-month restrictions to gun locks, nothing worked. (Something that I have been the first person to investigate empirically for many of these laws, and I also had been unable to find evidence that they reduced violent crime.) The study was not the work of gun-control opponents. The panel was set up during the Clinton administration, and of its members whose views on guns were publicly known before their appointments all but one had favored gun control. Something that I wrote up about the panel three years ago is still relevant. While the panel dealt with a broad range of gun control issues, only one issue has received attention on different blogs: right-to-carry laws. In fact, the panel apparently originated with the desire from some to respond to the debate on that issue and to respond specifically to my research that concludes that allowing law abiding citizens to carry concealed weapons reduces crime. I originally overheard Phil Cook and Dan Nagin discussing the need for a panel to "deal with" me in the same way that an earlier panel had "dealt with Isaac" Ehrlich's work showing that the death penalty deterred murder. They agreed and Nagin said that he would talk to Al Blumstein about setting up such a panel. Needless to say, that is what ended up happening. 1) James Q. Wilson's very unusual dissent is very interesting (only two out of the last 236 reports over the last 10 years have carried a dissent). Wilson states that all the research provided "confirmation of the findings that shall-issue laws drive down the murder rate . . . " Wilson has been on four of these panels and never previously thought that it was necessary to write a dissent, including the previous panel that attacked Isaac Ehrlich's work showing that the death penalty represented a deterrent. Wilson said that that panel's conclusion raises concerns given that "virtually every reanalysis done by the committee" confirmed right-to-carry laws reduced crime. He found the committee's only results that didn't confirm the drop in crime "quite puzzling." They accounted for "no control variables" - nothing on any of the social, demographic, and public policies that might affect crime. Furthermore, he didn't understand how evidence that was not publishabled in a peer-reviewed journal would be given such weight. The non-results are basically due to dropping all the control variables (particularly the arrest rate which is not defined when the crime rate is zero). When that happens a lot of observations with zero crime rates are introduced. The problem with using OLS when you have all these zero crime rates is that if a crime rate is already zero, no matter how good the law is, it can't lower the crime rate any further. There is thus a positive bias in these results. Plassmann's two papers (his piece in the Journal of Law and Economics with Nic Tideman and his paper with Whitley in the Stanford Law Review) show how you can address this as a count data problem. Although his research consistently shows statistically significant results that shall issue laws reduce crime, the National Academy report ignores the research. The panel's discussion of Duggan's results focuses on the regressions without any control variables and that use the OLS estimates when they have a large number of zero values for the crime rates. 2) As an interesting aside, there are a number of factual mistakes in the NAS report and those mistakes work against my findings. For example, Figure 6.1 makes a mistake where it shows the increase in violent crime of 7 percent in year one, when the amount is 5 percent (7-2, where 2 is from the trend). (Of course, the overall problem with the hybrid approach is discussed below.) There are significant drops in crime in Table 6-3 that are statistically significant, but they are not properly marked to indicate that is so. Even something trivial as the number of states currently with right-to-carry laws is wrong, 36 (not 34) (and if Minnesota is included the number is 37). 3) Last year there was a debate over the use of clustering between Ayres and Donohue and me, but the statements of the NAS panel corresponds extremely closely to what was written in my original paper with David Mustard. 4) p. 127: "We focus on the conflicting results . . ." No attempt is made to give readers an idea of the frequency or importance of unusual results. Take the results in Table 6-3. For Plassmann and Whitley, the panel doesn't mention that Plassmann and Whitley say that there are "major problems" with the particular regressions that the panel decides to report and more importantly that the effects in those regressions are biased towards zero (see point 2 above). For Moody's results, they show only two specifications of all the results that he reports and don't mention that the one weird result that he got was from a specification that he flagged as problematic and not controlling for other factors. Even with the very selective sample of regressions that they pick, there is not one statistically significant bad effect of right-to-carry laws on murder. Only one case for robbery and that is one problematic specification from Ayres and Donohue. 5) Hybrid model. The so-called hybrid model used by Ayres and Donohue finds that the law dummy variable is positive while the trend variable indicates that crime rates decline over time. While Plassmann and Whitley do a good job explaining why the "hybrid" model produces misleading results and the panel never discusses their critique (looking at the crime rates on a year by year basis show no initial increase in crime), it still would have been useful for the panel to at least say whether the "hybrid" results produced a statistically significant temporary bad effect. The problem with determining statistical significance is that when both the dummy and trend variables are on at the same time, we are concerned about the net effect not just the dummy variable by itself as Ayres and Donohue argue. The answer for all those results in the panel's Table 6-4 is "no." 6) Reset tests. Professor Horowitz's discussion of the reset tests seem too strong since I provided the panel with the reset tests done for a wide range of estimates. Even accepting that the Reset test is appropriate (and no one else on the panel also uses this test in their work), there are many estimates where the results pass this test and he should thus conclude that those indicate a drop in violent crime. 7) Using too many control variables. Bartley and Cohen and I report all possible combinations of the control variables and show a great deal of consistency in the results. The only difference between these and those discussed in the NAS report is that these regressions included the arrest rate because of the zero crime rate problem. 8) Process. While the NAS is in name an academic organization, the process was hardly an academic one. Members of the panel were forbidden to talk to me about the issues being examined by the panel. Despite promises to get my input on the panels' review as it went forward, that never occurred. In particular, Charles Wellford promised me that I would be able to look at the tables and figures in the report. If I had been involved, I could have helped catch some of their mistakes. When the report was finally released to the public, I was promised that I would get a copy at the beginning of the presentation and that I would be allowed to ask questions. I was told that they preferred that I not attend the presentation, but there would be no problem with me asking questions. Instead even though the presentation ended a half hour earlier than scheduled because there were supposedly no more questions, my questions were never asked. (I had one main question: Professor Wellford mentions all the research that has been done on right-to-carry laws, but if he is correct that right-to-carry laws are just as likely to increase as decrease crime, can he point to one refereed journal article that claims to find a bad effect from the law?) Despite promises to the contrary, I did not receive a copy of the study until well into the afternoon and then only after a reporter from USA Today sent me a copy. Minor notes: Despite claims to the contrary, I responded to the Ayres and Donohue study in January of 2004. (Simultaneously, it goes unnoticed that Ayres and Donohue themselves ignored virtually all of Plassmann and Whitley's points.) In commenting on the report, others have raised additional issues that the NAS study did not find relevant. As to the claims raised again in these posts reguarding Jim Lindgren's investigation of the "phantom survey," many are apparently unaware that David Gross, David Mustard, and I have said that Lindgren has grossly mischaracterized what we said to him. For comments by Gross and Mustard, please see statements 3 and 4 in this link. For a general response to the charges on the survey and other issues you raise see this link. False claims have been made with regard to these issues and the pseudonym. Claims have also been made by Jim Lindgren regarding the demographic control variables, but he fails to note that it is only for the state level regressions and not the county level regressions where some of the significant results are affected. Given all the combinations of control variables that have been examined, even in that case, one wants some theory for why you selectively include what appears to be a weird combination of demographic controls. I think that Lindgren is a biased observer. He was upset after a critical piece that I published on his work in 2003 and his attacks started shortly after that. Further his attacks are untrue. Final comments. It is hard to look through the NAS panel's tables on right-to-carry laws and not find overwhelming evidence that right-to-carry laws reduce violent crime. The results that don't are based upon the inclusion of zero values noted in point 1 above. Overall, the panel's own evidence from the latest data up through 2000 shows significant benefits and no costs from these laws. My impression is that Gary Kleck also has a very similar reaction to the panels' findings regarding surveys on self defense. I express no opinion on this debate, but I thought I'd pass along John Lott's reactions. Related Posts (on one page): John Lott Responds John Lott and the National Research Council's Report.-- Bad News for John Lott: