John Lott Update on His Reported 1997 Study.--

I have been meaning to post on several developments this spring relating to economist John Lott, which I am doing in two posts.

As some of you may be aware, there was a dispute over whether John Lott ever did a study of defensive gun uses in 1997, as he claimed to have done. Interested people can review most of the ins an outs of that dispute [by reading comments at Lott's site] and at Tim Lambert's Deltoid site (Lambert is Lott's most vigorous critic). In December 2002 and January 2003 I wrote a long, tediously detailed, and ultimately inconclusive report on whether Lott ever did a study in 1997, finding good evidence that Lott had a major data loss, but little evidence that Lott conducted a study in that year.

Immediately after that report appeared, Minnesota gun activist and former NRA board member David Gross came forward to say that he had been interviewed for a gun survey, and he thought that he was interviewed in the spring of 1997, probably by people working for Lott.

Here is what I later wrote to Instapundit about the Gross account:

As people may have heard, but may have not quite understood, I found Gross a credible witness. Unfortunately, that is a lawyer's term of art. It is possible to have credible witnesses on both sides of a case telling inconsistent stories. What I meant is my opinion that most people who heard him would give credence to his account (and that I found him generally believable), not that his account would necessarily trump any other evidence.

The part of Mr. Gross's written public statement that was slightly different from what he told me concerned who called him for the interview. When I asked him if he remembered anything about who called, he said that he "was beginning to think" that the call came from students in Chicago, perhaps at Northwestern or the University of Chicago, but he was very uncertain about whether the call came from a Chicago area source. In his public statement issued after he talked with me more than once, however, Gross's very uncertain memory became a bit more certain, suggesting that the call probably came from the University of Chicago. That and the timing (which he was also not certain about) were the only things that pointed to him having been called by Lott as opposed to another survey organization.

As I delved into the other studies being done in the 1996-97 period, I found that Gross's description of the questions that he was asked fit a 1996 Harvard study by Hemenway & Azrael better than Lott's account of his study questions. First, Gross said that the person who called him was interested in a defensive gun use that happened a few years before he was surveyed, but was not interested in a defensive use that occurred many years before that. This would not fit Lott's survey, since Lott [told me that he] asked only about DGUs in the prior year. It would fit the Harvard study perfectly, which asked about DGUs in the prior 5 years, but excluded events before that. Further, Gross said that he gave a narrative account of the event, which the caller was interested in. Lott's study had asked closed-end questions, which would make the narrative superfluous, while the Harvard study was one of the first to ask for a narrative account of DGUs. Last, Gross reported that there was a question about state gun laws, which Lott did not ask, but the Harvard study did.

Some weeks ago John Lott emailed me to report that the data for the 1996 Hemenway study had been released and that the demographics for each respondent from the state where Gross lived (Minnesota) did not fit Gross. I checked those data and also downloaded and checked the open-ended narrative accounts in the 1996 Hemenway study of defensive gun uses from Minnesota. Since neither the demographics nor the descriptive accounts matched Gross to any Minnesota respondent, it seems clear that Gross was not surveyed by Hemenway in 1996.

Interestingly, even in 2003 neither Tim Lambert nor David Hemenway thought it likely that Gross had been surveyed in the 1996 Harvard study. Since Gross's account of the questions asked fit Lott's claims about his 1997 study in some respects but not others, we are left with several possibilities. If Gross is not lying (and despite his strong pro-gun orientation and some minor changes in his story, I still think that it is more likely than not that Gross is basically telling the truth as he remembers it), then either Gross or Lott misremembered some of the questions asked or Gross was surveyed for yet another survey.

By the way, John Lott also wrote me that another survey participant came forward a year ago, but I have not interviewed him or her.

Last, it appears that Lott is no longer fulltime at AEI. Although his page there is still reachable from Google, it is no longer linked from the AEI scholars page. He is listed as "Former Resident Scholar."

John Lott's Defamation Lawsuit Against Steve Levitt.--

John Lott's sensational lawsuit for defamation against economist Steve Levitt (and co-author of the bestselling Freakonomics) has received only limited comment in the blogosphere. When I first read that Lott was supposedly suing because Levitt and Dubner had written in Feakonomics that Lott's work had not been replicated, I thought that sounded like a weak claim, since there are several meanings to the claim that work has not been or cannot be replicated.

One might mean that:

1. Using different data drawn from a different era or a different population, a replicator found that the patterns in the data between the new study and the earlier one are meaningfully different.

OR

2. Supposedly using the exact same data, a replicator was unable to get the same coefficients in the same models.

Even if one means the second sort of a failure to replicate, the reason is usually that the original researcher recoded the data in some arguably reasonable way that he has since forgotten, or that he copied the results from the wrong model into his paper, or that he miscopied results, or that there is some communication problem between the original researcher and the replicator. In other words, usually someone claiming a failure to replicate is claiming that the results of the original study are unreliable, but in some relatively innocent way.

The passage that Lott objects to is covered in this excellent article in the University of Chicago student newspaper, the Maroon:

The lawsuit states that the book "damages Lott's reputation in the eyes of the academic community in which he works, and in the minds of hundreds of thousands of academics, college students, graduate students, and members of the general public who read Freakonomics."

The contested material is on pages 133--134 of Freakonomics, in which Levitt writes that researchers have been unable to confirm Lott's conclusion that right-to-carry gun laws actually reduce crime.

Freakonomics states, "Then there was the troubling allegation that Lott actually invented some of the survey data that supports his more-guns/less-crime theory. Regardless of whether or not the data was faked, Lott's admittedly intriguing hypothesis doesn't seem to be true. When other scholars have tried to replicate his results, they found that right-to-carry laws simply don't bring down crime." According to the lawsuit, Lott acknowledges that his findings have come under scrutiny in the academic community, but he maintains that he used "different data or methods to analyze the relationship between gun control laws and crime."

The lawsuit states that scholars who have replicated Lott's work have achieved the same results. "Every time that an economist or researcher have tried to replicate [Lott's] results, he or she has confirmed Lott's conclusion."

Carl Moody, a professor of economics at the College of William and Mary, said he successfully replicated Lott's findings and published the results in 2001. Moody said Levitt's accusation is wrong.

The lawsuit, which also names Levitt's publisher HarperCollins, states that the publisher acted with malice by failing to verify the statements. It seeks a court order to halt sales of Freakonomics until the statements are retracted or amended and also demands that Levitt and HarperCollins pay unspecified monetary damages.

Looking at the quoted passage from Freakonomics, there would seem to be two problems with it.

First, on the narrow view of replication, it may be factually incorrect if Carl Moody has indeed replicated Lott's analysis using the exact same data and published the results. Accordingly, it may be somewhat misleading to say that Lott's results have not been replicated when they have been in the narrow sense, but not usually in the broader sense (even in the broader sense, the results may be a bit more mixed than Freakonomics implies).

Second, and more troubling to me, Freakonomics juxtaposes concerns about whether Lott ever did his 1997 study with the nonreplicability of Lott's main studies presenting his main thesis: "More Guns, [lead to] Less Crime." These are completely different studies, linked mostly by the person who reported them. If I didn't know this fact, I might conclude from reading the passage in Freakonomics that the reason that Lott's research asserting that more guns leads to less crime supposedly hadn't been replicated (in either sense of the word) is that he may not have done it in the first place. That strikes me as misleading, since the work that people have tried to replicate was most assuredly done (and Lott has shared the data from his main study).

It would seem to me that this passage in Freakonomics should be revised to separate more clearly the two points being made and to explain the sense in which Lott's work has and has not been replicated.

Even though I find Freakonomics misleading in its juxtaposition of problems with two very different Lott gun studies, Lott obviously has an uphill quest in persuading a court that the book is defamatory. Here are the two crucial allegations from the complaint:

12. The term "replicate" has an objective and factual meaning in the world of academic research and scholarship. When Levitt and Dubner allege that "other scholars have tried to replicate his results," the clear and unambiguous meaning is that "other scholars" have analyzed the identical data that Lott analyzed and analyzed it in the way Lott did in order to determine whether they can reach the same result. When Levitt and Dubner allege that when "other scholars have tried to replicate his results, they found that right-to-carry laws simply don't bring down crime," they are alleging that Lott falsified his results. . . .

14. The allegation that other scholars have been unable "to replicate [Lott's] results" is defamatory per se because it attacks Lott's integrity and honesty in his profession as an economist, scholar and researcher. . . ."

It would seem to me to be a hard sell to persuade a trier of fact that it is defamatory per se to assert that researchers have been unable to replicate someone's work, given that replication does not mean only what allegation 12 says it means.

Interestingly, the complaint does not note Freakonomics' misleading tying of the 1997 study to the inability to replicate his work. That may be because Lott didn't want to raise an issue on which he is vulnerable, given the lack of records to support the 1997 study.

Bottom line: I think that Freakonomics is misleading in its juxtaposition of different studies, a juxtaposition that might bring one to conclude that the reason that the main More Guns, Less Crime research was not usually replicated in other studies is that there was some chance that a study was never done. Yet, to the extent that Lott's lawsuit is based on a failure to replicate being per se defamatory, I would think he has a difficult chance of winning.