John Lott's sensational lawsuit for defamation against economist Steve Levitt (and co-author of the bestselling Freakonomics) has received only limited comment in the blogosphere. When I first read that Lott was supposedly suing because Levitt and Dubner had written in Feakonomics that Lott's work had not been replicated, I thought that sounded like a weak claim, since there are several meanings to the claim that work has not been or cannot be replicated.

One might mean that:

1. Using different data drawn from a different era or a different population, a replicator found that the patterns in the data between the new study and the earlier one are meaningfully different.

OR

2. Supposedly using the exact same data, a replicator was unable to get the same coefficients in the same models.

Even if one means the second sort of a failure to replicate, the reason is usually that the original researcher recoded the data in some arguably reasonable way that he has since forgotten, or that he copied the results from the wrong model into his paper, or that he miscopied results, or that there is some communication problem between the original researcher and the replicator. In other words, usually someone claiming a failure to replicate is claiming that the results of the original study are unreliable, but in some relatively innocent way.

The passage that Lott objects to is covered in this excellent article in the University of Chicago student newspaper, the Maroon:

The lawsuit states that the book "damages Lott's reputation in the eyes of the academic community in which he works, and in the minds of hundreds of thousands of academics, college students, graduate students, and members of the general public who read Freakonomics."

The contested material is on pages 133--134 of Freakonomics, in which Levitt writes that researchers have been unable to confirm Lott's conclusion that right-to-carry gun laws actually reduce crime.

Freakonomics states, "Then there was the troubling allegation that Lott actually invented some of the survey data that supports his more-guns/less-crime theory. Regardless of whether or not the data was faked, Lott's admittedly intriguing hypothesis doesn't seem to be true. When other scholars have tried to replicate his results, they found that right-to-carry laws simply don't bring down crime." According to the lawsuit, Lott acknowledges that his findings have come under scrutiny in the academic community, but he maintains that he used "different data or methods to analyze the relationship between gun control laws and crime."

The lawsuit states that scholars who have replicated Lott's work have achieved the same results. "Every time that an economist or researcher have tried to replicate [Lott's] results, he or she has confirmed Lott's conclusion."

Carl Moody, a professor of economics at the College of William and Mary, said he successfully replicated Lott's findings and published the results in 2001. Moody said Levitt's accusation is wrong.

The lawsuit, which also names Levitt's publisher HarperCollins, states that the publisher acted with malice by failing to verify the statements. It seeks a court order to halt sales of Freakonomics until the statements are retracted or amended and also demands that Levitt and HarperCollins pay unspecified monetary damages.

Looking at the quoted passage from Freakonomics, there would seem to be two problems with it.

First, on the narrow view of replication, it may be factually incorrect if Carl Moody has indeed replicated Lott's analysis using the exact same data and published the results. Accordingly, it may be somewhat misleading to say that Lott's results have not been replicated when they have been in the narrow sense, but not usually in the broader sense (even in the broader sense, the results may be a bit more mixed than Freakonomics implies).

Second, and more troubling to me, Freakonomics juxtaposes concerns about whether Lott ever did his 1997 study with the nonreplicability of Lott's main studies presenting his main thesis: "More Guns, [lead to] Less Crime." These are completely different studies, linked mostly by the person who reported them. If I didn't know this fact, I might conclude from reading the passage in Freakonomics that the reason that Lott's research asserting that more guns leads to less crime supposedly hadn't been replicated (in either sense of the word) is that he may not have done it in the first place. That strikes me as misleading, since the work that people have tried to replicate was most assuredly done (and Lott has shared the data from his main study).

It would seem to me that this passage in Freakonomics should be revised to separate more clearly the two points being made and to explain the sense in which Lott's work has and has not been replicated.

Even though I find Freakonomics misleading in its juxtaposition of problems with two very different Lott gun studies, Lott obviously has an uphill quest in persuading a court that the book is defamatory. Here are the two crucial allegations from the complaint:

12. The term "replicate" has an objective and factual meaning in the world of academic research and scholarship. When Levitt and Dubner allege that "other scholars have tried to replicate his results," the clear and unambiguous meaning is that "other scholars" have analyzed the identical data that Lott analyzed and analyzed it in the way Lott did in order to determine whether they can reach the same result. When Levitt and Dubner allege that when "other scholars have tried to replicate his results, they found that right-to-carry laws simply don't bring down crime," they are alleging that Lott falsified his results. . . .

14. The allegation that other scholars have been unable "to replicate [Lott's] results" is defamatory per se because it attacks Lott's integrity and honesty in his profession as an economist, scholar and researcher. . . ."

It would seem to me to be a hard sell to persuade a trier of fact that it is defamatory per se to assert that researchers have been unable to replicate someone's work, given that replication does not mean only what allegation 12 says it means.

Interestingly, the complaint does not note Freakonomics' misleading tying of the 1997 study to the inability to replicate his work. That may be because Lott didn't want to raise an issue on which he is vulnerable, given the lack of records to support the 1997 study.

Bottom line: I think that Freakonomics is misleading in its juxtaposition of different studies, a juxtaposition that might bring one to conclude that the reason that the main More Guns, Less Crime research was not usually replicated in other studies is that there was some chance that a study was never done. Yet, to the extent that Lott's lawsuit is based on a failure to replicate being per se defamatory, I would think he has a difficult chance of winning.