Dan Slater (Wall Street Journal’s Law Blog) reports on an interesting story about errors in the academy. The Tulane Law Review recently published an article that purported to compare Louisiana Supreme Court Justices’ voting records with the campaign contributions to them from litigants and lawyers; the article asserted that
some of the justices have been significantly influenced — wittingly or unwittingly — by the campaign contributions they have received from litigants and lawyers appearing before these justices.
The New York Times reported on the story before the article was published, as did WSJ Law Blog.
1. Now it turns out that there many of the cases were miscoded — a rebuttal asserts that “in thirty-seven of the 186 opinions included in the study, the
information about the case on which Palmer and Levendis based their conclusions is
just plain wrong, such as how a Justice voted or even if the Justice was on the panel
that decided the case.” [UPDATE: After I posted this post, the rebuttal I linked to was replaced with a corrected version, and I adjusted the quote. The original version said 40 of the 186 opinions were miscategorized; the revised version says 37.]
The authors acknowledge that there were errors; one of the authors asserted that “with all the mistakes now corrected, … the study’s conclusions, broadly speaking, are the same,” but the revised study and the revised dataset has not yet been publicly distributed. (I e-mailed that author mentioning I was going to blog about the controversy, and asking whether he could provide a response that I could link to; but though he originally offered to pass along the revised dataset, he later said that because of a newly arising lawsuit threat, he was told not to distribute the data until it could be independently reviewed. This may well be sensible, but at this point all that can be said with confidence is that the original data is wrong, and that this has been admitted by the researchers.)
Incidentally, while it’s not clear from the article exactly how the data was gathered, it looks like part of the problem might be lack of checking by the authors or by the law review: The article asserts (p. 1298) that “Each case was thoroughly read and analyzed by a researcher. Once the cases and contribution information were gathered, we entered our observations into a standard data table.” If the article “a” in “a researcher” is precise, and if the researchers were research assistants and not the authors (seemingly likely, given the thanks in footnote * and the use of the term “a researcher” rather than “one of the coauthors”), this suggests that each case was read only by one research assistant, with no further verification.
I also asked the current Tulane Law Review editor-in-chief, and he reported that to his knowledge the law review cite-checking process did not check the underlying database. This is probably consistent with standard law review practice; law reviews generally check all the citations that appear in the text of the article, but I’m unaware of any practice of checking the case data that appears in databases that aren’t published within the body, footnotes, or appendix of the article. Nonetheless, it’s unfortunate that this happened, since cite-checking often (though not always) does uncover factual errors such as the ones that appear to have been present here.
2. But there’s more than this to the situation, I think. Even if the data were correct, the article would still be drawing what strikes me as an unsupported inference from correlation to causation. The article asserts in footnote 14 that
It is worth observing that this Article does not claim that there is a cause and effect relationship between prior donations and judicial votes in favor of donors