Dan Slater (Wall Street Journal's Law Blog) reports on an interesting story about errors in the academy. The Tulane Law Review recently published an article that purported to compare Louisiana Supreme Court Justices' voting records with the campaign contributions to them from litigants and lawyers; the article asserted that

some of the justices have been significantly influenced — wittingly or unwittingly — by the campaign contributions they have received from litigants and lawyers appearing before these justices.

The New York Times reported on the story before the article was published, as did WSJ Law Blog.

1. Now it turns out that there many of the cases were miscoded — a rebuttal asserts that "in thirty-seven of the 186 opinions included in the study, the information about the case on which Palmer and Levendis based their conclusions is just plain wrong, such as how a Justice voted or even if the Justice was on the panel that decided the case." [UPDATE: After I posted this post, the rebuttal I linked to was replaced with a corrected version, and I adjusted the quote. The original version said 40 of the 186 opinions were miscategorized; the revised version says 37.]

The authors acknowledge that there were errors; one of the authors asserted that "with all the mistakes now corrected, ... the study's conclusions, broadly speaking, are the same," but the revised study and the revised dataset has not yet been publicly distributed. (I e-mailed that author mentioning I was going to blog about the controversy, and asking whether he could provide a response that I could link to; but though he originally offered to pass along the revised dataset, he later said that because of a newly arising lawsuit threat, he was told not to distribute the data until it could be independently reviewed. This may well be sensible, but at this point all that can be said with confidence is that the original data is wrong, and that this has been admitted by the researchers.)

Incidentally, while it's not clear from the article exactly how the data was gathered, it looks like part of the problem might be lack of checking by the authors or by the law review: The article asserts (p. 1298) that "Each case was thoroughly read and analyzed by a researcher. Once the cases and contribution information were gathered, we entered our observations into a standard data table." If the article "a" in "a researcher" is precise, and if the researchers were research assistants and not the authors (seemingly likely, given the thanks in footnote * and the use of the term "a researcher" rather than "one of the coauthors"), this suggests that each case was read only by one research assistant, with no further verification.

I also asked the current Tulane Law Review editor-in-chief, and he reported that to his knowledge the law review cite-checking process did not check the underlying database. This is probably consistent with standard law review practice; law reviews generally check all the citations that appear in the text of the article, but I'm unaware of any practice of checking the case data that appears in databases that aren't published within the body, footnotes, or appendix of the article. Nonetheless, it's unfortunate that this happened, since cite-checking often (though not always) does uncover factual errors such as the ones that appear to have been present here.

2. But there's more than this to the situation, I think. Even if the data were correct, the article would still be drawing what strikes me as an unsupported inference from correlation to causation. The article asserts in footnote 14 that

It is worth observing that this Article does not claim that there is a cause and effect relationship between prior donations and judicial votes in favor of donors' positions. It asserts instead that there is evidence of a statistically significant correlation between the two.

But many other passages in the article seem to argue that there was indeed causation. The opening paragraph says (as I quoted above) that "This empirical and statistical study of the Louisiana Supreme Court over a fourteen-year period demonstrates that some of the justices have been significantly influenced — wittingly or unwittingly — by the campaign contributions they have received from litigants and lawyers appearing before these justices." (Emphasis added, in this quote and the later ones.) "Some justices may sincerely believe that they have not been influenced by the money they take, but sincerity makes no difference if the reality is otherwise." "Some studies have tracked the rise in contributions made to judicial candidates, but few have attempted to determine whether these increasing contributions actually influence subsequent adjudications involving contributors. This Article demonstrates that the debate is not evenly balanced." (This quote is just a sentence before footnote 14.) There are many more examples.

The trouble, as this other critique of the article points out, is that there's a perfectly plausible alternate explanation for correlation between contributions and voting patterns — that contributors contribute money to the election of those candidates whose ideologies they agree with, rather than that the elected candidates then decide based on the identities of their contributors.

Say, for instance, that we discover that a liberal state supreme court justice often votes in favor of plaintiffs in individual tort cases, employment cases, or environmental cases; and say that we find that those plaintiffs are often represented by law firms that have contributed money to the justice. It's of course possible that the justice is influenced by the identities of his contributors. But it's also possible that the justice is voting solely based on his view of the law — and the contributors contributed to him because they share his view of the law (or in any event find his view of the law to be good for them and their clients). Simple evidence of a correlation, even a very strong correlation, cannot distinguish between these two explanations, and thus can't show that contributions "influence" the justices' votes.

The authors' attempts to control in a way that eliminates the alternative explanation strikes me as quite weak. The authors look separately at voting patterns in favor of plaintiffs and in favor of defendants, and they conclude that

In cases where the defendant was the net contributor, Justice [A] ruled for the defendant's position 66% of the time, and Justice [B] 86% of the time. On the other hand, in cases where the plaintiff was the net contributor, Justice [A]'s vote was for the plaintiff's position 66% of the time, and Justice [B]'s vote was for the plaintiff's position 63% of the time. This is a swing of 32% for Justice [B] and 49% for Justice [A] when the net donor changes from being a defendant to a plaintiff. The marked shift favoring the net contributor, irrespective of being plaintiff or defendant, strongly indicates that it is the donation, not the underlying philosophical orientation, that accounts for the voting outcome.

But no judge has any inherent philosophical orientation for plaintiffs or defendants as a broad category. For instance, some judges may support cities as defendants (for instance, voting in favor of broad municipal immunity against various claims) and cities as plaintiffs (for instance, voting in favor of broad municipal power to get injunctions against nuisances) — not because the law firms that represent small cities have donated money to the judge, but because the judge generally supports municipal government authority.

Likewise, say a law firm tends to represent employers as defendants in many employment cases, but also represents employers as plaintiffs in other employment cases, such as cases enforcing nondisclosure or anticompetition agreements. In this situation, a judge with a generally pro-employer perspective will vote in ways this law firm likes — which may mean the law firm will try to help elect the judge even if the judge never pays attention to who contributed to him. (Recall that the study considered donations from lawyers as well as donations from litigants.)

3. Of course, it's certainly possible that judges' decisions may be influenced by whether the litigants or the lawyers have contributed money — or time or an endorsement or other things — to the judges' election. I don't want to suggest that anything I say above proves this effect is absent, either on the Louisiana Supreme Court or elsewhere. But the study's claim that it has "demonstrate[d]" such a "significant[] influence[]" is not adequately supported. And before one claims that identified judges have indeed been so influenced (not just that judges generally might be influenced, a proposition that our knowledge of human nature tells us must be true at least in some instances), it seems to me that one should have significantly more evidence than the article adduces. And this would be true even if the article's underlying database were accurate, or even if the authors can replicate their correlations after the database has been corrected.

4. Finally, note that the law school's dean has written a letter of apology to the Louisiana Supreme Court Justices, and the law review has noted the error on its site. (I don't know what further attempts at correction the law review might be taking.)

Thanks to How Appealing for the pointer.