In an earlier post, I compared Judge Sotomayor with other judges, including Judge Wood, who were ranked in a paper by Choi and Gulati. Choi and Gulati focused on three statistics—productivity (number of opinions written per year), quality (citations), and independence (propensity to disagree with same-party judges). I found that Judge Sotomayor did not compare well with the other judges, and concluded that "The bottom line is that Judge Sotomayor is about average, or maybe a bit below average, for a federal appellate judge." However, I also noted that a problem with the comparison is that I used data from the beginning of Judge Sotomayor's tenure (1999-2001), so as to make the comparison with Choi and Gulati's 1998-2000 data as fair as possible. If a "freshman effect" exists—if less experienced judges are worse than more experienced judges, which seems plausible—the comparison is not in fact a fair one.

To address this problem, Choi, Gulati, and I—with the help of Mirya Holman of Duke Law School and several Duke Law School students—collected and analyzed data from 2004 to 2006. We can no longer compare the results to the 1998-2000 data set because of possible changes in the composition of case loads and other factors, or at least we should be very cautious. Because we did not have time to collect 2004-2006 data for all court of appeals judges so that we can supply a complete ranking, we decided to compare Sotomayor with the other court of appeals judges who were rumored to be on President Obama's shortlist, and with a kind of control group consisting of court of appeals judges rumored to be on President Bush's shortlist in 2005. The results are at the bottom of this post (with apologies for the wretched formatting).

The freshman effect is vindicated: by 2004-2006 Judge Sotomayor has, on several measures, exceeded her colleagues. On total citations—a measure of general influence that reflects number of opinions as well as citations per opinion—Sotomayor ranks third after Lynch and Wood. On citations to top 20 opinions, Sotomayor ranks first. With 512 citations to top 20 opinions, Sotomayor would have been sixth in this category in Choi and Gulati's ranking (Wood's high numbers for 2004-2006 are also consistent with her performance in that ranking). On citations per opinion—which favors judges like Sotomayor who write few opinions—Judge Sotomayor dominates all of the shortlisted judges. She ranks third on law review citations per opinion.

Not everyone agrees that citation is a good measure of quality. And there may be hidden biases in the data. For example, if the tenth circuit hears less important cases than the second circuit does, a tenth circuit judge like McConnell may be cited less—because he hears disputes that are less important, not because his opinions are not good. In addition, judges in large circuits are hurt because the set of out-of-circuit opinions that could cite them is smaller than it is for judges in small circuits. Still, the numbers seem pretty robust, and the comparison here is with well-respected judges, not with ordinary judges. Sotomayor looks good.

We also checked how many cases received a Westlaw "red flag," which usually means that the Supreme Court has vacated all or part of a holding. Judge Sotomayor seems about average or better than average, controlling for the number of cases.

As in the first analysis, Judge Sotomayor does not rank high on productivity; she is at the median for this group. However, different circuits have different productivity norms. "Productivity" really means the number of published opinions; courts frequently dispose of cases with unpublished memoranda, and the practice across circuits diverges (reflecting perhaps how busy the circuit is, or judges' collective views about how their effort should be expended). In the Choi-Gulati dataset, the most productive second circuit judge published 109 decisions over three years; the least productive published 66; Sotomayor is somewhere around average or above average for the circuit. Meanwhile, Judge Wood sits in the super-productive seventh circuit. So given circuit norms, Sotomayor has more time to lavish on her opinions than Judge Wood does, and it is not surprising that her opinions are cited more often. In terms of overall quality, it is hard to distinguish the two judges.

We did not try to calculate independence scores because of lack of time. But we looked at dissents. The numbers here are so low that they do not convey much information. Sotomayor is about average. You can interpret this to mean that she is no more or less collegial than her colleagues, or that she is no more or less independent-minded than her colleagues.

All the usual qualifications, caveats, and disclaimers apply. But the data should put to rest the rumor that Judge Sotomayor is not a competent jurist. She holds her own among a highly respected group which includes the third (Wilkinson), eighth (Wood), eleventh (Lynch), and thirteenth (Jones) ranked judges on Choi and Gulati's composite ranking (Alito was sixteenth). If citations reflect quality, Sotomayor may well be one of the top appellate judges in the country.

Many thanks to the industrious Duke Law School students who collected the data: Christopher Battles, Heather Horst, Luke Ortner, Christopher-David McCurdy, and Seth Reynolds.