Much has been written about Jeff Rosen’s New Republic piece, which cited anonymous sources who disparaged Judge Sotomayor’s judicial ability. It is never easy to evaluate judges, or to evaluate their evaluators, especially when those evaluators insist on anonymity. Fortunately, data on judicial performance exist, and although the data have problems as well, they provide a firmer basis for evaluation.
The most complete effort so far to evaluate federal appellate judges is this paper by Stephen Choi and Mitu Gulati. Choi and Gulati use data from Lexis to measure three aspects of the judge’s performance—productivity, opinion quality, and independence. They then rank judges according to how they do along these three dimensions. The data set comes from the years 1998-2000 and includes 98 judges. Unfortunately, Choi and Gulati excluded Judge Sotomayor from the data set because she was appointed in 1998 and thus does not have complete data for that year. Judge Wood is in their data set and ranks eighth in the composite ranking. (Disclosure: Judge Wood teaches at my law school.)
To determine how Sotomayor would do in the ranking, I had some research assistants collect her data for the years 1999-2001. To address the “freshman effect” (the possibility that her statistics are worse for her earliest years because of inexperience), we also looked at her data from 2006. I provide a brief summary below. If you want more detail about the methods, you will need to read the variable definitions in the paper. And if you want a defense of these measures, you will need to read the paper, but in any event should know that they are controversial.
Productivity. Judges write opinions, which provide guidance to lawyers and the public. All else equal, a judge who writes more opinions is more productive, and provides a greater social benefit. Over the three year period from 1998 to 2000, the most productive judge published 269 opinions, the least productive judge published 38 opinions, and the mean was 98.1. For the comparable period from 1999-2001, Judge Sotomayor published 73 opinions. She would have ranked 68th out of 98. However, she was substantially more productive in 2006, publishing 36 opinions in one year (though that also is higher than in other recent years). (Judge Wood is fourth; then-Judge Alito was 72nd.)
Quality (1). Choi and Gulati measure quality by counting citations to a judge’s top twenty opinions. This approach avoids penalizing productive judges whose marginal opinion addresses narrower issues (as a citation-per-opinion measure would), while also avoiding excessive credit to productive judges who garner high citations by writing a lot. Judge Wood’s top twenty cases over three years received 327 outside-circuit citations, putting her 26th. The range is 96 to 734, with a mean of 277.9. Judge Sotomayor’s statistic is 231, which would place her 59th. (Alito was 70th.)
Quality (2). Judge Sotomayor’s opinions from 1999-2001 were cited 289 times in law reviews and other legal periodicals through May 31, 2004. Judge Wood’s opinions from 1998-2000 were cited 513 times through May 31, 2003 (16th). (Alito’s were cited 240 times (73rd).) Sotomayor would have ranked 65th.
Quality (3). Choi and Gulati also check what they call “invocations”—the frequency with which opinions written by other judges refer to the judge in question by name. They argue that invocations are most likely when the judge in question either has a good reputation or has written a particularly helpful opinion. Invocations range from 0 to 175 (excluding two outliers, the highest is 23), with a mean of 32. Judge Sotomayor was invoked 0 times (tied for last). Judge Wood was invoked 10 times (9th), and Judge Alito was invoked 5 times (28th).
Independence. Judges should decide cases in a non-partisan way; “independence” refers to the probability that a judge will dissent from a majority opinion written by a co-partisan or will write a majority opinion from which a co-partisan dissents. Choi and Gulati use a complicated variable that attempts to measure this tendency, and I simplify here. A score of 0 means that a judge is just as likely to disagree as agree with a co-partisan (or opposite-partisan). Negative scores mean that a judge is more likely to agree with co-partisans. Judge Sotomayor’s score is -0.153 (read the paper if you want to know how this score is calculated), which would have placed her 55th. Judge Wood has a score of -0.018, placing her eighth in terms of independence. (Alito was 10th.)
Choi and Gulati report a composite ranking—but for technical reasons (they controlled for various factors, for example, variation between circuits) it is difficult to put Judge Sotomayor into such a ranking. In addition, Choi and Gulati’s basic approach, which is to give equal weight to productivity, quality, and independence, is contestable, as is their decision to base quality on invocations and citations to top-twenty opinions. They check for robustness by reporting how rankings change as the weightings are adjusted, but their robustness checks are too elaborate to repeat here. Wood ranked eighth; Alito sixteenth. Sotomayor would be in the bottom half. However, I would not put too much weight on this conclusion. Choi and Gulati intended their composite ranking as an experiment to provoke discussion on how best to evaluate and rank judges.
One can, however, make some rough judgments based on the disaggregated rankings. The bottom line is that Judge Sotomayor is about average, or maybe a bit below average, for a federal appellate judge. These results are far from conclusive, but one might think that put the burden on Judge Sotomayor’s defenders to come forward with stronger reasons for her nomination than they have so far. Judge Wood is stronger—I would say that she is impressive, but others might weight the factors differently.
Judicial ability is not the only thing that matters for supreme court nominations, of course. People also care about the nominee’s politics (euphemistically called “judicial philosophy”), meaning how she will vote on abortion rights and related issues. For those poring over Judge Sotomayor’s and Judge Wood’s opinions for clues about their political leanings, a word of advice. Research shows that appellate judges don’t like to dissent, and they don’t like responding to dissents, and so authors shade their opinions to the political center of gravity of the panel. At the time when the data were collected, the seventh circuit consisted mostly of Republicans, while the second circuit consisted mostly of Democrats. That means that Judge Wood’s actual views are probably somewhat to the left of her majority opinions, while Judge Sotomayor’s views are to the right of her majority opinions—in the case of panels with mixed partisan membership. Focus on dissents and concurrences, which better reflect the judges’ actual views, or unanimous opinions when all members of the panel belong to the same party.
A final note. I have done my best to put these statistics together but it is possible that I have made errors. I am reasonably confident about my overall conclusions but not the exact rankings. The data are in any event too crude to allow for precise discrimination. Feel free to take a look at the data yourself and draw your own conclusions. If someone wants to gather data on the reversal rates of the judges, that would also be useful.
Maybe that should have had the authors pause and give thought to their methodology.
Also, something is amiss using Sotomayor's first few years on the bench to judge whether she's qualified for a promotion 9 years later.
Oppo take away: Alito is an independent judge/justice while Sotomayor will be an automatic liberal vote.
Perhaps Choi and Gulati recognize this, and perhaps limiting the set to "top 20" takes care of this, but I wonder if more senior judges might enjoy an advantage because they get to preside more often and take the most important/most interesting cases for themselves. If there's a legal issue of first impression moving through the circuits, no matter how good or bad your opinion is, it will likely be cited just because it establishes a new rule.
On the other hand, anecdotally I can think of a handful of younger judges who probably get cited a lot outside of their circuits.
Justin, your very last comment is addressed at least a little bit in the post by the fact that EP collected 2006 data for Sotomayor. I don't think the post is meant to suggest that Sotomayor is weak (indeed, the comparison to Alito makes me wonder how good the data are), and EP admits that the data "have problems" and represents only that they provide "a firmer basis for evaluation" than Rosen's anonymous sources.
If people declined to post information that was deeply flawed, we might not have any information at all on the prospective nominees. I'm not sure that we'd be better or worse for it, but this post is at least mildly interesting if for no other reason than it shows how difficult it is to evaluate judges objectively and how some people have tried in good faith to do so.
Or more generally, we might not have an Internet As We Know It.
This is only so if you assume that Judge Sotomayor is more conservative than the average Democratic second circuit judge. But it is just as likely that she is more liberal than the average second circuit judge- in which case her views would actually be to the left of the opinions she wrote.
Again, perhaps Choi and Gulati address this in their article, but this seems to me to have very little to do with any particular judge and more to do with the composition of any particular court.
For example, the Current Supreme Court has 7 Republican appointees. Only Breyer and Ginsburg were appointed by Democrats. Breyer and Ginsburg have similar approaches to judging, I think, but Stevens and Souter are often on the opposite side of cases from Scalia and Thomas (though Thomas is often off on his own). If I understand the study correctly, it would probably judge Breyer and Ginsburg as the least "independent" and Stevens and Scalia as perhaps the most "independent." I realize that there's a wide spectrum of opinion on how "independent" or not the particular judges are, but I think most would agree that that measure wouldn't work (though it might if you classified Stevens and Souter as Democrat appointees, though they're not).
As a second example, the 8th Circuit currently has 9 Republican appointees and 2 Democrat appointees (Bye was a Clinton nominee renominated by GW Bush) among its active judges. Including senior judges, the numbers are 14 and 3. Setting aside senior judges for the moment, Judge Bye's or Judge Murphy's "independence" depends solely on how often the other judge agrees with him/her. Including senior judges, the independence of Judges Bright, Bye and Murphy depends entirely on the other two and how they happen to agree. Also, statistically each Democrat is unlikely to sit with another Democrat very often and even more unlikely to sit with another Democrat very often on a case that poses a difficult issue with any potential to draw a separate opinion.
Finally, as people have noted in the past, Carter's appointees tended to be more liberal and Clinton's more centrist, and some Republican appointees are more libertarian than conservative. Calling a judge "independent" or not under that methodology seems to me to not worth a lot.
Source verumserum.com
I think this post is just another example of the phenomenon of trying to put a number on something to give it an air of objective authority. (I’ve named this phenomenon “Law and Economics,” or LE for short.)
Why? You are certainly free to arrive at the conclusion that their method doesn't arrive at or factor in some information that you find important, such as quotes cherry picked out of presentations.
Also, I think people can read independence in a multitude of ways. When a conservative and a liberal take a mainstream position and a third member of the panel takes an EXTREMELY conservative or liberal position in dissent, that matches their partisanship, that will be a show if independence, at least at first glance (I admit the opposite-partisan language confuses me). So a Thomas would come across as very independent, even if his vote was reliably conservative, if he regularly took extreme views and dissented due to his failure to believe in stare decisis.
Well since Alito was ranked 16th overall (out of 74) while Posner concludes that Sotomayor would be in the bottom half, I'm not sure how he was a weaker candidate.
What a joke.
Sometimes -- arguably most of the time -- citations to recent opinions are forced by who is first, not the quality of the opinion. A prime example of this is the horribly mismanaged (by everyone: the litigants, their counsel, the trial court {then-SDNY Judge Sotomayor}, and a couple of prominent amici) Tasini litigation. Judge Sotomayor's opinion received a fair number of citations per force: Nobody else had ruled on it. Ironically, even those citations that reached the same result she did tended to criticize the reasoning in the opinion. Being one of those critics myself probably makes me oversensitive to it... but that's not the only IP-based decision from Judge Sotomayor's pen for which I have the same qualms -- merely the most prominent-to-nonspecialists.
N.B. Thanks in great part to that mismanagement, a procedural iteration of the same "case or controversy" has just been granted cert.
Rather than mere "number of citations," perhaps noting the context of the citations should have gotten more attention. As it stands, though -- even after having read the underlying Choi and Gulati article -- I can't credit this measure as having any real significance.
I wonder how citations that disagree with or distinguish from a decision are handled. One of my colleagues (I'm in hard science) has quite a number of citations on one of his more controversial papers, at least half of which have the substance of "he's wrong". He'll readily acknowledge that the cite count on that paper is a bit misleading.
Generally, I'd say that the criticism in this thread of the original paper is not justified. The authors readily admitted its shortcomings, and merely intended to start a discussion about what it is that we expect from judges, whether their performance can be measured, and, if so, how. That seems to be a perfectly reasonable topic for an article in the SoCal Law Review.
and
This is always a fun game to play. Let's pretend we can quantify judicial merits and then rank the judges! Oh yes, this is empirical data. We're being scientific! Hey, people may actually respect us as legitimate scholars now! Oh boy!
Co-sign.
Indeed. Then again, some of the commenters seem to have a genuine preference for no data as opposed to admittedly flawed today.
Indeed. Then again, some of the commenters seem to have a genuine preference for no data as opposed to admittedly flawed data.
Yeah, because that way it's harder to pretend that we're being Scientific and have something Useful to Say. With "data," the illusion is too tempting.
1. The point of law is common-place. For example, the standard for summary judgment has been settled since the Supreme Court "trilogy" in 1986. Since then, there have probably been tens of thousands of cases which repeat the standard in some way.
That any judge is cited on this point is meaningless -- it just means that he or she can regurgitate what every other judge on the bench has known since 1986.
2. New statutes being explicated. The first time a statute is considered by a Circuit, it will inevitably create citable law (indeed, controlling in that Circuit), simply by virtue of explication of the basic elements of the statute or cause of action. Even after that, the next few cases may touch on areas or sections not considered by the first case. There is nothing terribly brilliant about being the first judge to explicate the details of a statute -- although you will be cited often, especially at the Circuit level.
My bottom line is that the numbers measure something but not necessarily quality. A more aggressive judge will be cited more but that's not necessarily preferable to one who is cautious and minimalist.
Unlike a fair number of empirical works I've read, though, Choi and Gulati don't make any outrageous claims about how good their numbers are or about having conclusively answered anything. They don't do the "we've used numbers so our conclusions must be unassailably correct" thing that a lot of empirical scholars do. Others might take their conclusions and oversell their conclusions ("two law professors measured the productivity and quality of 98 appellate judges and concluded that Alito ranked lower than 70th in both categories--clearly he's unqualified and was chosen for his ideology"), but I don't think Choi and Gulati have done that in the article.
FWIW, I don't think that Choi and Gulati were making all sorts of outlandish claims about their study, or were in any way "in bad faith." And I only think their work is "shoddy" in the "it doesn't really work" sense, not in any sense that would be assailing the personal character of the authors.
At the end of the day, you either think this study is (i) better than nothing, half-a-loaf, as it were, or (ii) an unsuccessful attempt to quantify something that is really hard-impossible to quantify, done in part because so many people think that rankings and numbers mean more rigorous analysis. I'm in the second camp.
Cost-benefit analysis applied to regulation also Sucks, am I right?
Nick
Also: Invocations range from 0 to 175 (excluding two outliers, the highest is 23)? Hi, Dick! Hi, Frank!
I didn't think there was bad faith in the article. Sorry if that was the impression I gave. I generally agree with Joseph Slater on his latest point at 2:01 pm.
Cato:
If you rely exclusively on it, or if you fail to take into account its weaknesses, yes, I would say CBA does suck when applied to regulation. Does it have its place? I think it does. Is it the be-all, end-all for regulatory analysis? Nope.
You don't need to divine that belief from obscure sources -- he comes and says the G/C is flawed!
Yes, anyone ever expressing an opinion with which I disagree should be disqualified.
Damn her for hoping!
I think the median would be more useful here. Or, at least, adding the mean excluding the outliers.
I don't discount the unquantified, unsystematic views of experienced advocates who've appeared before lots of judges, but I wonder whether too much of what we think of as judicial reputation is based on a handful of hot button cases where the judge came out our way or flashy dissents.
Yes, I think the study is pointless. Are those who did it schmucks for doing it? No. Do I think people who embrace these types of studies need to be warned (well-nigh constantly) that just because they have quantified something and conjured some statistics it doesn't mean that they're being "empirical" or scientific? Yes. Was that my point? Yes. Am I sounding Rumsfeldian yet by asking and answering my own questions? Maybe.
Law review citation seems likely to be more a product of writing controversial opinions or opinions on hot topics like constitutional law, rather than writing good opinions. Law reviews topics are not representative of all legal writing.
Invocation is also dubious. This may be as much as anything a product of the writing style of third parties, particularly within the circuit. It may indicate fame, but we want good judges, not necessarily famous ones.
Citation by fellow judges does seem like an indicator of quality.
Independence as a measure of partisanship also seems relevant in a political appointment, although not necessarily as a measure of "quality" that should be included in a composite ranking. It may also indicate a disinclination to build consensus, as opposed to non-partisanship.
I actually heard Cass Sustein talk about the judicial independence of Alito and why his marks are deceiving. It turns out that Alito was the most partisan judge of the circuit courts, meaning that he was substantially more likely to support administrative rules promulgated by Repulican administrations than Democratic. This created a false positive for his independence marks. Yes, he was independent. But the only reason Alito was independent is that he often broke with two conservative judges to the right to rubber-stamp rules promulgated by Republican administrations. Thus, Alito isn't a movement conservative -- driven by conservative ideals -- but a party hack.
I know that there are almost no Amerikan women who do science, math or engineering, buy why not appoint Miss California, then?
Law School is the futile attempt to turn a humanities sow's ear into some kind of a useful silk purse. I guess it works sometimes, but why not start with an undergrad in something substantive? (Answer: almost all would be white men.)
Yes. Elections have consequences.
Quite a bit.
Quite a bit.
So I have been told by my betters, though not by people who practiced before them, and so I fervently believe. But why do I believe it? And why do you? In what does this asserted, but rarely argued, superiority of Cardozo to Pound consist? We all think it obvious, but none of us can explain why. Richard Posner tried to assess the reasons for Cardozo's historical reputation, and it came down to, as I recall, quotable prose, a couple of landmark cases for which he got to write the prevailing opinion, and the fortuity of a Supreme Court appointment, even if his short tenure there added little to his actual accomplishments. Is judicial quality really so evanescent a thing as this?
Not that American ethnic politics ever made sense, but this is more nonsensical than most.
To see what a glowing evaluation looks like, check out the lawyers' evaluation of Samuel Alito at the time he was nominated for the Supreme Court. (Like Sotomayor, Alito was a federal court of appeals judge then):
Read the whole thing.
It should be up to the party (taken roughly as a proxy for people of a certain ideology) to decide questions of quality. Why? Because quality is in largely in the eye of the beholder. Liberals generally, though not exclusively, think that Scalia is a hack and Brennan a champion of justice; conservatives, of course, think the opposite. The perception of quality usually depends on the strength, boldness and clarity with which a judge articulates the viewpoint of his or her ideological compatriots. Those same qualities can inspire horror from ideological opposites.
The minority party, on the other hand, should focus on whether they can live with the nominee's philosophy and ideology, keeping in mind that they are unlikely to get anyone they actually like. It should be a good faith effort on the part of both parties to find a reasonable "consensus" candidate. Sharp ideological divisions are harmful to justice and harmful to the courts.
If you're interested, you can read the full address.
The key words here are "lived that life." This quote is pulled out of a section of her talk addressing cases involving racial and sexual discrimination, and how minorities and women were instrumental in changing the legal landscape.
In context, she is making a valid (almost obvious) comment: white males have not experienced racism or sexism. That does not mean they aren't qualified to rule on cases involving racism and sexism, simply that they have no life experience with those concepts.
Vrtra Theory writes:
This is of course wrong. Most white American males know anti white racism all too well. They or family members are routinely passed over at school, at work, for awards, promotion, selection, advance or recognition. The firemen case currently before the courts is one where white men are being denied promotion due to their race. This has gone on for forty years.
As a friendly neighbor we have watched with amazement as this racist practice grew for well over a generation. It is, of course, an American Apartheid system. That is what is under challenge by the firemen.
These racist and sexist practices have always shocked your neighbors and friends in all OECD countries. That is not the America they cheered a generation or two ago for its move towards equality. It is now institutionalized inequality.
Running the numbers judges would have known that forty years of applied racism - excluding white people from schools, jobs, promotions, awards and the benefits of a free and democratic society based entirely on their skin color would lead to a corrupt system of apartheid with the majority - the average white family in the US - having been "screwed" by the system.
If you have been out of math for a while start with an excel spreadsheet. There is extensive literature on admissions and jobs from the net. If you have a friend or contact at any university which is part of the Consortium of Financing Higher Education (COFHE) they might give you some private advice. They have some information in this area. They could lose their jobs – it is that bad. Ask them about that. If you are a judge or lawyer who hasn’t explored this issue you will be unpleasantly surprised. When visiting a US campus and chatting with contacts about this issue they now look over their shoulder. When thinking of offering litigation support pro bono think again. This may be sad – but “Tthe truth will set you free in an entirely new direction”.
Our best contacts, expert in this area, are afraid. They are afraid of losing tenure and grants and physical injury on and off campus. Several of you will know a colleague or two in this situation. What in the world happened to America? Is Senator McCarthy running the Affirmative Action Program?
Bad for everyone. This is a recipe for dislike of the favored group, anger at the racism, revulsion at a legal system which enforces it, and contempt for the smug and shallow who pretended they didn't know this would happen. There were no unintended consequences, it was apparent from the beginning.
Replacing one racist tradition with another. Brilliant.
Corruption of US universities. The drive by corruption of your universities, freedom of speech, freedom of research and suppression of dissenting expression on campus and especially in the social sciences is now expected. This system has done permanent damage to the “quota” as well as the “excluded” is a huge cost which is not normally discussed. There is a certain amount of literature on the cost to the “quota”. The cost to these people who were patronizingly introduced into schools and jobs is high. Now the natural tendency of the hard sciences and math to walk around looking at the social sciences with a “Me Tarzan – you Jane” is warranted since they can argue points in several of their areas without fear.
You now have a situation where possibly three or four times the number of white families in the US have faced racial prejudice than the entire black family population of the country. The America Apartheid solution: ironic yes, fatuous yes, a serious solution no.
Berkeley Comment was done in jest. One assumes that Sotomayor is suggesting that we need people on the bench who understand prejudice from a real world perspective? That would be a white male in the United States of America in 2009.
If you have a comment about spelling, typos, or format errors, please e-mail the poster directly rather than posting a comment.
Comment Policy: We reserve the right to edit or delete comments, and in extreme cases to ban commenters, at our discretion. Comments must be relevant and civil (and, especially, free of name-calling). We think of comment threads like dinner parties at our homes. If you make the party unpleasant for us or for others, we'd rather you went elsewhere. We're happy to see a wide range of viewpoints, but we want all of them to be expressed as politely as possible.
We realize that such a comment policy can never be evenly enforced, because we can't possibly monitor every comment equally well. Hundreds of comments are posted every day here, and we don't read them all. Those we read, we read with different degrees of attention, and in different moods. We try to be fair, but we make no promises.
And remember, it's a big Internet. If you think we were mistaken in removing your post (or, in extreme cases, in removing you) -- or if you prefer a more free-for-all approach -- there are surely plenty of ways you can still get your views out.