pageok
pageok
pageok
[Einer Elhauge, guest-blogging, May 22, 2007 at 6:01am] Trackbacks
Sabermetrics and the Future of Legal Empirical Studies

The book I have recently read that I think may offer the most insight into the future of legal studies is, of all things, a book on sabermetrics, called Baseball Between the Numbers. Sabermetrics, for those of you not into baseball, is the advanced statistical analysis of baseball. With Bill James as its most famous pioneer, it raised all sorts of probing analysis about which statistics really best determined the value of a baseball player, and what sorts of strategies worked and which didn't.

So what does any of this have to do with law? Well, what this book does is compile, in a readable way, the major points this advanced statistical analysis has taught us over the last few decades about matters that were previously resolved by tradition, customs, and intuitive reasoning. Some of those traditional views turn out to have some basis, others none or only in a limited way.

For example, advanced statistical analysis shows that batting averages are a useful statistic, but much less important to winning than on-base-percentage. RBIs are largely a distraction, and hot streaks and clutch hitting are stories we tell ourselves to describe statistical clumps that are really just random. Pitchers vary in their ability to strike out batters and avoid walks and home runs, but have little effect on the odds that balls hit in play will become outs, so their ERAs are worse predictors of their future performance that their rates of strikeouts, walks and home runs. The bunt is hugely overused, and generally reduces the odds of victory, other than in a few instances that can be specified with precision.

And it occurred to me that in law we now are largely where sabermetrics was in the early days when Bill James began cranking out his seminal Baseball Abstracts. The bulk of what we teach our students reflects tradition, customs, and intuitive reasoning. Little of it has been subject to rigorous statistical analysis.

In Contracts class, for example, I regularly teach that we can understand all contract law largely as default rules that either reflect what most parties would want or are thought most likely to trigger an explicit contract provision. Then we explore how courts and scholars have resolved such issues, which is largely through armchair reasoning. The issue cries out for rigorous statistical analysis, and we have little to offer.

In Antitrust, much turns on how we think firms are likely to behave. After a merger, will firms engage in Bertrand competition by pricing down to cost, Cournot competition by setting output in a way that depends on the output of others, or oligopolistic coordination on price or output? The traditional approach considers various factors that theoretically bear on this issue in particular cases, but the weighing of them generally turns on unavoidable judgment calls. It would be better to rely on the growing statistical analysis of how firms actually behave (often, it turns out, in ways that lie in between these models). It would be even better to have rigorous statistical data about what the price effects were of a particular method for deciding which mergers to approve or condemn. Right now we choose our merger law methodologies based on theory and never gather and analyze the data to see whether the theory worked.

We are probably even further behind in empirical analysis of basic legal strategy. What sorts of arguments are most effective with judges? Which with juries? Which sorts of contract design are most likely to avoid disputes latter? Which settlement offers are most likely to be successful? These are important things to teach our students, but all we can do is either tell them the received wisdom (which may well be wrong) or avoid discussing these issues (so as not to expose our ignorance).

In short, in law, we are currently still largely in the position of the baseball scouts lampooned so effectively in Moneyball for their reliance on traditional beliefs that had no empirical foundation. But all this is changing. At Harvard Law School, as traditional a place as you can get, we now have by my count 10 professors who have done significant statistical analysis of legal issues. We just hired our first JD with a PhD in statistics. The movement is not at all limited to Harvard, and seems to be growing at all law schools.

So we are hardly devoid of empirical analysis of law. We are just, rather, in our early Bill James era, and can expect the analysis to get more sophisticated and systematic as things progress. I expect within a couple of decades we will have our own book distilling the highlights of things we will know then that conflict with what is now conventional legal wisdom.

None of this means this new empiricism will replace traditional legal theory, much like sabermetrics has not eliminated the need for scouting. Indeed, it is clear to me that a lot of legal empirical analysis misses the boat because it has a poor or thin understanding of legal theory. Many empiricists are good at providing useful input to policy analysis, but surprisingly terrible at doing policy analysis about the implications of their own findings. There will also be some growing pains because it is not clear that empiricists are the best placed persons to teach law students, given that the students themselves need not learn how to do statistical analysis to become excellent lawyers.

But I have no doubt that empirical analysis of law will provide the biggest contributions to our understanding of law over the next few decades. That is where the low-hanging fruit is. The decline of doctrinalism will only accentuate this trend. Because anti-doctrinalist law professors can no longer persuade lawmakers with claims about what the law inherently must mean, they will find it more promising to try to influence lawmakers with findings about what effects particular laws would have. This brings us to my next topic, the death of doctrinalism, which will be the subject of my post tomorrow.

jack (mail):
Forgive me, but the first question that this conjures in my mind is:

Who's the legal equivalent of Joe Morgan?
5.22.2007 8:25am
justanotherguy (mail):
From my viewpoint (physics major, career spent in nuclear engineering and now a newly minted JD) statistics will not be the panacea to correct the obvious deficiencies in legal analysis and law. My crystal ball shows that absent market forces, the use of the new technique will be captured by those with agendas and it will end up being used by both sides in any debate.

This is exactly what happened in the public policy arena. Statistics and studies are found and used to support any proposition just as expert opinions are used to support any position in large tort cases. Daniel Bell wrote decades ago about the capture of the technologist, why would law be any different?

In baseball there are obvious market forces that reward and/or punish the rational use of facts. Those same forces are too remote or delayed to matter in law. Ask Dow about studies about scientific analysis on breast implant safety... oh I forgot, that's difficult since junk science and the Plaintiff's bar forced that company into bankruptcy in the 1990s.
5.22.2007 11:17am
Einer Elhauge (mail) (www):
Do you mean Joe Morgan the hall of fame second baseman or Joe Morgan the Red Sox manager of "Morgan Magic" fame? Either way I laughed out loud.
5.22.2007 11:19am
frankcross (mail):
That's pretty cynical justanotherguy, people do try to manipulate statistics, but there are limits to their ability to do so. For example, the cigarette companies put a lot of effort into the failed attempt to prove cigs harmless.

As for the law, it is true that market forces help encourage the search for truth. And you see litigators doing lots of research already on things like composition of juries.

The law schools, though, are definitely moving in the direction of PhDs and use of statistics. And we're making progress on issues like the degree to which judges decide ideologically and what factors may influence whether judges decide ideologically.
5.22.2007 12:31pm
LGVV:
He means the Hall of Fame Second Baseman (and overall terrible announcer and "expert" enemy of statistics in baseball) Joe Morgan.

I think statistical analysis might actually replace a lot of scouting, and success measured against a certain standard level of competition may one day be about as good in predicting what prospect moves on as scouting is today. Not there yet, but eventually, who knows. Scouts aren't all that accurate themselves.

One big difference between law and baseball, though, is that baseball's a game (obviously) where rules ensure that similar situations are going to occur over and over and over again. Any small differences between one at-bat and another, for instance, will likely get wiped out in the long run. I'm not so sure the same is true in a legal atmosphere--is a case just a case and an argument just another argument that can be logged and categorized? All the differing real-world circumstances and ideologies might make "legal stats" much less useful.
5.22.2007 12:33pm
Mark Field (mail):

Do you mean Joe Morgan the hall of fame second baseman or Joe Morgan the Red Sox manager of "Morgan Magic" fame? Either way I laughed out loud.


He means the former. Morgan regularly disparages sabermetric analysis in his television commentary. That's ironic, in a way, because Morgan could easily have been the poster child for "sabermetric virtues". He had a high OBP, stole bases with a low CS %, etc.
5.22.2007 12:34pm
steve lubet (mail):
Empirical studies are important, but most situations faced by lawyers and judges are immeasurably more complex than the finite set of choices facing baseball managers. Before anyone puts too much faith in statistics, check out Jerome Groopman's new book, How Doctors Think, regarding the shortcomings of "evidence based" medicine.
5.22.2007 1:07pm
Kovarsky (mail):
I don't know who Joe Morgan is but John Yoo is Barry Bonds.
5.22.2007 1:14pm
Kovarsky (mail):
I would argue that the area most in need of statistical rigor is intellectual property law, where various issues in both patent and copyright would ideally be sensitive to what we know about how exclusivity interacts with incentives to produce intangible assets.
5.22.2007 1:38pm
Blastings Thrilledge (mail) (www):
The souls of human beings are not subject to empirical analysis, and what law concerns (contrary to the belief of modern political theory) is the souls of human beings. Now, if you do not believe that human beings have souls, then certainly apply statistical analysis to people and perform regression analyses to determine which types of laws have the most positive correlation with law-abidingness or whatever, but you would miss the point.

Sabermetrics is essentially the science of subjecting baseball to rational analysis; that is, using the tools at hand in the best way possible to understand the subject of baseball. That begins by understanding what baseball is- a game is 27 outs on each side. This is the first principle of using numbers to understand baseball. It is very simple; it is self-contained; it is extremely finite, though the study of this closed system opens up immense possibilities and other fields of baseball study.

Law, on the other hand, is not so simple. The possibilities of human life are not finite; 'what is good,' 'what is just,' and what is 'pleasurable,' are not quantifiable; the relations of human beings, with each other and with God-the divine- are not self-contained but point always to otherness.
5.22.2007 2:15pm
Kovarsky (mail):
Blastings,

I don't think anybody is suggesting a quantitative approach to the due process clause; but there are certainly some areas - e.g. patent, copyright, antitrust - that can certainly benefit from rigorous statistical analysis.
5.22.2007 2:57pm
Brasher:
The one place that I can think of that statistical analysis has been applied with significant success is jury selection. An expert attorney or consultant can, through random sampling of the local population, screening of sample closing arguments, etc., determine with a high degree of certainty the outcome of a trial. Although this kind of sabernomics is not that relevant to answering "what is good" or "what is just," it determines the settlement outcome in many more cases than law students (and maybe professors) realize.
5.22.2007 2:57pm
Tillman Fan (mail):
I tend to agree that law is more complex than baseball, but I still think that "sabermetrics" is likely to play a role in the future. One area that I've been thinking about concerns simple statistics regarding number of, for example, employment discrimination cases filed in particular jurisdictions, with an analysis of how many are dismissed, how many are settled, how many are tried to verdict and the amount of each verdict. This information could be further broken down by presiding judge or specific county court. It seems to me that this would be useful information for employers and their lawyers in trying to place a value on cases. It also might lead to some fact-based forum shopping for more favorable venues.
5.22.2007 3:50pm
Rev Transit (mail) (www):
My graduate school education was rec.sport.baseball.
5.22.2007 4:29pm
David M. Nieporent (www):
Mark, I'm _shocked_ to find you in this discussion.

And James! So that's where you've gone.
5.22.2007 8:29pm
Mark Field (mail):

Mark, I'm _shocked_ to find you in this discussion.


I'm sure you are, but I'm not so sure why.
5.22.2007 9:05pm
Peter Young (mail):
It's a game for pansies and geeks and people with funny-sounding foreign names. Ooops, wrong thread; I thought this was the soccer post. But come to think of it, Minoso, Kluszewski, Berra, Cepeda, Zernial, Marichal, DiMaggio, Gehrig, Campanella ....
5.23.2007 4:24am
martinned (mail) (www):
L.S.,

Just to move away from baseball for a moment, what about the line of literature in legal academia that starts with Macaulay and McNeil? (i.e. relational contract theory)
That started out as an attempt to ground legal studies more in reality...

Refs:
Macaulay (1963), 28 Am. Sociological Rev. 55
Macaulay (1977), 11 L. &Soc. Rev. 507
MacNeil (1983), 78 Nw. U. L. Rev. 340
Bernstein (1992), 21 J. Legal Studies 115
Fox (2003), 54 Case W. Res. L. Rev. 1
5.23.2007 6:38am
Mary Katherine Day-Petrano (mail):
"One big difference between law and baseball, though, is that baseball's a game (obviously) where rules ensure that similar situations are going to occur over and over and over again. Any small differences between one at-bat and another, for instance, will likely get wiped out in the long run. I'm not so sure the same is true in a legal atmosphere--is a case just a case and an argument just another argument that can be logged and categorized?"

You should get the tape of Associate Justice Alito's speech at the 2007 Tampa Bay Area Federal Bar Assn. Gala on baseball and the law. A speech that was unforgettable.
5.23.2007 11:13am
Mary Katherine Day-Petrano (mail):
"Law, on the other hand, is not so simple. The possibilities of human life are not finite; 'what is good,' 'what is just,' and what is 'pleasurable,' are not quantifiable; the relations of human beings, with each other and with God-the divine- are not self-contained but point always to otherness."

What a bunch of balony. Someone must have eaten one too many of those psychedelic mushrooms.

I agree with Kovarsy that there are some areas of the law where a little Sabermetrics would go a long way toward sound decisions and justice. A failure to provide required auxiliary aids and services necessary for effective communication case under the Americans With Disabilities Act would be another candidate for some Sabermetrics.

A classic example: When rules decisions are made and statutes enacted that require things like (1) paper pleadings vs. total electronic internet virtual courthouse access, or (2) certain time deadlines, a person experienced with ADA jurisprudence and hundreds of different kinds of disabilities in varying combinations and permutations has to ask the following question --

Did some committee of *non-disabled* perfect people raised in a hard copy penicl and paper era just arbitrarily decide these things based on "tradition, customs, ... intuitive reasoning, and ideology?"

Or did anyone actually sit down and conduct empirical studies on the problem to analyse the sub-tasks required to present pleadings in hard copy paper format (transportation to place with copy machines, manual dexterity to collate and sort the paper, no dyslexia so the pages and exhibits can be put in the right order, ability to see and read teh paper print, ability to reach the copy machines, etc), and within the completely arbitrary and capricious time deadlines for each of these categories (query, are they all set by the amount of time it takes an entire Big Law firm staff of attys and paralegals and legal secretaries with all the bells, whistles, and trappings of modern technology including access to LEXIS or WESTLAW to do it?), including but not limited to --

* paraplegic wheelchair bound
* cerebral palsy
* multiple sclerosis
* chemical sensitivity
* carpal tunnel with developing diabetes (we now know chief Justice Robert's Toyota Motor case was not as simple as mere carpal tunnel, since science now knows carpel tunnel indicates developing diabetes)
* no arms
* autistic
* deaf or hearing impaired

* Someone with most of the above

Those who have made these types of decisions of (1) format used in the court system, and (2) amount of time necessary to accomplish the teasks involved in delivering that format, would have to empirically analyse, for just one example, how long does it take a wheelchair user to get dressed, leave his house, get his wheelchair on the hydraulic lift, drive to a Kinko's, take his wheelchair off the hydraulic lift, find a way to manuever his wheelchair thru the Kinko's doorway, and find an accessible table to sort the hard paper copies pleadings and an accessible height copy machine -- for their to be a sound basis (not just conjecture and speculation) to support a particular format and time deadline.

Now, how does the format and time deadline requirement need to empirically change to provide Due Process and Equal Protection to, say a wheelchair user missing his fingers so he cannot push the buttons on the Kinko's copy machine?

And what changes are required if the wheelchair user is missing fingers and blind?

To pontificate all the pie-in-the-sky 'God is Divine' 18th century philosophical balony as the basis is, for some, tantamount to not having a judicial system, since a judicial system based on such nonsense is not one accessible to all people.

Moreover, such nonsense is the source of teaching things like credibility can be determined by trial judges who look a person in the eyes and take note of fidgeting, etc., that when applied to an autistic, all looking into an autistic's eyes is going to reveal is autism and the stimming behavior that is doagnositc of autism, which does not provide any sound, valid measure of truthtelling.

For many years now, law schools have been graduating joint J.D./M.B.A. candidates who do have the legal training with the statistical empirical education, and for many years such degree candidates have been laughed out of the law schools. But, even just on a plain "law" basis, decisions are not supposed to be made on speculation and conjecture or steereotypes that are, in fact, not true or accurate -- what is the alternative?

Laywers need to learn and use more scientific, technological, empirical methods of arriving at decisions and conclusions.

I say this, by the way, as a graduate of a Catholic Jesuit law school steeped in the traditions of religion and philosophy -- and science.
5.23.2007 11:44am
Mary Katherine Day-Petrano (mail):
"One area that I've been thinking about concerns simple statistics regarding number of, for example, employment discrimination cases filed in particular jurisdictions, with an analysis of how many are dismissed ...."

Exactly, and an analysis of Americans With Disabilities Act cases show appx. 95% are dismissed by federal judges who do not understand either disabilities or science and technology.

Of course, the unspoken motivating reason for such statistical finding is more likely that federal judges (whose bretheren are state court judges in their districts and circuits) are afraid if they let disabled ADA plaintiffs win, then more will sue State court judges under Title II for monetary damages in both individual and official capacities, execute their judgments on the judge's personal assets, and garnish paychecks -- and this idea just freaks them out.

But then, we get the abusive excesses of such speculative and conjectural judicial decision making like that apparent in Goodman v. Georgia, US 2006, where even a unanimous Supreme Court could not stomach the cruelty.
5.23.2007 11:52am