Law professor blogs love to debate the law review submission process, and in particular the pros and cons of student-edited journals. The most common complaints about the current system are that placements reflect author/school prestige and students just aren’t informed enough to separate better articles from worse ones. Over at Prawfs, Fabio Arcila comments: “There seems widespread dissatisfaction with [the existing] state of affairs, yet inertia continues to reign.”
Here’s a way to get over the inertia, or at least to get real empirical evidence of how serious the problem might be. I propose a study comparing placements to peer assessments that would work like this:
(1) Pick 10-15 articles in a particular area of law accepted for publication in a wide range of journals in the last year.
(2) Ask 10-15 accomplished scholars in that field to rank the quality of the articles (with author names and journal placements removed).
(3) Compare the explicit scholarly ranking with the prestige (and thus implicit ranking) of the law review placements.
If the assessments of the accomplished scholars closely or roughly match the assessments of the journals as measured by journal prestige, then the complaints about student selections and their overreliance on schools and authors probably don’t mean very much. On the other hand, if the journal placements and the assessments of top scholars show little or no correlation, then I think the study would provide a very real boost to the complaints about the law review placement system.
Oh, and I acknowledge that asking for the evaluations of top scholars has its own serious methodological problems. Those scholars have their own biases, and they may be influenced by author identity even if not provided (as most fields are pretty small — an expert can often figure out the author). But those biases don’t matter for these limited purposes, as the major complaint about student-edited journals is their inferiority to assessments by scholars.
Joseph Slater says:
To get my bias out in the open, I think the law review publishing system is close to comically indefensible: second and third year law students (little peer review) reviewing thousands of articles in a season (no time to look even semi-seriously at most of the articles submitted), and usually no policy of blind admissions (hence the “letterhead” bias issue).
But if I wanted to do something like the test Orin proposes, I would do it a different way. Ask an experienced scholar in the field to list, say, the best 10-15 articles to come out in that field in , say, the last 5-10 years (insist they exclude their own, of course), then check the level of placement.
December 7, 2009, 5:15 pmA. says:
Notwithstanding the interdependence of the variables, it would be interesting to regress citation counts on journal of publication of articles. Presumably, gems of articles that are missed by the imperfect LR selection process would be heavily cited despite being in less prestigious journals, though perhaps less than they would be if they were published in more prestigious journals. A more sophisticated model would also control for author prestige by controlling for author publication count, citation count, and school. The more uniformly the citations are distributed, the more obviously the system is broken. Conversely, the less articles outside the top journals are cited, the more likely it is that the system is working well (or that no one reads the second-tier journals, which has its own lessons to teach us).
December 7, 2009, 5:54 pmT.J. Chiang says:
I think the problem might be more than the scholars being biased by the author identity, the scholars may even be biased by the actual journal placement (which would, I think, seriously taint the reliability of the test). Because fields are so small, and it is our job to keep up to date on the literature, we often know both the author of an article and where it placed. Indeed, often it is the knowledge that an article placed highly that induces one to read it.
December 7, 2009, 6:27 pmCurt Fischer says:
That seems like a much worse design than Prof. Kerr’s proposal to me. The problem is that your design will not distinguish the circulation/pretige of the journals in question from the scholarly quality of the articles therein. In other words, part of what determines how likely a scholar will rank an article as the “best” is how likely he is to read it in the first place.
December 7, 2009, 8:38 pmfrankcross says:
How about ex ante assessment. Pick the best articles from a group just posted on ssrn, rate them, and see where they end up.
December 7, 2009, 9:08 pmloki13 says:
As a (former) part of the problem, I can tell you that at its worst, the LR submision process is comically bad. At it’s best, it provides something nice on the resumes of law students.
Here’s a few facts-
1. Reviewers go through so many articles they cannot devote a serious amount of time to any particular one of them; while ones that make the cut get further scrutiny, very good articles are often weeded out by overworked gatekeepers for no discernible reason.
2. Moreover, these gatekeepers often have no knowledge of the subject matter of the article. You could have someone reading an immigration law article that has never taken immigration law, admin. law, or even con law.
3. There are bizarre metrics used that aren’t disclosed- like, for example, minimum number of footnotes per page. And if you don’t meet it, you get kicked out. Go figure.
4. Here’s one of my favorite examples. Take two all-time classics- Coase and Brainerd Currie. Both of them wrote THE articles in their respective fields (L&E, Conflict of Laws). Neither of them would likely get published today. How do I know this? As an experiment, I circulated them to the reviewers at the LR (along with some other articles) to calibrate how the review process was working. They were both unanimously kicked out. I don’t know if I was more saddened that neither could have made the cut, or that no reviewer realized what they were reading.
December 7, 2009, 9:26 pmTweets that mention The Volokh Conspiracy » Blog Archive » Measuring the Accuracy of Law Review Placements -- Topsy.com says:
[...] This post was mentioned on Twitter by Eugene Volokh, Eugene Volokh. Eugene Volokh said: Measuring the Accuracy of Law Review Placements: Law professor blogs love to debate the law review submission p.. http://bit.ly/5WgeT5 [...]
December 7, 2009, 10:00 pmGU says:
People are focusing on the fact that law review editors read so many articles that they cannot give enough time to carefully consider all (or most) of them. But how will the faculty take over these duties so easily? On my Top LR we have seven people reviewing articles (and we receive thousands of articles every year—thanks Internet!). The average submissions editor spends 15-20 hours a week reviewing articles. How many professors really have time (and desire) to do this?
Maybe I’m wrong, but if legal journals decide to go the more traditional peer review route, then we will have significantly fewer journals. Competition to just get published, let alone in a top journal, will increase all the more. And with peer review, the “peers” will tend to keep out reasonable work that diverges from their preferred orthodoxy (it happens in every field, to varying degrees, that use this “gatekeeper” system).
My take is that calls for drastic changes to the law reviews are mostly sour grapes from professors who had bad luck on an article or two. (“Only a stupid law student wouldn’t recognize the brilliance of my article! Laurence Tribe would have seen it immediately!”). If you’ve ever been on the submissions committee of a law review, you know that most articles submitted are garbage.
December 8, 2009, 1:18 pmbyomtov says:
Are law review submissions refereed? That is, do editors send out articles they tentatively like to scholars in the field for comments?
December 8, 2009, 9:07 pmKARSILIKSIZ CEK says:
If you are a current Duke Law student who applied to one of the school’s journals, you have probably emailed Above the Law in the past two hoursConversely
February 25, 2010, 11:17 am