Law Professor Takes on Law School Exams:
Northwestern Law Prof Steve Lubet, who I believe runs the clinical programs at Northwestern, has an article in the American Lawyer arguing that traditional law school essay exams are in need of major reform:
There is almost nothing about the typical law school examination that is really designed to test the skills involved in law practice. And many aspects of exams are positively perverse. Take time pressure, for example. By their nature, exams are time-limited, usually to about three or four hours, during which it is necessary to assess the problems, decide on the answers, marshal the material (whether strictly from memory or from an "open book"), and then write, hopefully, coherent answers. There is no opportunity for reflection, research, reconsideration or redrafting. You simply dash off your answer and hope you got it right.
He continues:
The dirty secret (if it is a secret) is that law schools rely on exams primarily because they are easy to grade. The intense time pressure guarantees that the answers will be relatively short and, even more important, that quality will differ significantly. Exams do a great job of dividing test takers into measurable categories, even if those categories measure nothing more than an ability to take tests in an artificial, nonlawyerly setting.
  I have worried a lot about about this same dynamic, although I don't think the situation is quite as bad as Lubet suggests. I suspect the reason for the traditional dominance of in-class three-hour law school exams is less the need to generate short and varied answers than it is to limit opportunities for cheating. Take-home exams are the most common alternative to the traditional approach; they offer the benefit of giving students the opportunity to reflect at length on their answers in a way that is a bit more like most types of legal practice. But take-home exams also create a window for unethical students to bend (or break) the rules.

  With that said, I don't know why law schools couldn't increase the amount of time allowed for in-class exams and then impose word limits on answers. Most law students take their in-class exams on laptops these days, which would make the shift to word limits easy to administer. This approach would be fairer for students who perform less well in the highly pressured atmosphere of a three-hour exam, and wouldn't impose a substantial burden on professors.

  Any thoughts? Should law schools that follow the traditional approach consider switching from traditional three-hour in-class exams to five-hour or six-hour in-class exams with word limits on answers? I am assuming the exams are open book (which I think is the most common approach, and obviously the approach that most closely resembles legal practice).

  I have enabled comments. Thanks to CrimProf Blog for the link.

Alan (mail):
Oh I don't know. The Bar Exam measures your ability to know many more subjects in a much shorter period of time.

While I agree that law school exams don't test skills you'll need in practice, necessarily, they do provide valuable skills. You will need to be able to analyze and make decisions more quickly when in court to timely object and preserve your appeal.

Personally, if professors are that concerned with preparing their students for practice, they should lobby to make the skills-based classes a mandatory part of graduation. Require a student to take mediation or negotiation or trial practice, just like you have to take ethics.

Just some thoughts.
3.8.2005 12:34am
Eugene Volokh (www):
I agree entirely about the time limits; I do give 8-hour open-book, open-notes exams (with word limits), precisely because I don't want to grade students on their ability to work under extreme time pressure. The great majority of legal tasks require quick work, but work with at least some time to reflect.

I actually used to give 48-hour take-home exams, chiefly because I thought they'd be more realistic. But students hated them, and for understandable reasons.
3.8.2005 12:53am
Jake Kreutzer (mail):
I think what many professors don't appreciate (or don't care about) is that in a strictly curved class you aren't doing students any favors by giving us extra time to work. A four hour in class exam requires four hours of intense concentration. An eight hour in class exam requires eight hours of intense concentration. My answer on the eight hour exam will probably be better than the answer that I would have come up with for the four hour exam, but that doesn't matter- what matters is how my answer stacks up against everybody else's answer.

Also, I feel that the problem with word limits is that everybody asymptotically approaches the same answer. There is only so much discussion that can fit under the word limit, so everybody winds up discussing the same sized subset of issues. Your grade winds up depending on some combination of randomness plus which subset of issues the professor finds more important (which is a different type of randomness).

A similar phenomenon happens when an exam has too little content for its duration. One exam we had last year was 4 hours long and had about 3 hours worth of stuff. Result: most people hit everything, and grading was essentially random for the top half of the class. Good times.

And then there's the whole issue of grading four months of work based on a single test...
3.8.2005 1:13am
Best exams I've taken in law school were those with relatively short, and strictly enforced, word or page limits. I've generally known better while taking those exams whether I was on target, helping to avoid the feeling of randomness that can be associated with receiving grades on essay tests where questions consist of little more than "Antitrust law is muddled - Discuss."

Plus, if the professor expects the students to think so precisely about the question, chances are the professor has thought about it at length as well, and hasn't just thrown the exam together to beat the registrar's deadline.
3.8.2005 1:17am
jtuck (mail):
Unless being able to type quickly is what differentiates good lawyers from excellent lawyers, many law school exams do a poor job of ranking students. Specifically, I found my performance on time pressured issue spotting exams correlated strongly with my typing speed. This makes sense when an exam has more issues than time and the curve is generated based on how many issues the student addresses.
However, the correlation didn't hold on exams that simply asked fewer tougher questions. Often these exams would come with a word limit or a mandatory 45 minute planning period where no writing was allowed. Performance on these exams seemed exams seemed to reflect our ability to engage in legal reasoning under pressure rather than our capacity to apply rules to facts at a higher speed than our peers.
3.8.2005 1:38am
Robert Lyman (mail) (www):
Ugh, who wants to sit in a classroom even longer than we already have to? It might be fairer or more realistic, but it would be even more awful for students than it is today. And it's quite awful enough as it is.

The evidence I have indicates that almost nothing about law school has anything to do with the actual practice of law; it has to do with the intellectual quirks of law professors. I bet you could be a far better lawyer after 3 years if you went straight to practice after learning the basics in your 1L year. The firm you're working for could send you back for a year or even two of specialized training after you've settled on a specialty, rather than forcing you to guess what you might be interested in down the line. So why the focus on exams? They may be a symptom, but they aren't the problem
3.8.2005 6:34am
Nick B. (mail):
My heretical view is that the scores needed to pass the bar exams should be dramatically raised and that graduating from law school should not be a requirement for sitting for a bar exam.

I'm not holding my breath.
3.8.2005 8:39am
James Grimmelmann (mail) (www):
Far and away the least stressful exams I've taken in law school were 24-hour take-home exams with strict word limits (limits shorter than for some in-class 3- or 4-hour exams). The ratio of time to space means that it's difficult to stress or spend too much time working, because there just isn't enough to do to take up all that time. But those of us in those classes also agreed that our answers were much better (and definitely better-written).

To deal with the all-answers-converge-to-identity problem, I've seen professors try several things. One is to ask open-ended policy questions (or questions with a policy component). Another is to ask questions for which full answers would be vastly longer than the allowed space, so that every student will edit down and pick out the most important issues differently.

Of course, looking at the whole affair from a few steps back, what amazes me about law school exams is not that they're unprepresentative of what lawyers do. Most exams in any kind of school are unprepresentative. After all, exams are there to figure out how to grade students and, sometimes, to help students lock in a term's worth of knowledge.

Rather, the ridiculous thing about law school exams is that they're so high-stakes. A semester's worth of material and grading is packed into a single test, sometimes with a single question. Problem sets, papers, midterms, term projects, response papers, composition exercises . . . there's a huge arsenal of evaluation techniques out there, and yet law teachers mostly rely on a single high-stakes end-of-term assignment.

Why? The best explanation that I can think of is that law teachers masochistically grade everything personally out of a misplaced norm of professional pride. A single compressed exam is more comparatively attractive than it should be because it's more feasible to grade than a system that generates more pieces of paper. When I compare my law school classes with my college classes, I think that students would get better feedback and better evaluation if the grading were done (in the first instance) by TAs, rather than by professors stretched too thin.
3.8.2005 9:12am
Brian (mail):
I think Orin's proposed exam solution is the best compromise - it's actually the same solution that I advocated among my peers in law school.

No one likes take home exams; but the a 3 hour exam is rather arbitrary and tends to reward the best test takers, not the best students. The fairest exam I took in law school was similar to Orin's proposal: a 4 hour exam with word limits.

There are two conditions for such exams to be successful. First, the word limits have to be long enough to be fair, to avoid the problem that Jake mentioned (i.e., everyone writing only about the same basic issues). Second, and I disagree with Jake on this point, but I think if the point of a longer exam is to remove the time pressure, then the professor should design an exam that is 3.5 to 4 hours for the 5 hour time frame. The key then is to design an exam that is sufficiently challenging to students, so that the exams scores will fall in an appropriate range.
3.8.2005 9:19am
Christen Millard:
Personally, I give a take home, unlimited time, open everything (but no discussing with other people) exam with page limits. There are two reasons for this. First, I have to proctor my own exams, and that requires me to give up 3 hours on a Saturday at Christmas time. Second, I don't want to have to try to read the students' writing -- my law school doesn't let students use laptops for adjuncts' exams. However, I want to minimize the opportunity for cheating, so the students are allowed to access anything they want. Is that a perfect solution? No. But this year, I convinced my students that they didn't need to waste time looking irrelevant stuff up on the internet, and the quality of the answers improved.
3.8.2005 9:40am
The first law school exam I took was an in class three hour exam. It was the worst grade I received in my entire academic career. I simply ran out of time. It was in no way reflective of my knowledge of the subject matter and I went on to achieve much better grades for courses I put much less effort into simply because I learned how to game the mechanics of the test.
3.8.2005 10:07am
John Steele (mail) (www):
As for what kind of exam the students want, I periodically have them vote, and the winner by a substantial majority is always the 24-hour take home with word limits.

I agree that 3-hour issue spotters resemble the bar exam, but if I had to choose what format most closely resembles the real world of practice I'm familiar with, it wouldn't be an issue spotter. If an associate walked into my office and spewed out short analyses of 15 issues, most of which raised no big problem, we'd need to work on developing that associate's judgment. That kind of issue spotting is necessary at the very outset of matter, but the higher art is bearing down on the relatively few issues that matter.

So, having 8 hours or 24 hours to develop a sustained argument on an important point -- an argument that anticipates and deals with the strongest expected counter arguments -- is a staple of legal practice.

I usually give a "pick two of three essay questions" format, and include the a sustained argument on a narrow point, an issue spotter, and a policy or normative argument.
3.8.2005 10:10am
Stephen Aslett (mail):
The idea that law school exams reflect pressure that one will encounter in real-life practice is disingenuous. Granted, while there will be times when a lawyer needs to think on his feet, a lawyer will never have to analyze all angles complex multi-jurisdictional civil procedure problem in three hours. Even a lawyer who needs to write an emergency brief isn't expected to do so without the use of reference books. Most of legal work (the challenging parts anyway) are careful research, writing, editing (whether of transactional documents or briefs) and oral argument (for a few lawyers), client counseling, and anticipating and planning for contingencies. None of these skills are tested on a three hour closed book brain-dump exam.

Actually, I don't have a big problem with law school exams. I understand that law students, in terms of raw intelligence, are pretty close to each other, and there needs to be some mechanism by which faculty can separate the best from the average. A high stakes test is certainly an efficient way to do that. The problem, however, isn't with law school exams but with the pedagogy that leads up to the all-important law school exam.

I know that most law school professors earnestly want their students to do well. While a few professors do "hide the ball" and try their darndest not to give important informations, I've found in my limited experrience that most professors aren't like that. Instead, they do their best to explain the relevant rules and discuss the cases. Unfortunately, what professors do in class only minimally prepares one for exams. It's like obsessively learning the rules to a boardgame before actually sitting down to play it. While knowing the rules (i.e. going to class, learning the rules, and gleaning a few standard fact patterns from the cases) is necessary to play the game (i.e. the exam) well, one needs to play many games before one becomes good at a particular game. Yet students are expected to play the best game of their life the first time they sit down at the table. (And no, I'm not counting midterms--a short, ungraded exam with little feedback is not adequate preparation, especially when it occurs weeks before a final exam.)

Law school pedagogy is good at teaching the rules, but it's lousy at teaching the application, the most important part. The law school exam system might be defensible if professors gave many practice exams and provided detailed feedback for students. But no one does it. Exam guidance professors give, if any, is a recitation of IRAC. But that instruction, as evidenced by the lousy answers many professors get back, plainly isn't sufficient.

The standard response to this lack of in-depth feedback and lack of practice exams is that students are grown ups now and should learn to take exams well on their own. Sure, students should be responsible, but this response is disingenuous too. Students can take all the practice exams they desire, but that won't teach them efficient exam writing and issue-spotting. Most students will get marginally better, but they won't get as good as they would if they were taught by an experienced exam-taker. While I'm sure some students could develop a good golf swing by simply watching Tiger Woods swing five times, most would improve better and faster if an experienced teacher were there to show them how it's done and tell them when they're doing something wrong.

If professors honestly think that closed book time-pressured exams are good preparation for real world practice (rather than just an efficient sorting mechanism for employers and law review) and think that their job is to prepare students for that practice, then exam instruction should be far above what it is now. What's ultimately important is not that the law school can identify the 5 best students in a class (though that's fine), but that everyone in that class can do at least a DECENT job of analyzing a legal problem. Otherwise, the clients are the ones who will ultimately suffer.

Unfortunately, the sense I get is that the law school is structured (oddly enough) in a way that benefits the law school. In-depth exam teaching or legal writing instruction would require much smaller student/faculty ratios. They would require professors to take time from publishing articles and consulting in order to teach a 21 year old how to write an exam. I imagine no professor wants to take on more exam grading work or take time from profitable and prestige-inducing activities in order to counsel 1Ls. I imagine that no law school administrator would want to reduce profits by hiring more faculty or writing instructors. I imagine that no one (even law students) would want to replace the one-shot it's-over-and-done-with law school exam with a serious of extended writing assignments in each class. But if we're serious about educating students (rather than creating think-tank like enivornments for legal scholars), reforms like this must be made. Otherwise, why should students pay $30,000 a year for the comfort of professors?

As a high school teacher, I had to give (or at least try) to give in-depth instruction to over 100 students. If that meant staying after school or giving up part of my lunch break to explain something, I did it. I gave plenty of exercises before the final exam to let students know what they were getting into. I had to give up many a saturday afternoon and night grading papers and exams and writing lectures. All for $20,300 a year. I find it disconcerting that professors, who after all are in the business of EDUCATION, are so willing to keep the status-quo, which doesn't adequately prepare students to take their own exams, through in-depth instruction and repeated practice and feedback, because it's easier for them that way. (After all, who wants to be the first to rock the boat?) I can't imagine that it would be much harder than what high school teachers--who must teach less brilliant students and deal with severe disciplinary problems to boot--do for far less than six-figures, forty hours a week instead of three to six hours a week.

I know professors mean well, but I feel they mean well in the same way we mean well for the tsunami victims. We'll donate money, maybe give up a little but of our time, but almost none of us will go over there and effect real change. It's much easier to throw out a token exam or give a token exam preparation lecture and mean well than it is do to something that will really prepare students.

So, what say you in your defense, professors of the Volokh Conspiracy?
3.8.2005 10:24am
Honest Abe (mail):
And then there's the whole issue of grading four months of work based on a single test...

What professors ought to do is give weekly writing assignments on the material, with feedback. Every Monday the students get feedback on last week's lessons, and an outline of what they were supposed to have learned, every Friday an assignment to be turned in Saturday morning so the prof can grade it over the week-end and let them know if they are learning on track or not.

The real question is whether or not profs are supposed to be teaching. There is a good body of pedological science on what works and what does not work in teaching and learning. Profs know or should know that information -- after all, it is their profession.

If they ignore or fail to implement, they are guilty of gross negligenc and deserve all the standard punishments.

With proper intermediate testing, a final would not even be necessary.
3.8.2005 10:31am
Larry rothenberg (mail):
David Kopel is right. Contrary to what the NW professor thinks, real life laywers often have "no opportunity for reflection, research, reconsideration or redrafting. You simply dash off your answer and hope you got it right." in advising a client on a trade sanctions issue, I once had to answer in the space of hours, whether the client could do something involving a transaction with Cuba. It involved reviewing the relevant regs, drafting a memo, and briefing a partner who then briefed the client. It was just like a law school exam. It may be different for litigators, esp. for the type of appellate ligitation that law school professors (who tend not to have been real-life laywers) engage in, but lawyers who advise clients on transactions and deals are under intense time pressure.
3.8.2005 10:33am
I completely agree with James Grimmelmann when he questions why a single exam determines the work of an entire semester. I still find it incomprehensible that law schools go to such trouble to select students of such a variety of skills, interests, and degrees, and then forces them to succeed through only one narrow path. I graduated from GW a few years ago, and my first week of school my contracts professor (the fabulous B.K.) collected resumes from every student in the class, read them, and then took 15 minutes out of class to tell us about ourselves. He seemed delighted with the enormous diversity of majors, former careers...he pointed out engineers, social scientists, photographers, journalists, Hill staffers, paralegals, English majors, Philosophy majors, Biochemists, Political Scientists. My own extremely dull resume not withstanding, I was enormously impressed by the range of talents amassed by the school's admitting committee, a process that must have taken great patience, thought, and deliberation. But why the effort in attracting and admitting students of different interests and skill sets, if they are only allowed to perform a particular way to earn the high grades? The fact is that some people are likely to perform better in oral presentations, some people are particularly skilled in working on team projects, some people fall apart on time-pressured exams but write beautifully in research papers. Why not determine a student's grade by several projects, rather than one?

Please understand, I'm not suggesting we need to ensure everyone gets better grades to feel good about themselves. But the practice of law is broad enough to recognize and reward a variety of skills, whether in the courtroom or writing regulations or law review articles or counseling clients. Why can't law school allow the same?
3.8.2005 10:58am
Mr. X (www):
It's not the written exam that's the problem, it's the 'tradition' of high-stakes testing. End-of-semester exams do nothing to identify whether students are having trouble with the material while there's still time to correct it.

In the interest of charity, I'll assume that it's the weight of tradition that keeps this outdated practice going and not laziness on the part of law school professors. Regardless, the practice should be changed.

Yours truly,
Mr. X

3.8.2005 11:50am
Stephen Aslett writes:

"The law school exam system might be defensible if professors gave many practice exams and provided detailed feedback for students. But no one does it. Exam guidance professors give, if any, is a recitation of IRAC. But that instruction, as evidenced by the lousy answers many professors get back, plainly isn't sufficient."

I can't speak for other law professors, but I take several steps to try to help my 1L students gear up for my exam. For example, I give a "fake" midterm -- a practice midterm that is ungraded -- and then discuss it with students as a group. I also give students a fairly long memo I have written on how to take (and how not to take) law school exams. Perhaps most importantly, I give students an answer guide to each of my prior exams, usually about 15 pages single-spaced per exam, consisting of comprehensive discussions of every possible right or wrong answer as well as common strengths and weaknesses of different approaches. Finally, I also explain to my students how I grade and tell them the format of the exam so there are no suprises.

Of course, this may not make the current law school exam system defensible; indeed, the purpose of my post is to consider if we can find a better way.
3.8.2005 11:51am
adam (mail) (www):
Eight hours and page limits seem reasonable; I'm just glad that we've moved away from handwritten exams, which penalized people like me whose penmanship was always less-than-adequate.

What's odd about law school exams -- and perhaps unfair -- is the extent to which classroom pedagogy focuses on examining all sides of an issue, while the exams require (a) correct answers and (b) some level of advocacy.

I liked my Evidence exam the best: here's the trial. Here's a piece of evidence they'd like to get in. What rule(s) govern(s), and what's the proper result? [Repeat 25 times.]
3.8.2005 12:37pm
Ralph Adam Fine (mail) (www):
Professor Lubet's suggestion to eliminate time-testing examinations because, as he wrote, "[t]here is almost nothing about the typical law school examination that is really designed to test the skills involved in law practice," is not only out-of-touch with what the practice of law requires—as so ably said by Dave Koepl—but, in my view, is but another step towards the mediocrity that is enveloping our profession. I guess Professor Lubet has never had a trial judge tell him mid-trial to "let me have a brief on that tomorrow morning," or faced an immutable deadline on an initial-public-offering filling. C'mon guys, law school is not only supposed to give needed skills to represent folks who place their lives and fortunes in lawyer's hands, but law schools should also weed out those who cannot make the grade. And, yes, that means thinking on one's feet and under pressure. Obsta Principiis!

Ralph Adam Fine
Judge, Wisconsin Court of Appeals
3.8.2005 2:23pm
SupremacyClaus (mail):
Judge: I agree. I suggest this reform. Instead of having 50 facts on an exam, let have 100. Instead of 25 being legally germane and subject to IRAC, let's have 50. Instead of an 1 hour time limit, let's have a half hour. Instead of English, let's use the 20,000 character alphabet of Mandarin Chinese.

This would weed out the slow, the ignorant, the inefficient. As a client, I certainly seek someone able to IRAC the elements of my case in 1 minute each, from memory.

The Mandarin Chinese proposal is forward looking to the time they buy out this country with a credit card, thanks to devastation done by the highly efficient, speedy, highly knowledgeable lawyer.
3.8.2005 2:49pm
Prof. Kerr,

Is there any chance you could post (or otherwise make available) your advice for taking law school exams online?

3.8.2005 2:52pm
I took only one closed book exam in law school, and many of my open book exams were 24-72 hour take-home exams. I thought they allowed for reflection and in-depth analysis. That being said, I felt obligated to write concise, well-reasoned answers because of the time I had to reflect on the questions. The profs always took a long time to grade them, but I doubt that would have been different if the exams were limited in some way.
3.8.2005 2:58pm
Neal (mail):
It's fallacious to assume that because time pressures in the profession require most lawyers to "dash off [an] answer and hope [they] got it right," the best way to test law students is to force them to do just that on exams. The current approach may indeed be the best way to test (and teach) law students, but that's an empirical question that should be informed by educational research and theory, not just folk wisdom and force of habit.

One key difference between lawyers and law students is that the former have already refined their analytical skills through law school, bar preparation, and private practice. When they "dash off [an] answer and hope [they] got it right," they therefore have some cause for optimism. Law students, on the other hand (and particularly first-year law students) may have little confidence in their ability to argue from precedent, reason analogically, etc. To encourage such students to rush their answers may actually discourage the development of more important, foundational legal skills.

It could be that law schools, and the legal profession as a whole, would benefit from giving students more time to formulate precise answers early in their legal educations, and to increase time pressures gradually over the course of the law school experience. I find this approach intuitively appealing, because I think law schools should concentrate first and foremost on teaching students how to engage in rigorous logical analysis.

Speed is important, but it is a secondary skill that comes inevitably with practice. When I took swim lessons as a child, my instructor didn't urge me to sprint as quickly as I could on the first day of practice. The focus at first was on the quality of my stroke; on applying new knowledge with precision. Once the basic skill is mastered, it makes sense to increase expectations and demand faster performances. But to put a premium on speed for a novice swimmer or law student invites mad thrashing in the pool or sprawling, incomprehensible exam answers.

Of course, all this begs the question of what we mean by the "best" way to test students. Are we concerned solely with what testing method best assesses a student's current abilities or future chance of success, or are we concerned with choosing the testing methods that best promote student learning? To what extent should law school testing serve other constituencies, such as prospective employers, law professors, and law review? I don't have answers to these questions, but I think they're worth asking.
3.8.2005 4:08pm
Amy S.:
Could it be that law school professors are trying to convince themselves that this is the best method of testing because it worked for them? We know that all of the profs. at top schools performed exceptionally well in law school; maybe they believe that those who do well law school exams are better lawyers because they perceive themselves to be the better lawyers.

Many professions require quick thinking under time pressure, but do not use the single-massive-issue-spotter approach to testing. Empirical evidence shows that more frequent testing more accurately corresponds to knowledge-levels. More importantly, given what most attorneys do for a living, one would think that law school testing would stress organized writing, at least beyond that joke of a class called legal writing. For a final, why not give students one to two days in which to write a brief a la moot court, where students are forced to identify and evaluate certain core issues from a made-up record/transcript? Students would be forced to pick out relevant raw information, digest it, and synthesize it into a relevant argument. Although creating this exam may take more effort on the part of the tester, with a page limit, it would not be a greater burden for professors to grade than an issue-spotter, and would actually probably be more enjoyable to read. If that's a better way to learn, the "deter cheating" rationale seems to me just a lame attempt to bury the problem.
3.8.2005 9:23pm
Robert Schwartz (mail):
2 thoughts.

1. Law School bears an only incidental relationship to the real world of lawyers, which is fine, the only way to make it better would be to make it shorter 2 years instead of 3. There has to be a better defence of its pedagogy than realism. If you were being realistic, you would find complete morons to grade the exams (in the real world we call them clients).

2. 3 hours is a long time for an exam. Long enough to make the X-games. Making it longer will not make you smarter. It will just allow you to screw yourself up in more intricate ways. I hated take homes even more than the regular exams. At least with the 3 hour exam, you got to go home eat and sleep after the 3 hours was over.
3.10.2005 1:22am
SupremacyClaus (mail):
Bob: Do what the moron tells you or it's no money. Now, fetch.

Exams trained you well.
3.10.2005 10:14pm