I’ve blogged before about how law professors grade exams, and I wanted to blog some more about some of the difficult judgment calls professors have to make when grading. Exams that get everything right or do everything wrong are easy to grade. But they are also very rare. Most exams do a mix of things well and a mix of things poorly. Coming up with a grade is a difficult and nuanced job that requires applying a meaningful, consistent, and accurate judgment of how that particular mixture should measure up against all the other mixtures.
To see the problem, imagine a law school issue spotter. The professor sees two major issues in the question, issue one and issue two. The first issue requires working through complicated doctrine but has a reasonably clear answer, at least once you work through all the detailed parts. The second issue requires applying a vague standard to ambiguous facts. It’s not hard to see that the standard applies, but applying it well requires a mastery of the facts and the details of the legal standard; the outcome can be argued for either side, as it’s intentionally right down the middle.
Now imagine you have to grade the following 10 exams:
1) Exam 1 gets lost in the doctrine working its way through issue 1, but it ends up getting the correct answer by chance after making two mistakes that canceled each other out. The exam does a decent job applying the standard for issue 2.2) Exam 2 provides very little analysis of issue 1 and simply announces the result (albeit correctly). The exam then does a very good job of applying the standard for issue 2. The exam then spends time discussing a third issue that is not actually raised by the facts; the discussion reveals that the student does not understand the law relating to that third issue.
3) On Exam 3, the student has a misunderstanding of one of the basic doctrine issues that leads the student’s analysis of both issues to start off on the wrong foot. However, once you get past that one error, the rest of the exam is excellent: The student is getting absolutely everything wrong, but it all seems to go back to one error.
4) Exam 4 goes through the basic issues reasonably well, but refuses to express any view of the strength of any argument or to reach any conclusion. Every argument, no matter how weak or how strong, is described as a position that “may apply.”
5) Exam 5 does the best job of anyone in the class on Issue 1. However, the exam completely misses Issue 2.
6) Exam 6 starts by discussing two issues not actually raised by the question; the answers reveal that the student has a basic misunderstanding of a topic not tested by the question. However, the student then addresses the two issues raised by the question and does a very good job with both of them.
7) Exam 7 starts with an outstanding answer to issue 1. It then starts to answer issue 2, but the student runs out of time after just a few sentences.
8) Exam 8 has misread the facts. However, it does a pretty good job with both issues 1 and 2 if you assume the facts that the student incorrectly thinks exist.
9) Exam 9 is hard to follow, as the student is a poor writer. It’s kind of hard to tell what the student was thinking with the first problem, as it’s hard to follow what the student was saying, but the student does end up with the right result. The analysis of issue 2 is easier to follow, and the student does a so-so job with it.
10) Exam 10 starts off terribly with an analysis that suggests that the student really has no idea what he is talking about. However, at the end of the answer, the student has a section called “alternate analysis” in which the student writes that he “may have misunderstood the problem” and that there is a different approach that is “arguably applicable.” The student then has a short answer to both issues that is exactly correct.
What makes grading so difficult and anxiety-inducing for professors — or at least for me – is that accurate grading requires a consistent and coherent method, rooted in a concept of the purpose of grading, for how high or low to mark each of these possible combinations of errors and correct answers.
Some professors grade with a checklist that gives “points” for certain things, but I have never found that works for me. My ultimate goal in grading exams is to gauge a student’s lawyerly ability. I try to imagine the exam author is another lawyer who I have the option of working with, and I gauge how enthusiastic I would be about working with that lawyer — and how much I would trust and rely on the lawyer’s judgement and analysis — based on the quality of the answer. I don’t think a checklist can really capture that. A checklist creates a kind of objectivity in grading, but the objective thing measured doesn’t seem to relate to anything a future employer would actually want to know. Instead, I make a judgment call, an overall assessment that factors in my judgments as to what each of these errors suggests about a student’s lawyerly ability. It requires a very careful read of the answer, and often a re-read, to get an accurate sense of where in the scale of answers that answer best fits.
Anyway, I mention these things not because I expect readers to have great sympathy for law professors. We have the single greatest job in the world, so no need to feel sorry for us. Rather, I thought readers might just be interested in the kinds of difficult judgment calls that go into grading law school exams.
Finally, I suspect some readers will say that most of the ten exams here made major errors and that most deserve bad grades. If you think that, you have an inflated sense of how well you took exams back when you were in law school. Your complaint is also addressed by the law school curve: In a curved class, it just doesn’t matter if all the answers are great or all the answers are bad. All that matters is the distribution in quality among the various exams that is then superimposed onto the pre-set curve to yield the actual grades.
NickM says:
After that post, you now owe me a beer.
Nick
January 18, 2010, 1:11 amJohnF says:
I don’t think looking for the most lawyerly responses precludes allocating points for various items. What is it to be “lawyerly”? Some things might be (1) understanding the facts of a situation, (2) understanding the question you have to answer, (3) recognizing what doctrines should be looked at to reach an answer, (5) finding (or knowing) what those doctrines say, (6) applying the doctrine to reach an answer. Obviously, the list is not definitive; items can be added or subdivided further. Moreover, this is not a list of things that go into any particular answer; these are items that go to what makes a good answer in general, breaking down your principle that a good answer is one that a good lawyer would give.
Once you have your list it becomes easier to see which items the student got and which were missed. The interesting question is an exam like number 6, which screwed up a question that was not asked. Even though he got right the correct questions, his assumption that certain things played into an answer when they didn’t is not good lawyering. He should be dinged on item 2 to some extent.
Now, I’m not saying a list should be applied mechanically. But it is important for a professor to figure out what counts as good answering and what doesn’t. Having lists is a good way to organize thinking on grades. And there would be no reason not to let the students know what’s on your lists as well.
January 18, 2010, 1:27 amOrin Kerr says:
JohnF,
I agree it’s straightforward to create a list of the factors that go into grading: The list would include about 20 or 30 pretty uncontroversial qualities. The hard part is identifying their proportion and measuring each factor in a way that produces an answer that has something to do with what you want to measure — especially complicated given that each factor can be dependent on the other factors. All in all, I think it’s turtles all the way down: My experience is that you’re just creating the illusion of objectivity by adding the step of listing the factors.
In terms of telling students how I grade, I have a couple of techniques, at least in my 1L classes. First, I give a lecture to my 1L class near the end of the semester explaining how I grade. Second, I hand out essays addressing each previous 1L exam I have given that contains a thorough discussion of every right answer and wrong answer given. It is about 15 pages single-spaced for each previous exam, and there are now seven of them, totaling a little over 100 pages. (I write each answer after finishing the grading so all the answers I have just read are pretty fresh in my mind.) I then urge students to read and re-read the old exam essays; They are pretty much a blue print of how I looked and evaluated every answer on every exam.
January 18, 2010, 1:44 amDSM says:
Some of these problems are applicable even to grading in mathematics or physics. For my sins I’ve had to grade a fair number of exams, and always have a difficult time dealing with cases (3) and (8) above, where there’s a basic misunderstanding which means that everything that comes after is false and so the argument is unsound but the deductions are valid.
I’ve actually been on both sides of this.. in a philosophy of science class I once submitted a paper that the professor liked well enough, but discussed a different chapter of the text than he’d assigned. (He gave me a decent grade, deducted marks for the error, and gave me the opportunity to rewrite if I liked, which I rather lazily passed on.)
January 18, 2010, 2:13 amRich Rostrom says:
ISTM that an exam designed this way will almost inevitably produce problems of this kind.
It might be better to break the exam into pieces where each of the qualities to be tested is examined separately. For instance, one might have a section where the facts are laid out for the testee, and only the subsequent reasoning is tested. This would avoid the problem with Test 8.
OTOH, it may not be possible to state the facts more clearly than they were given in the test. In which case, Tester 8 fails. Understanding the facts as explained (or the spec for the program, or the diagnosis, or the food order) is an absolute requirement. It doesn’t matter how brilliant the subsequent work is if the basis is wrong.
I would also fail 3 (fatal error of understanding) and 4 (“maybe” is not a responsive answer).
1 should either fail (for the internal fatal error) or get minimal partial credit for getting the right answer by luck.
5 should fail if dealing with issue 2 is a requirement.
Of the others (and IANAL or a teacher)
2 – pass with serious ding for not working through issue 1.
6 – pass with small ding for the unnecessary work.
7 – fail, as 5, or minimum pass for at least starting on issue 2.
9 – Pass with poor grade. Writing skills are fundamental.
10 – Pass with minimum grade.
January 18, 2010, 2:15 amjosh bornstein says:
Orin,
I thought you made a very interesting post, and it’s not something I’d given a lot of thought to. (If I had during law school, I think I definitely would have spoken to my professors about their grading method(s).) My recollection is that most of my profs used the checklist formula, although I know for sure that one of them moved your grade up or down a bit based strictly on his subjective view of your writing ability. (He and I became friends after I finished his class, and he often confided [lamented??] about the poor writing skills of many law students.)
One advantage of the checklist method is that it is consistent. Presumably, if you had the ability to ‘forget’ a student’s answer, and you were thus able to grade her exam today, and then again 2 weeks later, she would end up with pretty much the same grade. Also, when a student comes to your office to review an exam, it would be fairly easy to run down the exam. (“You missed this issue. Saw that issue but misapplied the law. Great job on this issue and the 4 sub-issues. Etc.”) I think your approach is better in the sense of making grades correspond to some sort of lawyerly ability. And since it’s clear that your method requires much more time and effort in the grading process, I think you deserve props for being willing to take the time to get at a ‘more fair’ grade.
On the other hand; since you try to use a more holistic (totally my word; not yours) approach to grading, I wonder how easy it is to remain consistent. I wonder if, for example, you graded my paper directly after grading 4 really dreadful exams, you might not subconsciously bump my grade unfairly higher. Or the converse, if you would unfairly lower my grade, if you read it right after reading a final exam written by EV. Can you share with us your experiences in this area? When students come in to review their exams (or challenge their grades??), do you ever think to yourself, “Wow, I really gave an original grade much lower (or higher) than what I see now.”? Or, “I have *no idea* why I liked/disliked this answer so much originally.”
January 18, 2010, 2:15 amOrin Kerr says:
Josh,
Yes, the difficulty with my approach is definitely the need to find and then adhere to a standard. It takes a long time, and it requires grading and regrading, usually with breaks in between each answer to try to minimize anchoring effects.
As for maintaining the same standard, this is the issue I really sweat over. I have a few techniques that I think help out. My key problem is that I know that I have a slight tendency to get a little tougher on students over time: It usually kicks in over time, when I have started to see the same mistake over and over again, and I start to perceive the error more readily than I did at the beginning. I adjust for this in three major ways. First, I regrade the first set of exams I graded when I’m done until my new grades converge with the old (usually five or six exams in). Second, once I settle on a standard, I jot down some notes on roughly what kinds of answers have what kinds of scores so I remember and stick to the standards over time. Third, I reverse the order I am grading the exams after each subsequent question to try to cancel out the bias (or at least minimize it). I’ll also go back sometimes and pick a few exams out of the stack to regrade to see if my new grade matches up closely with the old. If it does, I’ll be pretty confident that my standards are constant. In the relatively rare case it doesn’t, I then dig in and start to regrade a lot of exams to make sure I’m being accurate the second time.
The upside to being such a freakazoid about grading is that every time astudent has come to me wondering about his or her grade, I have re-read the exam and felt my grade was correct. (At least so far — knock on wood.) It’s a good feeling, and one worth all that work, I think. Also, I always heavily notate the 1L exams as I grade them, so there isn’t any mystery as to why I gave the grade I did.
Oh, and FWIW, in my first year of teaching, I initially tried grading with a checklist. The problem was that the checklist scores had only a slightly connection to how good the exams were: The scores were easily replicable but largely meaningless. That’s awesome for professors, as they don’t have to think too hard when grading and have an easy answer if students think their grades are off. But it struck me that the system was easy for professors at the cost of making grades fairly arbitrary. That’s why I gave up the checklist idea and went with the method I use now.
January 18, 2010, 2:34 amSitnah7 says:
In my law school experience, I consider it a somewhat lucky break to even have old exams to look at. A model answer or two feels like the scene in “The Paper Chase” where the main character breaks into the library to steal Kingsfield’s old contracts notes. A 100 page compendium of past exam answers, right and wrong, is wholly beyond my ken.
January 18, 2010, 2:51 amD.R.M. says:
Hmm. Since there’s a curve, all I really have to do is rank the papers relative to each other, don’t I? They’ll then fall out on their own. So:
#8: On the one hand, getting the facts wrong means everything he did with them was wrong. However, he’s proven he knows the law on this topic and can apply it to a set of facts. He possibly just screwed up on reading comprehension in the high-pressure context of an exam. Notes will include a reminder to pay more attention to the problem he is presented, to understand it fully. (A repeat of the same mistake on a later exam will be judged like #3, since it would seem to indicate a deeper inability to understand the facts instead of a simple mistake.)
#1: Overall correct, and the mistakes in the reasoning on issue 1 were clearly identifiable.
#9: Correct, but I can’t actually tell how much of that was luck and how much was reasoning, and even if it was all reasoning, communicating it is important.
#2: Failure to discuss issue 1 in detail is just another form of writing incomprehensibly on the issue. Misapplication of a third issue to the facts is an additional serious defect.
#6: What I was testing him on, he got right. But adding two irrelevant issues is a serious defect.
(These first five are actually all relatively clustered near each other; specific details of execution might jostle them around a bit.)
#4: Knowledge demonstrated; application failed. I’m assuming this is the first time I’m grading him, so I’m being lenient. A repeat will drop below #5.
#7: I know he saw the second issue. I don’t know if he understood it. He did get the first one right. Even if I assume he’d have gotten the second one really wrong, he’s still pulled ahead of the people below him.
#3: Perfect reasoning from a fundamental misunderstanding of the law is still fundamentally flawed.
#5: Missing half the problem is about the same as #3, but he doesn’t even have the advantage of showing me as much correct reasoning.
#10: He’d be much higher if he had fully committed to the “alternative analysis”, saying for certain his first answer was wrong. In that case, I’d be looking at, in effect, another #9 (lack of clarity, due to brevity instead of incomprehensibility). But this way, I’ve got all the vacillation of #4 and a worse version of the mistakes of #3 on top of the uncertainty as to whether he finally got the right answer by luck or understanding.
January 18, 2010, 5:41 amIlec says:
Professor,
Thank you for an enlightening post. The first thing that occurred to me in reading your post and your approach to the grading of papers, was the current mania in many fields for “objective” standards – checklists and the like. I think that you have cogently argued that not everything can be reduced to an “objective” checklist. We must sometimes rely on subjective judgement in grading papers or making decisions. In those cases we need to focus on putting the right people in the jobs so that we can have confidence in their subjective judgement.
January 18, 2010, 7:23 amTweets that mention The Volokh Conspiracy » Blog Archive » Judgment Calls in Grading Law School Exams: One Professor’s Perspective -- Topsy.com says:
[...] This post was mentioned on Twitter by Peter Black, Eugene Volokh. Eugene Volokh said: Judgment Calls in Grading Law School Exams: One Professor’s Perspective: I’ve blogged before about how law p… http://tinyurl.com/ybh4tjt [...]
January 18, 2010, 8:13 amFantasiaWHT says:
Give them all B’s.
Hey, the curve (at least at my school) didn’t require that anybody get A’s or AB’s.
January 18, 2010, 8:21 amTRE says:
Personally handing out 100 pages of commentary and old exams is a very fair approach.
I think you are probably ahead of the curve in grading OK. :)
January 18, 2010, 8:35 amJoe says:
These categories helps to understand various judicial opinions.
January 18, 2010, 9:00 amlgm says:
Ten examples!? Might Professor Kerr hate grading so much that he would rather type a long essay on grading for his blog that actually look at papers? Not rational behavior, but human.
Exams have a negative effect on students. Exam writing forms a good part of students’ pre-professional writing. But exam writing differs from normal writing, in that the writer believes that the reader already knows the answer. The goal of the writer is not to explain the answer, but to convince the reader that he/she knows it. This allows, for example, putting steps in an argument out of order.
January 18, 2010, 9:04 ambrubaker says:
New College of FL, the honors college of the state university system of FL, uses narrative evaluations rather than grades. The education is first rate. I would think that narrative evaluations would give employers much more information than grades provide. Wouldn’t this solve your problem?
January 18, 2010, 9:28 amDavid M. Nieporent says:
Just throw them down the stairs.
January 18, 2010, 9:35 amBen P says:
As I read that, in a class of 10 students you just gave either 5 F’s and 3 D’s or 4 F’s and 4 D’s, 1 C and 1 B.
I’d hate to be in your class.
January 18, 2010, 9:39 amDavid Newton says:
Something that needs to be thought about here is bias in grading. Why are papers marked in such a way that the authors can be identified? In all the university exams that I ever took and all the professional exams that I have ever taken I was not identified by anything other than a candidate number. This was precisely to avoid the possibility of marking someone more harshly or leniently based on anything other than their performance in that particular exam.
I would also suggest that the correct method for grading things is to have a subdivision of marks into correctness marks and method marks. If someone uses the correct facts and applies them wrongly they get good marks in the first and poor in the second. If someone uses the wrong facts and applies them correctly then they get poor marks in the first and good in the second. If someone uses the correct facts and applies them correctly they get good marks in both sections. The comprehension and method marks should be separate from one another since both are vital skills for a lawyer and both need to be tested. On the marking schemes I have seen when considering method marks points are to be given for particular points or anything of a similar level. For example when considering second amendment law as an individual right, there would definitely be a point for citing Heller and correctly explaining its ratio. If Heller was cited and explained correctly but then applied to facts which differ from those set down in the exam question method marks would follow but comprehension marks would not.
The separation between method marks and comprehension marks allows a less subjective evaluation of the answer to a question but I would argue still allows the evaluation of lawyerly skills as both comprehension and good method are required in the field.
January 18, 2010, 9:41 amlucy says:
I think this post demonstrates the case for multiple choice and other types of objective questions.
January 18, 2010, 9:50 amsardonic_sob says:
To me JohnF and Orin Kerr’s remarks are the heart of the matter regarding all the disparate types of answers. The professor has to communicate clearly to the students how they want the students to answer. If I knew how the professor had advised them they wanted the answers to be, I could make a reasonable analysis of who should get what grade. With no more information than is present in the original post, I can’t do that. I can tell you what my subjective opinion is on the relative quality of the answers, but since FOLLOWING INSTRUCTIONS is a key part of being a lawyer and I don’t know what the instructions WERE, my subjective opinion is not worth very much. :)
I’m a patent lawyer, among other things, and I am here to tell you that a mediocre patent application properly written and formatted is approximately nine hundred thousand times better than a brilliant application improperly written and formatted. (That’s just a rough estimate.) Courts vary widely in their latitude – some make the USPTO look like kindergarten teachers, and some are basically along the lines of, “Well, you gave me a piece of paper before the statute ran, even though it was incomprehensible slop. We’ll see what we can work out.” But, darn the luck, there’s no way to know what sort you’re going to get in advance so you had better be able to FOLLOW INSTRUCTIONS.
All that being said, while I love teaching, I could never be a law professor, for exactly this reason. I’d run out of hands. You have my respect and empathy.
January 18, 2010, 10:46 amTheNino85 says:
I’ve heard this argument a lot, and I’m in physics, not law. I imagine that people use it in every post-baccalaureate educational system. It does have some merit. After all, it’s correct; exams do have a negative impact on students. Even students that understand the material amazingly well can start to panic on an exam, and once you start to panic on an exam, you can kiss your A grade goodbye. Exams are also not like the real world in *any* discipline, even for those who plan on staying in the ivy towers for the rest of their lives.
The problem, time and time again, with this argument is: what else are you going to do to measure your student’s understanding? Oral exams have the same problems, and in many cases seem to amplify the flaws of examinations. You still have all the problems inherent in exams, but now you have issues of the fear of screwing up in front of an audience added to the mix. People will cheat on homeworks that are given, and the more you make homework a part of the grade, the more rampant cheating will become. Making students give presentations is an excellent teaching tool, and it correlates nicely with the real world, but with presentations, the students can pick and choose what they want to talk about. Deliberately ignore things that they don’t understand. And if you start quizzing them on those things, well, it’s just an oral exam with a presentation component. Furthermore, with presentations, students often don’t know what the important things are to present unless you tell them that they’re important. They’re students. If they knew what the important things were, they’d be teaching the class, not taking it. So at best, they’ll be able to give presentations on what they *think* is important.
So, yes, exams are terrible. You think taking exams is bad, try writing and grading them. The students have the easy part, honestly. But they’re pretty much the only reliable way to measure student understanding.
January 18, 2010, 10:50 amOrin Kerr says:
Hmm. Since there’s a curve, all I really have to do is rank the papers relative to each other, don’t I?
No, unfortunately. There are four exam questions for each exam, each of varying weights toward the final exam, and these 10 exams are just a small subset of the 80 exams you need to grade. So ranking 10 answers to one question doesn’t really help.
January 18, 2010, 10:57 amuh_clem says:
I would hate to be represented by a lawyer who skated through law school on multiple-choice questions. Yes, grading essay questions is much harder and more time consuming, but requiring students to do something that at least resembles what they’ll be faced with after graduation produces much better graduates. And not just for lawyers.
One observation about the exams: you are teaching law, not deductive reasoning. The students who get the law wrong but then reason flawlessly from an incorrect premise (eg student #3) should fail or receive a low grade. Giving them a break for sound logic is akin to giving them a break for correct spelling or good penmanship – it’s not what you’re testing for, logic is expected.
And student #4 who “refuses to express any view of the strength of any argument or to reach any conclusion” has a bright future ahead of him in middle management.
January 18, 2010, 10:58 amOrin Kerr says:
David Newton writes:
David, law school exams are anonymous: All I have is a randomly assigned ID number.
January 18, 2010, 11:00 amsardonic_sob says:
I once answered one of the questions on a physics exam (basic mechanics, first year stuff) in such a way that I got the exact right answer. I mean, I was right to the sigfigs given in the question. But I saved it for last and I didn’t really set it up right and frankly, it was some kind of bizarre coincidence that I got the right answer, since my method hadn’t got anything to do with how you should actually go about solving the problem.
I got, IIRC, 3 out of 10 of the points for that problem. I felt quite ill-used and pointed out that I’d got the right answer and with very little analysis, too! The response: It was an accident you got the right answer, I gave you three points for at least trying and for getting an answer. (Several students hadn’t even finished the test.) You didn’t know how to do the problem. Shut up and be happy you got anything.
As I said, I felt ill-used. But upon reflection, that’s *exactly the way I’d have graded the test.* You wouldn’t think that that kind of thing could happen in fields where there is a “right answer,” but it does.
January 18, 2010, 11:14 amMark says:
No, unfortunately. There are four exam questions for each exam, each of varying weights toward the final exam, and these 10 exams are just a small subset of the 80 exams you need to grade. So ranking 10 answers to one question doesn’t really help.
Kicked in the ass by Arrow’s Theorem again!
January 18, 2010, 11:16 amWhadonna More says:
I note that there’s no mention of evaluating the particular elements that were taught in class/assigned work, but a stated goal of “grading exams [] to gauge a student’s lawyerly ability”. As a result, the grades received will be more about the general skill of taking law exams than about the learning of the course subject matter.
Not surprising, but one of many market failures in the law education system. OK gets paid to teach subject X (something other than “Lawyerly Writing 101″, I presume) and then gives grades that aren’t directly related to learning subject X.
If Exam 2′s “extra” issue was a major one in the class materials (and not WAY off point), while the issue 1 wasn’t a big focus, that student SHOULD get a decent mark. Exam 3 should get a big deduction and a worse grade than Exam 8, but probably wont.
January 18, 2010, 11:26 amOrin Kerr says:
Whadonna More,
You misunderstand. The number 1 lawyerly ability is knowing and understanding the law. A student who understands the law and applies it correctly in a way that shows the student’s understanding gets an “A,” and there isn’t really any debate about that.
The problem here is different: How to scale the many different kinds of errors, mistakes, and midjudgments that students make on an exam, given the almost infinite combination of errors that can be made.
January 18, 2010, 11:46 amJoseph Slater says:
Excellent post. I’ve seen all those “types” of answers over the years. Obviously, the solution for at least most of them is to give some but not full credit; still, how to rank the different kinds of mistakes is difficult. And so is making sure you are internally consistent even if you have a ranking.
IMHO, the biggest offender is #3. If you have the basic law wrong, I can’t see giving you much credit for explaining correctly what would logically follow from that fundamental misunderstanding. As to #5, assuming the prof. thought the two issues were equally important/difficult, wouldn’t the answer be to give around half the available points for the question to the student who did a great job on one issue but missed the other one? [or do you not use points at all, Orin?]. The one I struggle with the most, I think, is the one that gives the “right answer” with little or no analysis.
A number of the others raise the question, “what if the student says the right thing but also says some incorrect things? This is tough, because law students like to hedge, and especially first years like to throw everything up against the wall to see what sticks. I think you have to deduct some points for saying things that are clearly wrong, but again, trying to be consistent is tough.
As to consistency, I’m a strong believer in grading all the answers to one problem before moving on to all the answers to the next problem (as opposed to grading exam-by-exam). This, I think, is the best way to try to ensure consistency. Although, opposite of Orin, I think I’m sometimes harder on the first handful of problems I read, because my expectations are higher. So I go back and double-check those.
Finally, I think it’s a credit to Orin that he takes this issue so seriously. While law teaching is a great job, grading is tedious, sometimes frustrating (hey, I never said anything like that in class!), and nobody ever pats you on the head and says, “good job grading!” Scholarship, service, and classroom teaching can yield positive feedback from others, but I’ve never had anyone tell me that I did fine job grading. But grading is very, very important to students and to the pedagogical process. That’s not to say grades are the be-all and end-all of anything, but law profs do need to take it very seriously.
January 18, 2010, 11:55 amPJens says:
I do not understand why people are so hard on exam 7. As a consumer, I want to hire a lawyer who has outstanding answers, even if it takes more time and money to get them. One outstanding answer to one question is far better in my mind than getting the premise incorrect, or misstating the facts on two. Yes, law does have time limits, but speed comes with experience, something learned at 1L level. If I think my lawyer is doing an outstanding job on the work, but not completing it fast enough, I hire another one to help them! Am I missing something?
January 18, 2010, 12:04 pmBryan Gividen says:
PJens,
Though I don’t know if I agree with this aspect of law school exams, they are designed to separate the students in their potential as attorneys (at least in part). If you gave every student infinite time (or the benefit of a doubt in Example 7) then there would be little differentiation in grades. A time limit accomplishes two things. 1) It helps potential employers see how the student works with heavy pressure. 2) It helps determine how quickly a student can concisely grapple with legal questions.
Another problem is that some students would strategically over-answer whichever question the student knew better hoping that the professor would give them similar credit for a question they knew less about. That’s problematic in and of itself.
January 18, 2010, 12:13 pmMiguel Dickson says:
Out of curiosity, Prof. Kerr, do you or other professors you know at all consciously or unconsciously compensate for the likelihood that certain errors are artifacts of the exam process? For example, misunderstanding the law/doctrine which students have presumably studied over the semester seems like a basic failure of legal understanding, which should be fully punishable in grading, whereas getting a particular fact (even if a crucial one) backwards at least quite often seems an artifact of artificial time-pressures, where in real life, given anything like due diligence, a lawyer could not reasonably make such mistakes. [Not to mention they'd be dealing with these facts for weeks, not 3 or so hours.]
This seems to be particularly the case since exams, almost by their nature, either tend to err on the side of too few facts (where the author knows exactly what facts were ‘implied’ by the scenario, because they had a more-or-less distinct mental picture of the scenario, but a student feeling rushed fails to make the ostensible connection), or too many facts, where a student accidentally skims over a small fact that would crucially change the analysis..
January 18, 2010, 12:36 pmlucia says:
Orin–
Grading is agonizing in any subject where the students need to apply specific concept to even moderately complex problems. Some analogous issues came up in undergraduate fluid dynamics courses. I wrote up a formal rubric, which worked like a charm in 99% of cases. Thats said, I think rubrics may be easier in a fluids class than in any course that involves grading writing.
The reality is that either you write a formal rubric, or you have one in your head. It’s not clear which method saves time. Bad formal rubrics are worse than no rubric. Writing good rubrics can be a time sink because you really need to think about very small things a student has to get right, and how they can go wrong before you can write a formal rubric that saves time when grading.
To some extent, to write a rubric, you also have to have a notion of how the students should be approaching the problem you put on the exam and you have to be willing to recognize that you do give credit for things some practicing lawyers might consider utterly trivial. The students are just learning, and if you don’t give points for things they are just learning, papers that exhibit mastery of 50% of the course material can end up with zeros!
IANAL, but it looks maybe they do something like this:
* Spot correct issues without including non-issues. points:____/ out of ___.
— spot issue 1.
— spot issue 2.
— don’t ‘spot’ non issues.
* Identify doctrines for issue 1. points:____/ out of ___
— doctrine a
— doctrine b.
— any more that apply.
* In light of doctrines, identify relevant facts for issue 1. points:____/ out of ___
— important fact 1.
— important fact 2.
— etc.
* For issue 1, interpret doctrines in light of facts & the issue. points:____/ out of ___
— advance possible argument 1.
— diagnose strength of argument 1.
— repeat for however many plausible arguments, you the rubric writer, anticipate exist.
(Note that in this part, you as the grader have the flexibility to give points for decent arguments even if they are based on a mistake about the doctrine or facts above. It’s here you specifically ding students for misinterpretations that might cancel but lead to the correct result. That sorts out students 1 and 3. Student 3 got ding earlier on; student 1 loses points here. Also, student 4 loses out at the diagnose strength part.)
* Repeat the things you did for issue 1 but for issue 2. points:____/ out of ___
* Come up with some sort of summary. points:____/ out of ___
* In law, you probably get a few writing style points. points:____/ out of ___
— this could included points for brevity, not bringing up irrelevant topics etc. Exam 6 does poorly here.
Obviously, the total number of points should sum to the total for the problem. Also, there will be some slop in interpreting when the student has given enough arguments, the most important ones etc. But when rubrics can be written, they do help. You don’t even necessarily have to hand out the rubric, you can keep it as a checklist for yourself just to prod you for consistency.
Of course, even with a rubric, you will still end up with challenges. This one is particularly agonizing:
The false start with a correct finish happens in engineering!. It reads as if the first approach was totally wrong and the student realized it.
I might draw a rule separating the “different approach” for the first one. I’d grade based solely on the different approach and they get whatever they managed to get. (The student might have saved you time by crossing over all the old drek before turning in the paper. But, students don’t always do that.)
With respect to test 5: On many rubrics, this test gets a very bad grade because they didn’t spot issue 2. If that’s what you think best, that’s fine, but if you want to reward knowing how to deal with something over being able to spot the issue, you can give their total points for “dealing with issues” as the (2/3* the issue they dealt with best + 1/3rd times the issue they deal with poorly.) The more conventional weighting of 1/2 and 1/2 really kills people who missed an issue. Either way is fine– it just depends on what you judge is a correct balance. (IANAL, but I suspect that on an exam, I want to weight toward figuring out if they know how to deal with issues, but I also want them to spot them. Since spotting them at all was already specifically given points, the 2/3s, 1/3rd might be useful in the ‘dealing with issues’ portion of the grading. This weighting also helps students who might run out of time.)
With rubrics, you almost never need to worry about students raising issues not on the test. They just don’t get any points for those lengthy, time consuming discussions. Time limits being what they are, the students generally end up missing lots of stuff they needed to include.
January 18, 2010, 12:41 pmPJens says:
Bryan, I agree with what you say. The person who answered exam 7 may not have ran out of time at all, they could have just sat there stumped. And yes time limits serve a valuable purpose in helping to evaluate students. I still say though that as a client, I would rather have my lawyer do part of the job outstandingly well and either need help or more time to complete the work, than receive poor or incorrect advice.
In this example, I would choose exam writer 10 as my first choice for attorney (got both answers exactly correct in the end and on time). My question though is why people are so hard on exam 7, while easier on exam writers who didn’t do as well on either part of the test?
January 18, 2010, 12:51 pmAdam Kamp says:
As a law student, I find this comment quite interesting, because I think most of us do really value consistency in grading. Just the other day, I was privy to a conversation where one student asked another, “Is the professor a hard grader?” and the thoughtful response was “Well, he’s a very _fair_ grader.” In short, we DO recognize and appreciate consistency. It’s just that there aren’t too many instances where we are asked to communicate that–after all, class evaluations for obvious reasons are handed out BEFORE the exam. But, while many students would prefer “easy” graders, our primary need is for fair graders. (I personally don’t like easy graders either, because it compresses the curve and I’m just competitive enough that I want to WIN, not just be one of a large pool at the top of the curve. If there are that many people who are doing stellar jobs, great–but let it be stellar.)
January 18, 2010, 12:54 pmOrin Kerr says:
lucia,
Interesting, although I’m not sure that approach so well-suited for law school exams.
In law, unlike in fluid mechanics, there is often no right answer. Sometimes the right legal test is obvious, and the goal is to provide an analysis that gets to why the problem is difficult. You are grading them for the quality of the analysis. It’s hard to grade that on a checklist, I think. It’s like evaluating how good a meal is, or how good a musician is.
January 18, 2010, 1:56 pmOrin Kerr says:
Adam,
How do students know who is a fair grader? Also, in a curved class, what does it mean to be an “easy” grader?
January 18, 2010, 1:57 pmChris Travers says:
Given my experience in the chemistry lab, that would probably be my exam….. (for example running all the right tests on all the wrong samples…)
I passed my college chemistry classes not on my ability to take tests or to conduct proper lab experiments but rather on the basis of being able to help struggling students understand the proper approaches.
January 18, 2010, 2:08 pmArrowSmith says:
I feel that grading on a curve does disservice to students. They pass, but end up failing at finding a job because they didn’t really NAIL the material. Now I’m talking from an engineering perspective, but I think the same principle applies across all professions.
January 18, 2010, 2:08 pmChris Travers says:
A bigger issue is that it penalizes students for taking classes with other smart students.
January 18, 2010, 2:21 pmRandy says:
I guess what the Prof is really saying is that the grading system is completely inadequate. It forces everyone to put placed in one of five boxes in an almost arbitrary way. And this is from what most would consider a conscientious professor! For those who no longer care, or don’t have the insight, one can only shudder to think how they assign the grades.
This further goes to show that the students with an A average shouldn’t really be considered as ‘better’ as the B students, who in turn are not any worse than the C students. And yet there are considered that way. Worse, professors who should know better often DO consider the A students to be ‘better’.
I know, I know — grading is a bad system, but you need to separate students somehow, and no one has come up with anything better, have they?
I disagree. First, we need to acknowledge that the grading system isn’t for the benefit of the students or the school, but for employers who want an easy way to pick off the ‘best’ students. For schools that need to distinguish the best students, you could have an optional paper assigned and let students run free to write about a topic, without a time limit, just a due date. This way, a student could spend lots of time writing and rewriting to polish it up and that would give a much better peek at who the best legal minds are in the class. Those who write up a good paper could be awarded a special award. (This is different from a regular exam in that students have time to think about the issues, go over the facts to make sure they get them right, and also the school can publish the criteria for winning the award. Any there should be as many winners as there are papers that meet or exceed the criteria).
The benefit of this is that you can have multiple issues that cross many disciplines. Often times a property case, for instance, might have constitutional issues and civ pro issues. That’s more the real world anyway.
January 18, 2010, 2:26 pmWhadonna More says:
Orin Kerr: Whadonna More, You misunderstand. The number 1 lawyerly ability is knowing and understanding the law.
That’s not an obvious definition of the phrase “lawyerly ability” but I’ll take your word for it. I’m going to go work on my athletic ability by reading about professional sports I hope to play one day.
January 18, 2010, 2:41 pmAdam Kamp says:
Obviously, the answers are going to differ from person to person, but for me, it would be if I get an exam back, read the comments, and say, “Okay, I see where I went wrong and why it was graded the way it was.” I remember feeling that way about my first semester Contracts exam, for example. As for an easy grader, this would seem to be an issue in a really tightly compressed distribution, with a lot of people at the high end of the curve. Since I’m on the other side of the table, it’s tough to be certain, but it doesn’t seem like professors are willing to make the difference between an A and a B just a few points out of a hundred, so the higher grades are well-populated. (Also, our school doesn’t seem to have faith in a true curve, or some professors make the distribution quite gentle).
In some ways, this is a tough issue for me to describe, because my grades are good enough that I don’t notice issues other students might have.
January 18, 2010, 2:43 pmlucia says:
Orin,
Agreed. But aren’t there sometimes clearly wrong ones? And don’t good legal arguments require certain elements?
I think qualitatively, some of your difficulties come up in classes on design because in those you can end up worry about things that ultimately don’t matter much, and people will disagree on what’s the best design. I never taught those, so I never tried to come up with a rubric.
Often, the final exam gets replaced by a presentation in those course. The presentation is often graded by a panel. Trying to give students guidance on what panels generally look for can be challenging. That said, when you are on one of these panels, there is general agreement on what’s good or bad. This is true whether we create short checklists of things we are looking for or not!
I tend to think that people grading nearly always do have some sort of informal rubric in their head even if it’s not on paper.
Even with a meal, you might be able to come up with a rubric that if applied with some flexibility works a bit. You might need to decides a category of meal– for example, formal dinner, picnic, BBQ etc. You could look for elements– is the meal nutritionally balanced etc. In my opinion, a formal dinner requires dessert and a certain elegance about the dishes, table cloths etc. If someone misses those, well, that not as good as one with dessert, proper linens, silver etc. (My sister does these well, I don’t.) On the other hand, a pizza dinner or BBQ doesn’t need dessert, and the spread can be casual. My husband and I do BBQ’s well.
One reason I imagine we can create rubrics for meals is as kids, we actually used to rate each others meal demonstrations in 4-H. There is subjectivity. But to grade, you need to have some notion of what’s good. I just don’t necessarily know what rates as good in a legal argument.
January 18, 2010, 2:49 pmSKI says:
Gotta disagree with this thought. Most “law” isn’t learned in law school, it is learned while practicing. Law School isn’t really designed to teach what the law says but to re-wire brains to determine, and advocate, for how the law applies top a particular set of facts and circumstances.
This board provides an endless set of examples of experienced lawyers and professors vehemently disagreeing on what the “law” is for particular situations. There is a reason for that.
January 18, 2010, 2:54 pmBruce Hayden says:
I will respectfully disagree, at least as it comes to writing applications. I would much rather have the badly written application that has much too much disclosure to the well written one that has just enough disclosure for the initial set of claims. The former is like the test taker who just dumps as much as he can, hoping that he will hit the right points (and the prof will ignore the rest). I just spend way too much of my time prosecuting reasonably well written applications where the initial claim set gets rejected based on the prior art, and there isn’t anywhere else to go. Yes, I may ultimately have (35 USC) 112 P 1 & 2 problems, but they are far easier to overcome than lack of disclosure.
Of course, both your and my applications are well written AND contain sufficient disclosure. It is everyone else’s applications that I am talking about.
January 18, 2010, 3:33 pmJM Hanes says:
Most of the public discussion about the constitutionality of healthcare mandates seems to resemble Exam #2!
I found myself remembering a comment from an Algebra teacher, long ago, who informed my parents that I understood the concepts perfectly, I just couldn’t add and subtract. The concepts have stood me in good stead, but you would not have wanted to hire me as your accountant.
Lucky the student with a professor who devotes copious time and attention to both the process and practice of grading! When I was in college, an instructor’s annotations were worth their weight in gold. I had a pretty solid sense of what I was getting right. A good grade was gratifying; a bad grade alone told me next to nothing about where and how I went wrong — the very thing I was obviously least equipped to discern myself.
Grading may be the most neglected teaching tool in the professorial arsenal. At its best, it provides an invaluable combination of judgment and mentoring on an individual basis. It’s a shame that it is so widely regarded as an unwelcome chore, when it actually represents a best use of the considerable amount of time it requires. It’s where the professorial rubber hits the road, and as a challenge to the teacher, it looks like a feature, not a bug, to me.
~
What defies my comprehension is the argument for grading on a curve. Although relative merit may be easier to determine, it strikes me as antithetical to your own lawyerly standard, in particular — a standard which seems almost self-evidently sensible. A curve speaks solely to competence within one discrete group of students, under the tutelage of one specific professor whose approach to both teaching and grading, as you suggest, may vary not only in substance from another’s, but may not be internally consistent over either short or long haul.
So too, how could an interested party, or the student himself, possibly compare the relative competence of that class, that professor, and the resulting grades to that of another class and/or professor? How could one assess even the relative competence of an individual professor’s students from one class to the next? Wouldn’t a group of capable, industrious pupils, all of whom meet your lawyerly expectations, essentially look identical to a group of obtuse, lackadaisical pupils, none of whom you deem fit to practice? The average student could end up on a completely different rung of the ladder depending solely on circumstances beyond his control.
Such a system seems arbitrarily unfair to both student and teacher.
Thanks for shedding a fascinating, thought provoking, light on what is often one of the great mysteries in education.
January 18, 2010, 3:35 pmRick says:
An observation:
In my CivPro exam, there was a bit of mathematics required to get to an aggregate dollar amount for minimum jurisdiction. Although I excel in math, in my haste, I miscalculated and came up with less than $75K. Thereafter, my answer went awry rather quickly because I went through the litany of why jurisdiction was lacking, rather than any of the issues that arise after jurisdiction is found.
I was scored very close to zero for that question (despite my otherwise excellent grasp of jurisictional issues) . . . because calculators are not the kind of thing that are regularly brought to a CivPro exam.
Good luck with the grading…
January 18, 2010, 3:42 pmBruce Hayden says:
This was one of the reasons that I was a lot happier in math and science than in liberal arts. More than once, I made an arithmetic mistake somewhere through the calculations, yet would get maybe one or two points off, despite getting the wrong answer. The prof would just follow my calculations through after that, and if I did everything else right, give me credit for that.
My favorite story though involved an accounting exam. I had every debit and credit reversed in the exam, but otherwise turned in a perfect exam. The prof had a choice of giving me a 99 or a 0. I got the 99, and didn’t make the same mistake again. I like to think that he would have done that for anyone in his class, and not because he was a fraternity brother of mine, and that we had two of his sons join the house right after that.
January 18, 2010, 3:46 pmSenator Christmas says:
Professor Kerr, it seems like the issues in your sample exams boil down to:
1. Understanding the question as given
2. Logic (given the question as understood by the student)
3. Coming up with solution (given question/logic as understood by student)
4. Communication of all of the above
Which do you feel is the easiest and the hardest for a deficient student to improve upon?
January 18, 2010, 4:02 pmUncle Fester says:
The whole post proceeds from the flawed premise that “law professors have the single greatest job in the world.”
I beg to differ and cite the following as both hypothetically and likely better jobs:
1. The fireman who steers the back of a hook and ladder truck
2. Astronaut
3. The Dos Equis beer guy, who is advertised as The Most Interesting Man in The World.
Clearly, all of this discussion that affects the lives and futures of students is thus rendered moot, rather like arguing how many angels can fit on the head of a pin.
January 18, 2010, 4:11 pmChrisTS says:
I would come down hard (though not hardest) on the student who refuses to take a position. I admit that I would also be tough on the poor writer (why is such a poor writer in law school?)
But, mostly, I sympathize with Orin. I am as much of an obsessive as he is, and as Joseph Slater points out, no one seems to care. I gave some intro student a ‘survey’ last year, asking them to indicate what they most wanted to see on their corrected work. ‘Comments’ came in last. ‘Grade’ came in first.
January 18, 2010, 5:14 pmLaura(southernxyl) says:
Wow. I remember my daughter’s frustration in high school, that she got back her grades but not her tests. How can you improve your test preparation if you have no idea what you’ve done wrong?
If it were me, I think I’d ask myself beforehand what elements I am going to be looking for. The list would probably look a lot like Senator Christmas’s. Then I’d assign points (I know you said you don’t like to do that) so that, for instance, you could get up to 20 points for understanding the facts. Understand three out of four facts and you get 15 points. I’d tell the students up front what I’m going to be looking for. If beautiful language doesn’t make my list but I’m going to go medieval on unnecessarily labyrinthine sentence structure, they need to know that.
Because of exam 9, I wonder if the school offers a writing workshop at all. Yes, I know that a person who can’t write won’t get into lawschool, but there are probably students whose writing in legal formats needs to be worked on.
I think the idea of handing out past exams, marked up with the professor’s comments, is a great idea.
Finally, I will mention something that one of my daughter’s biology instructors did, that she deeply appreciated. She always studied her tail off for his exams, b/c he wanted in-depth answers on everything, and invariably he asked some stuff that she simply hadn’t gotten in the depth he wanted, and skipped stuff that she had. So one of the questions was always: “Tell me about one thing that you prepared for this exam, that I didn’t ask about.” She always wrote several paragraphs on that one, and always got full credit.
January 18, 2010, 7:37 pmRandy says:
Laura: ” I remember my daughter’s frustration in high school, that she got back her grades but not her tests. How can you improve your test preparation if you have no idea what you’ve done wrong?”
All law profs should be required to return the papers, and then also have written up a model answer. This way, all students can compare what they wrote with the model and see for themselves where they went wrong ) and right)
January 18, 2010, 9:43 pmBarb says:
I’m in awe at what Orrin does, and I wish more teachers in other disciplines did it as well. Too often we give students vague guidelines or rubrics, when what would really help are some actual works annotated to identify strengths and weaknesses.
I’m also struck by how similar the grading on a curve concept is to the old 6.0 figure skating judging. One skater hits all the jumps, but has lousy spins and no musicality or connecting footwork, while another is a spinning top with great musical timing, fabulous footwork, and splats on two jumps. Thus the judging challenge, particularly at lower elite levels where skaters make lots and lots of mistakes. Still, someone needs to win. Skating had a technical mark and an artistic impression mark, and a set of rules that defined ordinals within that. (A 5.8, 5.9 skate wins over the 5.9, 5.8 skate in the long program, but vice versa in the short program where technical skills mattered more.) It sounds complicated, but it actually wasn’t a bad method to holistically assess complex performances with many components and many possible errors, and while there were often arguments about whether the skater ranked first should have been second, there usually wasn’t much argument at all that the skaters ranked one through five were substantially better than those ranked 6-10, and so on. The numbers were reasonably internally consistent, and some of the bigger arguments revolved around what should or shouldn’t be especially valued, or the differences seen in what judges from different countries valued most.
January 19, 2010, 12:40 amsardonic_sob says:
I think what we have here is a little divide over the definition of “brilliant application” and “mediocre application.” I pretty much agree with everything you said. I wasn’t thinking about the amount of disclosure as one of the parameters of that evaluation, more along the lines of technical correctness and “good enough” text versus snappy prose and brilliant logic. :)
January 19, 2010, 10:37 amJoseph Slater says:
PJens and Bryan: One possible compromise that is used in law schools is the take home exam. You don’t feel as much time pressure, but there is a limit on what you can do.
Adam Kamp: thanks for the response. I’m glad that students are not just looking for “easy” graders. My point was more of a whine: no student (or peer or administrator) has ever said to me, “hey, good/fair job grading!” Students do say positive things about teaching on evaluations, peers say positive things about scholarship and service, but nobody is even in the position to evaluate a professor’s overall job of grading. Just another thing that makes the process so unrewarding, although it is very important.
Randy: I’m a fan of your posts, but I will respectfully disagree with you in part. The differences between one grad and the next (B+ to A-) can often seem and even be fairly arbitrary. The differences between an A and a C are not. The A papers are noticeably, significantly, objectively better at what law school tests are designed to test. Now, one can quibble with whether law school tests are the best evaluating measure, and I think folks with some low grades can be excellent lawyers, but there is a real difference.
Rick: A few years ago, I was testing on a legal rule in labor law that, to apply, required a majority of the relevant employees to have supported unionization at one point. I wrote the problem so it was clear that only 10 of the 30 employees ever supported the union. A student wrote something along the lines of, “the rule is that a majority must have supported the union at some point. Since 10/30 (33%) is a majority the rule applies.” Partial credit.
January 19, 2010, 3:28 pmChrisTS says:
Laura:
Usually, they ask, “How can I get an A?” I answer, “Do you read my comments?” :-)
P.S. The saga of the hypothetical student continues. The energy some people put in to trying to game the system – and deceive everyone – rather than into the work, is … astonishing.
January 19, 2010, 6:53 pmv says:
前列腺光波治疗仪
January 19, 2010, 10:04 pm