U.S. News Measurements of Post-Grad Outcomes:

My colleague (and one-time VC guest bloger), Andrew Morriss has co-authored an interesting new paper with William Henderson on U.S. News' use of post-graduation criteria in its annual law school rankings. This aspect of the rankings, which include employment rates, average salaries, and bar passage, account for 20 percent of a school's overall rank, and have been the focus of substantial law school efforts to increase their ranks. The abstract of the paper, Measuring Outcomes: Post-Graduation Measures of Success in the U.S. News & World Report Law School Rankings, is below:

The U.S. News & World Report annual rankings play a key role in ordering the market for legal education. This Article explores the impact and evolution of placement and post-graduation data, which is an important input variable that comprises 20 percent of the total rankings methodology. In general, we observe clear evidence that law schools are seeking to maximize each placement and post-graduation input variable. During the 1997 to 2006 time period, law schools in all four tiers posted large average gains in employment rates upon graduation and nine months, which appear to result from a combination of competition and gaming strategies. Law schools in tiers 2, 3, and 4 have also increased 1L academic attrition, which may be an attempt to increase the U.S. News bar passage score.


33yearprof (mail):
I'm not allowed to teach first year classes anymore because I actually award "F's" to those who earn them. In a seventy person torts class, it's not hard to find the bottom if the exam is difficult enough to produce a good intellectual spread among the answers.

It was lauded in the 1970's and early 80's as "rigor," joked about in the late 80's and 90's as my "Kingsfield complex," then eliminated by offering me only upper level courses in the new century.

IMHO, law school faculties have abandoned their traditional "quality control" function. It's not worth the grief you get for awarding a fair but poor grade to subpar work. So teachers just bottom out at a "C-" and let 'em slide on through.
1.6.2007 2:06pm
ZF (mail):
But it's still those nasty businesspeople who compete against each other at the expense of their customers, right? We do still inhabit a pristine moral universe compared to them, right? I need to be sure we will still get to speak of them disdainfully.
1.6.2007 3:20pm
Eugene Volokh (www):
ZF: Any chance you might spell that out a little? I know your comment was sarcastic, but while I'm usually pretty good at figuring out the point behind the sarcasm, I'm having a hard time grasping it here.
1.6.2007 3:49pm
Jay (mail):
I agree with the Prof. Is that an allusion to something?
1.6.2007 4:00pm
Erik Puknys (mail):
I haven't read the article, but how can across the board "large average gains in employment rates" be attributed to law school "gaming"? The period of the study (1997 to 2006) was one of unprecedented explosive growth in the demand for associates (and salaries) in law firms across the country (except for a brief slowing in 2000-2002).

Thus, I'd think we'd expect to see an increase in employment rates (and average starting salaries) during the period of the study, and not because of law schools' reactions to US News's methodology, but simply as a reflection of a marketplace that saw dramatic changes in the last ten years.
1.6.2007 4:27pm
Robert Schwartz (mail):
Rick: How can you close me up? On what grounds?
Captain Renault: I'm shocked, shocked to find that gambling is going on in here!
[a croupier hands Renault a pile of money]
Croupier: Your winnings, sir.
Captain Renault: [sotto voce] Oh, thank you very much.
Captain Renault: Everybody out at once!

For the Culturally Impaired
1.6.2007 4:56pm
Waldensian (mail):

ZF: Any chance you might spell that out a little? I know your comment was sarcastic, but while I'm usually pretty good at figuring out the point behind the sarcasm, I'm having a hard time grasping it here.

Ditto -- I hate reading snark and not understanding what it's about.
1.6.2007 5:27pm
dick thompson (mail):
I have a problem with the use of raw salary data in studies like this. The problem is that if you have raw salaries of $100K average from NYU and $75K average from South Dakota U, then it seems like the NYU people are doing better. However when you match the salary with the cost of housing and living in general, then the metric changes totally. I would rather see something such as average salary compared to local average salary. That would seem much more meaningful.

As an example assume I was offered a job at 100K in NYC and also a job at 80K in Cincinnati. When you look at it the 80K job actually paid me a higher salary in cost of living terms than the 100K salary did. The metric that the magazine is applying does not account for this difference in any way.
1.6.2007 5:58pm
Elliot123 (mail):
Of all the people who enter law school, what percent eventually pass the bar? What percent finish LS? What percent of LS grads pass the bar? Anyone have that handy?
1.6.2007 6:35pm
Sadly, the numbers provided by law schools about the employment rate and the average salary rate are grossly inaccurate and exaggerated, all in a futile attempt to "increase" the school's placement in the rankings business.
1.6.2007 7:13pm
I for one welcome our magazine overlords:
Everything is broken!
1.6.2007 7:28pm
Cornellian (mail):
I'm not allowed to teach first year classes anymore because I actually award "F's" to those who earn them. In a seventy person torts class, it's not hard to find the bottom if the exam is difficult enough to produce a good intellectual spread among the answers.

Why do the worst exams merit an "F?" If a grade of "F" indicates the student lacks even a basic comprehension of the material that's one thing. I'd have to wonder about the effectiveness of the professor who gave out a lot of "F" grades of that sort since a basic grasp of torts is not beyond the ability of anyone capable of getting a half decent score on the reading comprehension section of the LSAT. If the "F" grade simply means "worst paper in the class" then the law school is perfectly entitled to say as a policy issue they're not going to fail people who do understand the material (albeit at a basic level) just because the other students in the class have an even better understanding of it. Seems to me that's a very reasonable position to take.
1.6.2007 11:10pm
Ahmed (mail):
F means failure, it doesn't mean worst student in class.

If the school has any sort of decent admission standards, then F's should be very rare. Too many very intelligent and hard working people are trying to get into law school for the top several schools to have a significant number of students who fail to learn the law.

Second, the manner in which these grades come about is also very annoying to me. I sit and read a thousand pages and get grilled in class for a few months learning how to (I would hope) practice law, but my exam is a three hour marathon to write a 15 page issue spotter? Bizarre. Make me write 30 briefs in torts, make me write 15 K's and briefs for Contracts. Just assign the work throughout the semester to enforce IMPROVEMENT DURING THE SEMESTER, and let the people can lawyer best get the great grades. Why do we cling to this bizarre final exam = 100% of the grade notion?
1.7.2007 12:11am
DHR (mail):
The study author's comment about the increased 1L attrition rate is interesting in light of (the pending case?) Bentsey v. St. Thomas School of Law. When the authors say, "Law schools in tiers 2, 3, and 4 have also increased 1L academic attrition," it strongly supports Bentsey's contention that law schools are intentionally admitting students who are then academically dismissed, especially given their US News rationale.

In addition, this language from the study suggests the St. Thomas' public statements are misleading. When George Sheldon, Associate Dean of St. Thomas Law School says, "Why would you admit people and dismiss them early if you're trying to get their resources?" the authors argue that it may be related to improving rankings; the plaintiff argues it's to improve bar passage and prevent ABA reviews of their accreditation. Likewise, when Sheldon says, "...the attrition rate at the school is about 12 percent, typical for law schools ranked similarly to St. Thomas" this would also be misleading because he's comparing other schools engaging in the same behavior. Just because every other law school is doing something improper does not make that conduct any more acceptable.

When this case was on here a few months ago, there were a lot of comments about how frivolous the claim was. I haven't read the study nor seen the data, so I don't know if this alleged increase in attrition is true or if the authors are intentionally taking aim at St. Thomas' statements. Regardless, I think it would be worth revisiting this issue in light of this study (once someone has read it and reviewed the data, of course).
1.7.2007 1:06am
Peter Wimsey:
I am generally sympathetic to the idea of dismissing some students for academic reasons after their first year, particularly at schools like St. Thomas that admit a large number of potentially academically unprepared students. IMO, it is much better for the students to be released after the first year with $25k in debt than to graduate after 3 years being $75k in debt and then fail to pass the bar and get legal employment. And of course there are opportunity costs.

Nor do I believe that - at any school - admission is a guarantee of graduation; even marginal students are smart enough to figure out that their C undergrad average suggests that law school may be difficult. But I do think that these schools do serve a useful purpose for students who, despite marginal academic achievement, can nevertheless succeed in law school and in practicing law. But of course it is a gamble, for the school and the student - and if the student is willing to risk his own money and otherwise meets the school's qualifications, he should be admitted, even if he may not succeed. And again, if he won't succeed, it's much better to cut his losses after the first year than after the third.
1.7.2007 2:49pm
eric (mail):
I agree with Peter. What about the 27 year old guy who does well enough on the LSAT but stayed drunk throughout undergrad but has grown up? What about the person fresh out of undergrad with a good GPA but bad LSAT's. If you want those, then you have to be fooled by a few who just cannot make it. They should know very quickly that this is the case, hopefully after the first semester. Perhaps bar passage should count for much more than 1L attrition.
1.7.2007 3:06pm
ZMW (mail):

I was that 27 year old guy (and boy did I have fun in undergrad, but the real world grew me up quick). Went to what is a third or fourth tier school according to the rankings. Did rather well. And was much more schooled in trial work than most of the first or second tier folks I matched up against in the courtroom my first year out.
1.7.2007 3:54pm
DHR (mail):
I agree with the idea of dismissing students academically when they are not capable of handling the workload or are unqualified academically. However, the authors of the study appear to be saying law schools are increasing attrition rates for reasons unrelated to a student's actual performance, namely to improve US News rankings.

From what I understand of Bentsey's argument, the cut off point was a 2.5 gpa or so, but the average class grade was only a 2.8, give or take. He's arguing a quarter of the class is being cut. That's a lot of law students cut on a statistical basis. I'm critical of the idea that all those people didn't have even the minimal legal skills necessary or would never have developed them (especially as assessed after the first year). Admittedly, one or two people being dismissed is one thing, but in a class of 200, dismissing 24 (12%) to 50 (25%) of them is another entirely.

If that increased attrition is a trend unrelated to actual performance, then the actual circumstances (i.e. drunk throughout undergrad or bad LSAT's) are irrelevant - it's not just some individual person but a class of people who just didn't make the cut. The people not making the cut could be the unlucky person who just had a bad day, or studied the wrong thing, etc.

So while I agree that dismissals are generally acceptable, my point is that when they are unrelated to actual academic performance, then that's pretty unfair to them, considering the extremely negative consequences of such a dismissal (namely, the fact that it is very difficult and up-hill battle to be re-admitted to a new law school afterwards).
1.7.2007 4:45pm
Ardbeg (mail):
Well, about seven years ago when I applied to law school I researched the system like crazy (analyzed statistical data, called admissions officers, hung out on discussion boards, other embarassing behavior), trying to make up for disappointing rejections from having "blown" the college admissions game. Anyway, a few things I noticed about one particular "national" law school that seemed cheesy to me. (I shall not name this school so as not to upset alumni, but suffice it to say it's in Chicago and doesn't have the word "Chicago" in the name, and no, I didn't attend law school anywhere in the Midwest). First, over the previous several years their median LSAT had gone up a decent bit, but their 25th percentile GPA had plummetted, doubtless because the former played a huge role in rankings while the latter was irrelevant. Second, their "employed at graduation" number jumped from 85% to an improbable 99% in one year, and the rumor (confirmed by at least a few annecdotal cases) was that jobless bottom-of-the-class grads were being offered low-paying "research" positions with little actual work responsibility. Third, among those law students at the "Trinity" (US News top 3) law schools that would admit to receiving any rejection letters, the only schools named were usually the other Trinity schools and this unnamed school. Which is to say, I suspect they were rejecting "overqualified" applicants that they knew were unlikely to attend to reduce their acceptance rate and boost their matriculation rate. And from the time I started watching rankings my junior year in college to the time I ceased caring after 1L year, this school did manage to jump approximately a half dozen places in the rankings.
1.8.2007 1:24am
JRM (mail):
I went to a top 20 law school in the Midwest which boasted something like a "95% employed at graduation, 99% employed after 3 months" type statistics with great salary figures. These figures were one of the reasons I chose the school. They were also competely misleading.

As it turns out, they get the stat by sending graduates postcards in mail asking if they are employed and, if so, what they are doing and how much they are making.

Who do you think returned the cards—the people making $100k a year for big law firms or the people living in their parent's house and working part time as a trainer at the local gym?

And besides the embarrassment aspect of it, it only hurts my chances of getting a job if I let the school know I am unemployed and they have to report it and it lowers their ranking. Why would I do that?

They know the figures are not accurate, but they report them anyway, because that is how the U.S. News rankings game is played.
1.8.2007 8:59am