Responding to Critics (1): A New Test of the Mismatch Theory:

The basic argument of Systemic Analysis is simple: if there is a very large disparity at a school between the entering credentials of the "median" student and the credentials of students receiving large preferences, then the credentials gap will hurt those the preferences are intended to help. A large number of those receiving large preferences will struggle academically, receive low grades, and actually learn less in some important sense than they would have at another school where their credentials were closer to the school median. The low grades will hurt their graduation rates, bar passage rates, and prospects in the job market. This is what I call the "mismatch effect."

My paper tested this idea by comparing the outcomes of whites (who generally receive small or no admissions preferences from law schools) with blacks (who generally receive large, race-based preferences) to compare the outcomes of students who start with similar credentials. My results are robust and, as I'll discuss in coming days, have withstood criticism pretty well. But I and everyone else agree that it would be preferable to compare blacks with other blacks. In other words, the ideal control group for examining blacks who receive large racial preferences would be a group of blacks who received smaller preferences, or no preferences at all.

As I discuss in my Stanford "Reply to Critics", such a comparison group not only exists -- we now even have data on their outcomes. After Systemic Analysis had gone to press, Ian Ayres and Richard Brooks at Yale pointed out that the Law School Admissions Council, in one of the surveys administered to students in its Bar Passage Study (a major source for my paper), had asked the students in detail about how they applied to, and selected, the law school they attended. About ten percent of the 1800-odd blacks in their study reported that they had chosen to pass up their "first-choice" school even though they had been admitted to that school. Most of these students apparently went to a lower-choice school because of financial aid offers or for geographic reasons. The data suggests that these black "second-choice" students had credentials substantially closer to those of their classmates. Compared to other blacks, these blacks closed nearly half the credentials gap.

These "second-choice" students are not a perfect control group, of course -- no one was randomly assigned to attend schools offering different levels of racial preference -- but it is about as good a chance to test the mismatch theory as we are likely to have for some time. If the theory is right, then the second-choice students should have better outcomes: higher graduation rates and more success on the bar. In the table below, I make predictions about how the blacks going to their second-choice schools should perform, based on simple linear assumptions (if blacks going to second-choice schools close one-third of the credentials gap with their classmates, they should close a proportionate amount of the outcomes gap, once one controls for index differences).

If the theory is wrong, in contrast, then of course the blacks going to second-choice schools should have about the same outcomes as blacks who took full advantage of the preferences they were offered. In the data presented below, we'd expect the blacks going to second-choice schools to do slightly better, since they somewhat better index scores than the average black law student (but this difference alone would only close about one-eighth of the gap in outcomes).

The actual outcomes look like this:


White Success Rate

Success Rate for Blacks Other Than Those Going to Second-choice school

My prediction of success rates for blacks going to second-choice school

Actual Success Rate for blacks going to second-choice school

Graduate from Law School





Pass Bar on First Attempt





Pass Bar Eventually





Proportion of Original Cohort Becoming Lawyers





These are pretty remarkable results. The "mismatch" predictions are either right on target or, in some cases, too low. The differences in success rates between black law students generally and those going to their second-choice schools are huge. As with everyone else, the black second-choice students' outcomes depend heavily on their grades. But these blacks are substantially less mismatched than other blacks, and they get substantially higher grades (they average about ten percentile points higher in their classes -- another outcome exactly in line with predictions).

Many critics of Systemic Analysis, when they come to the question of why black law students have such low graduation and bar passage rates, either offer no explanation or rather wearily suggest a "something about race" problem. These data offer a very clear example of how well blacks can perform.

There are two sorts of objections one might raise about this data. First, are the samples involved large enough to produce statistically significant, reliable results, or could these results somehow be a fluke? And second, is there some way that the blacks going to second-choice schools are systematically different (other than their slightly higher credentials) from other black law students? I think the answers are (a) the results are very reliable and (b) there are no alternative explanations for these results. But these require slightly longer explanations, and I'll elaborate in my next post.

Donald (www):
What do the numbers say about whites at their second choice schools vs. whites "other than those going to their second choice" school? If there is a gap between the two that is comparable to the one for the two groups of black students, that would refute the argument that the disparity is due to race preferential admissions policies, wouldn't it? And wouldn't the more likely explanation then be that all students who attend their second choice school (a) may be at a less rigorous institution; (b) are more likely to receive financial assistance at the second choice school, and thus more focused on their studies than on the need to pay rent, but food, etc.; or (c) some combination of these.
6.10.2005 5:56pm
Troy Hinrichs:
I'll give my all-time favorite response to such studies...

"I don't care what the numbers say, this disparity is due to historic and ongoing racism."

I love that response. No matter how blue the sky looks -- it is actually gray.

In all seriousness, Donald's point is a good one and if all second choice students succeeded at substantially rates wouldn't that confirm what conservatives already know? People are just "people" and not all that different.

Great and brave work Mr. Sander.
6.10.2005 6:29pm
Mike1830 (mail):
Responding to Donald's comment: Even if your hypothetical study is conducted with whites and the same results are shown as with the study of black students, that still does not undermine the conclusion that race prefences or ANY preference completely devoid of merit, allows students to attend a school (e.g. a school too competetive for the student) where the student is at a severe disadvantage and less likely to succeed.
6.10.2005 6:34pm
VC reader (mail):
In response to Donald, I may have missed the issue but I thought that Sander has been saying all along that the mismatch effect is independent of race. What makes it a race issue is how the admissions process determines who will be admitted with substantially lower credentials.
6.10.2005 6:53pm
Wince and Nod (mail) (www):
They may be harder to study, but we may be able to find whites who are given preferences to attend a school for which they are less academically qualified. I'm refering to legacy students. It ought to be even easier to find white students who passed up a legacy situation for one where they were more academically qualified.

If your theory predicts the performance of legacy students too it would pretty much knock down the "historic and ongoing racism" argument.

6.10.2005 7:06pm
Shepard Barbash (mail):
Siegfried Engelmann, the country's leading expert in the field of reading instruction, has a vast body of data generated since the 1960s that show the effects on learning rates of appropriate and inappropriate (ie. mismatched) placement of students in a given curriculum. His data corroborates Sanders' observations: students placed at the appropriate level of a curriculum progress faster and eventually pass students who have been misplaced at a higher level of the same curriculum. See esp. his article, "Student-Program Alignment and Teaching to Mastery," at his website (the piece is near the bottom of the site on the left).
6.11.2005 2:15pm
jab (mail):
Question about methodology:
Do you check for Simpson's Paradox?
This is a sneaky statistical problem that can really warp grouped data when there may be hidden or lurking variables.

Here is a couple websites that give examples of Simpson's Paradox:

6.11.2005 9:14pm
SupremacyClaus (mail):
The Bar exam is the ultimate employment test. No Bar, no job in the law.

It has a racial disparity. It lacks any scientific validity in the construction of the test, in its reliability, in its correlation with real world quality of lawyering, or any consumer protection. It refuses to reveal any of its data on test construction. It refuses to reveal data on age discrimination, and ethnic discrimination.

It violates Title VII of the 1964 Civil Rights Act. It should be banned. Damages are due those who failed it due to race disparity.

Why is the self-dealing lawyer not subject to these sanctions, like everyone else?

Answer: The criminal cult enterprise is in charge of the 3 branches of government, has dealt itself absolute immunity, and does as it pleases with impunity, enforced by Airborne Divisions.
6.11.2005 11:26pm
Mahan Atma (mail):
The biggest problem with your analysis is one you briefly acknowledge, but cannot overcome. This is NOT a control group you are looking at, because the group is self-selecting, not random.

Like many social scientists, you have a tendency to downplay the weakness of observational data, e.g. data not derived from randomized, controlled experiments. You do so because no such data exists to test your hypothesis -- a plight to which I am sympathetic, having a former career in social science -- but this shortcoming does not somehow convert your analysis into a scientific one.

The point is a simple: There's no control group because there's no randomness involved. That's it, end of story. Hence any use of the term "control group" is fraudulent, no matter how you qualify it. And if you choose to hang the phrase "statistically significant" on these differences, that will underscore your misuse of the data, because without some randomness, the concept of "statistical significance" is utterly meaningless.

Unfortunately, it's a distinction that will be lost on your audience members, many of whom are lawyers (a notoriously illiterate group when it comes to statistics and quanititative analysis). But I'm a lawyer with advance degrees in statistics and quantitative methods, so I'll call out your analysis with the label it deserves most: Nonsense.
6.12.2005 12:29pm
Michael @ CIR (mail):
The problem with "second-choice" theory is that the LSAC data is very poorly suited to use it. Ayres and Brooks point out a number of these difficulties, viz., that some respondents viewed the term "first choice" as the first among all law schools (regardless of whether the student applied), and others saw it as the first choice among those to which they applied (regardless of whether they were admitted). Some people who applied to only one school identified the school they went to as their "second choice." Many who were accepted at only one school listed it as their "second choice," and many more who were only accepted at two listed their school as their "third choice." Even for those who viewed the phrase in question as the "first choice" among those schools that accepted them, the assumption is that they preferred the more competitive and prestigious school (as opposed to, for example the school with the better temperature and climate or the better Asian Law studies program). Sorry, but if you're trying to replicate Dale and Kreuger, I think both you and Ayers/Brooks will need a much, much better data set. (Ayers and Brooks used a narrower subset of the data that you criticize, and they came up with the opposite conclusion as you. But at least they recognize the limitations of the data.)

That being said, you do say in your draft "reply to critics" that you made some controls to make the data more reliable. (Draft Reply, n.24.) I remain a skeptic about that, but if you want to convince people of the validity of this approach, you should at least tell them what the controls were.
6.12.2005 3:34pm
Mary Campbell Gallagher (mail) (www):
I know about the bar exam. My company teaches bar exam courses, and I have written a book, Scoring High on Bar Exam Essays. We teach many re-takers, black and white, here in New York. Their law schools not only offer them no support but take no interest in them. The same deans who strenuously defend preferences in admissions neither know nor care what happens to their graduates who are unsuccessful on the bar exam.
6.12.2005 9:04pm