pageok
pageok
pageok
Nursery School Revisited: Does the Representativeness of a Sample Matter?

I have now read three pieces to come out of the Block & Block Berkeley longitudinal study besides the one on political orientation, and I am generally very impressed both with the thoroughness and the quality of the work. They did a staggering amount of work and collected a wealth of data.

A. Is the sample representative?

I, nonetheless, think there are still sampling and measurement problems with the political orientation study. In that study Block admits that the sample is not representative on the main outcome variable—conservatism/liberalism—but he disagrees that this is a problem.

The debate continues at Michelle Malkin's site, where she excerpts part of an email sent by Jack Block. Block informs Malkin that one of the two nursery schools was indeed open to faculty & staff, but in the other, faculty & staff children were excluded. Presumably, this was school that the Blocks in 1980 described as a "parent cooperative," which was part of the UC-Berkeley Child Study Center, though it was administered by the Berkeley Public Schools. One might reasonably wonder about the political makeup of two schools, both part of the university's Child Study Center, one open to UC-Berkeley faculty & staff children, the other a "parent cooperative" in Berkeley, California.

Given that the question of the representativeness of the sample has arisen and that Block has admitted in the new article that the sample had "relatively few participants tilting toward conservatism," it would be good for Block to point people to any paper that gives the demographic breakdown of the education and SES of the parents. Chris from MixedMemory (who has written the best blog post I've read in support of the nursery school study), has suggested to me that he seems to recall that such SES information is in one of the studies to come out of the project. If not, perhaps Block can release that information on the parents of the 95 children.

Ultimately, it would be more important to know:

1. How many of the 95 subjects self-identified as conservatives, moderates, or liberals?

2. How many of the 95 subjects self-identified as Democrats, Republicans, or Independents, if that question was asked?

3. What was the party ID and political orientation of their parents in 1969-71, if that is known? (As part of the thoroughness with which the study was done, parents were studied in 1969-71 as well as children).

B. Does representativeness matter?

Block also makes this following dismissive argument in his email on Malkin's site:

More important, the analyses were within the sample. Logically, the Malkin analysis therefore is fundamentally irrelevant and inapplicable. I suppose one cannot expect hasty and untrained reporters to be familiar with the logic of research.

Here Jack Block goes too far. Only if the relationships that he is examining are straight line linear AND he has good measures of who is or is not conservative or liberal would it not matter whether the sample was representative.

I remember once suggesting something like Jack Block's argument to Norman Bradburn, a psychologist specializing in questionnaire framing and the former provost of the U. of Chicago and research director of the Natl. Opinion Research Center. I said that, as long as I was measuring some basic psychological process, it was not too important whether the sample was representative. Bradburn archly replied that, of course, representativeness did not matter, so long as I wasn't planning to generalize my results to try to shed light on what people outside my survey were like. I got Bradburn's point.

If there are really very few conservatives in the Blocks' sample (as he admits), then it wouldn't make sense to describe what conservatives think or even what the relatively conservative think. It would be better to describe the results as reflecting what liberals and moderates think, or to contrast what the "relatively liberal" and "relatively moderate" think.

Further, on some of the issues used to try to separate liberals from conservatives, it appears that moderates (not conservatives) would score at one extreme and liberals at the other, so it would matter very much to the results if the sample was substantially skewed to the right or substantially skewed to the left. When the relationships between two variables are U-shaped, it is important to have representative samples from all parts of the spectrum or it is possible to get major relationships backwards.

In short, representativeness does matter.

UPDATE: In the comments below, the question is raised whether the Blocks intended their results to be generalizable to the general public. Their nursery school article interestingly suggests that they put at least some stock in the representativeness of the sample and its possible application beyond its unique setting:

This configuration of personality characteristics, although methodologically based on quite different procedures, is especially reminiscent of earlier speculations by Fromm (1941), the Berkeley studies of the authoritarian personality (Adorno et al., 1950), Rokeach (1960), and Altemeyer (1981), among others. Providing additional conceptual and informational support for the present findings, is the attractive recent review by Jost et al. (2003b). The congruence between our findings regarding adult conservatives and prior empirically-based understandings attests to the general representativeness of the present sample as adults and, therefore, the likely veridicality of the unique nursery school results.

(By the way, "veridical" is defined as "1. Truthful; veracious . . . 2. Coinciding with future events or apparently unknowable present realities.")

Note that in this part of their article the Blocks do view their sample of adults as representative of the larger sample of adults in society. As for the relevance of having a representative sample, here the Blocks do not consider representativeness "fundamentally irrelevant and inapplicable," as Block does in his email quoted above on Malkin's site.

Although the Blocks think that the "representativeness of the present sample as adults" is important enough to mention because it leads to the likely "veridicality" of their observations of nursery school students, it is not entirely clear what they mean by "veridicality" in this context. They could (and probably do) mean that their results are likely to be generalizable to other observations of nursery school kids in other eras and places, or they could mean something less. In any event, they don't consider "representativeness" "irrelevant and inapplicable."

Related Posts (on one page):

  1. Nursery School Revisited: Does the Representativeness of a Sample Matter?
  2. Update on the Berkeley Nursery School Study.--
  3. You Can't Tell a Conservative Without a Score Card.--
Challenge:
I wonder if treating this "study" with the seriousness you do is really accomplishing what you desire. It is, after all, more or less name calling. That is, first and foremost, its fundamental problem.

The study:

"At age 23, relatively Liberal young women are assessed independently as: vital, motivationally
aware, perceptive, Xuent, bright, with extensive and esthetic interests, somewhat
non-conforming. Relatively Conservative young women were characterized as: conservative,
uneasy with uncertainties, conventional, as sex-typed in their personal behavior and
social perceptions, emotionally bland, appearing calm, and candid but also somewhat moralistic."

Every characteristic is loaded by language to sound good (if liberals have it) and bad (if conservatives have it). Uneasy with uncertainties? How about uncomfortable with moral relativity, having a defined sense of right and wrong. Emotionally bland? How about dependable, emotionally stable. On the other hand, liberals are described as vital, motivationally aware, non-conforming (but not too non-conforming!), perceptive, and bright.

It's obvious that this study is nothing more than name-calling. Yes, of course it has problems with its methodology. But, in my view, that's the least damning of its many problems.

Any one could rewrite the above list to sound negative to liberals and positive to conservatives.

Liberals were indulgent, ostentatious, hedonistic, overly sympathetic to criminals and sinners, and relatively selfish. Conservatives, on the other hand, had a defined sense of right and wrong, believed in personal responsibility and self-sacrifice, displayed emotional stability, and were more likely to appreciate the benefits of family life.
3.26.2006 4:39am
Federal Dog:
So again I ask (since no one could name anything last time): Can anyone name an analogous effort (or efforts) by conservative psychiatrists/psychologists to stigmatize left-of-center thinkers as intellectually deficient or psychiatrically impaired? Why are these repeated "studies" always carried out by leftists against people who dissent from left-of-center political orthodoxies? What does that tell us about the political confidence and professional competence of people who try to pass off mere partisan insult as some sort of psychological "research?"


Why haven't people learned anything from identical long-standing and documented Soviet abuses of psychiatry to demonize all political dissent? Aren't they teaching 20th-century history in the universities anymore?
3.26.2006 7:37am
Fishbane (mail):
Why are these repeated "studies" always carried out by leftists against people who dissent from left-of-center political orthodoxies? What does that tell us about the political confidence and professional competence of people who try to pass off mere partisan insult as some sort of psychological "research?"

You're welcome to head to grad school and start your own study. I would encourage you to do so - I think there's a real gap in much academia that would be well served by serious rightists doing serious research. I do believe perspective matters, if only in the sort of questions one tends to ask.

This study may indeed have problems, but calling it 'mere partisan insult' misapprehends the nature and methods of inquiry, and demonstrates either a lack of understanding of how sociological research works, or...

Why haven't people learned anything from identical long-standing and documented Soviet abuses of psychiatry to demonize all political dissent?

Ah. If they're from Berkeley, then they must be commie pinkos. Got it.

Riddle me this, though. Is it more responsible to slime and defend, or to do your own study to refute? Which stance sounds more like what the study claims to show?
3.26.2006 8:56am
Fishbane (mail):
Me: I think there's a real gap in much academia that would be well served by serious rightists doing serious research. I do believe perspective matters, if only in the sort of questions one tends to ask.

I wrote in haste. I meant "and not only in".
3.26.2006 9:01am
Federal Dog:
"Ah. If they're from Berkeley, then they must be commie pinkos. Got it. "


WOW. I'm sorry, but when I said "Soviet," I was talking about the Soviet Union (see, for example, the thoroughly documented work of historians Medved and Medved). How exactly should we interpret your wildly emotional misreading of my question?


Feel free to take a breath and reread it: Can you name any right-of-center figures who have used "psychological studies" to stigmatize political dissent as intellectually deficient or psychiatrically impaired? I honestly cannot, and no one else thus far has been able to name any either.
3.26.2006 9:13am
markm (mail):
One more problem is that "conservative" has become a meaningless word. Over the last fifty years, it has been used for neo-nazis like Richard Nixon, timid libertarians like Barry Goldwater, would-be theocrats, imperialists ("neo-conservatives"), isolationists (Pat Buchanan), those who want to shrink the government and especially federal government, and even for the President who is presiding over the most rapid expansion of the federal government in over 20 years.

Liberals like to demonize their opposition by defining conservatives as prissy authoritarians, and this seems to be the definition Block used - but it's true only of a small part of the Republican base, and it may be more true of moderate Democrats and Republicans than of the Republican party as a whole. Finally, he drew his sample from Berkeley, so if he found enough "conservatives" out of 104 people to draw any conclusions at all, he must be counting the center and center left as "conservatives".
3.26.2006 9:49am
Cabbage:
How ironic, Markm! In your post decrying the meaninglessness of "conservative" you apply "neo-nazi" to Richard Nixon.

Your inappropriate use of "neo-nazi" helps render that term meaningless... If Nixon is a neo-nazi, what do we use to describe actual neo-nazis?

In fact, I believe it would be fair to characterize YOUR abuse of the language as a linguistic Holocaust! =)
3.26.2006 10:07am
Ray (mail):
It is a valid point to make by questioning the base methods at work here. That is, the Left trying to assign some kind of psychological short comings to conservatives. Ideological opponents have always of course, questioned their opponents' motives, ethics perhaps, and things of that nature. But this is rather absurd in its reach.

Another interesting aspect of the issue is that the Left has typically adhered to the nurture over nature idea of human development, and this study flies in the face of that idea. That we're hard wired to be one way or another from such an early age. Right or wrong on their speculations on ideology, they are (unwittingly I bet) giving credence to the heritability of IQ, and indeed the very existence of g which has been anathema to most Lefty psychologists (among other controversial aspects to the idea of nature over nuture).
3.26.2006 10:18am
Fishbane (mail):
WOW. I'm sorry, but when I said "Soviet," I was talking about the Soviet Union (see, for example, the thoroughly documented work of historians Medved and Medved). How exactly should we interpret your wildly emotional misreading of my question?

"Wildly emotional". That's neat, too. Me, I'd read my statement as "mildly sarcastic", but I suppose intelligent minds can differ.

And since you were narrowly talking about the former USSR, I can only assume that this:


Feel free to take a breath and reread it: Can you name any right-of-center figures who have used "psychological studies" to stigmatize political dissent as intellectually deficient or psychiatrically impaired? I honestly cannot, and no one else thus far has been able to name any either.

has nothing to do with the topic at hand, which is the Berkeley study, and you have no desire to draw a parallel. In any case, I cannot cite any such study, which reinforces my original point - social conservatives should do more academic work. Please, it would go a long way to making the debate more interesting. I disagree, and would love to argue more than the talking points of the week. Substantive disagreement is positive.

Markm got close - 'conservative' is a big tent, as is 'liberal'. I think he soft-pedals the authoritarian impulse that runs deep in the Republican coalition, but that may be me, and I don't disagree that it is present in the Democratics. Everybody wants to rule the world. I'd rather pay for counterproductive social programs like farmers not farming than pay for counterproductive foreign policy. Somehow, we got both this time. I've somehow become a liberal now, despite being nearly an anarcho-capitalist, because the Bush administration is about as antithetical to what I want for the country as I can imagine.

No fears, though, I'll be a conservative again once the Democratics get office again. Really, I did like the gridlock we had under Clinton. Suboptimal, but about the best one can hope for.
3.26.2006 10:21am
cirby (mail):

You're welcome to head to grad school and start your own study.


...and in pretty much every major psych department in the US, you're never going to get your degree if you try to defend a politically conservative thesis.
3.26.2006 10:40am
anonymous coward:
"Can anyone name an analogous effort (or efforts) by conservative psychiatrists/psychologists to stigmatize left-of-center thinkers as intellectually deficient or psychiatrically impaired?"

Maybe not precisely what you're looking for, but there are plenty of bogus right-wing studies about homosexuality.
3.26.2006 10:42am
Dick King:
If essentially all the parents from the study are liberal, and essentially all the subjects grew up in a very liberal mileau, then even if the study shows what it claims to show it doesn't distinguish the hypothesis "whiny kids don't grow up as left as their non-whiney contemporaries" from the [to me, more plausible] "whiney kids grow up with politics different from their mileau".

-dk
3.26.2006 11:03am
Federal Dog:
"Maybe not precisely what you're looking for, but there are plenty of bogus right-wing studies about homosexuality."


Thanks, AC, but that is, indeed, quite distinct from the question of political orientation. I simply find it fascinating that the left feels the need to repeatedly resort to bogus psychological studies to stigmatize dissent, whereas the right has not felt this need. I wonder what that fact reveals about the psychologies involved.
3.26.2006 11:03am
Average Joe (mail):
Near the end of the post, Jim Lindgren notes that:

When the relationships between two variables are U-shaped, it is important to have representative samples from all parts of the spectrum or it is possible to get major relationships backwards.

Not only is this statement obviously correct, it is also likely relevant to the study in question. I have read that there are indeed traits where people who are strongly liberal and conservative both score higher than people who are moderate. Unfortunately I have forgotten the exact source, but the gist of the study that I remember involved some tests of either civic knowledge or verbal ability or something like that, in which the highest scorers were conservative Republicans and liberal Democrats. Moderates of both parties and Independents scored lower. (Of course, it is also easy to make up a trait that would have such a U-shaped relationship, e.g., strength of commitment to policial causes vs. political orientation.) In any such case of a U-shaped relationship, a sample that was heavily weighted one side of the "political spectrum" would produce a spurious postive correlation.
3.26.2006 11:36am
Ken Arromdee (mail):
And since you were narrowly talking about the former USSR, I can only assume that this:...

has nothing to do with the topic at hand, which is the Berkeley study, and you have no desire to draw a parallel.


I suspect that he was trying to draw a parallel, but the parallel isn't what you think it is. He's not comparing them to Communists because their beliefs are equivalent to Communism, but rather because they abuse psychology similarly to how Communists did.
3.26.2006 12:16pm
Mahan Atma (mail):
Prof. Lindgren, I think you misunderstood Block's point:

He's saying representativeness is irrelevant because his analysis only applies within the sample. In other words, he isn't trying to generalize to anybody outside the sample.

The people in the sample are who they are. If the political makeup of the subjects is nothing like it is in the rest of the country, what difference does it make to an analysis that's explicitly limited to the sample?

The proper scientific response is simply to say: This study applies to a small handful of people who grew up in Berkeley; it's basically meaningless to anybody else. Period, end of story.
3.26.2006 12:16pm
Mikeyes (mail):
"Why haven't people learned anything from identical long-standing and documented Soviet abuses of psychiatry to demonize all political dissent? Aren't they teaching 20th-century history in the universities anymore?"

The Soviet abuses of psychiatry (which is a medical field) are well documented and are appalling in their scope. But that has very little to do with the paper presented which is a study by psychologists (who are not medical doctors) and this study did not torture or imprison anyone as far as I can tell.

The problems presented are more in the line of bias, small samples, and possibly forcing the facts to fit the hypothesis. It may be bad research (I don't know, I don't really understand the methods) but it is not in the same league as the abuses in the USSR.


The answer to the question of bias is to do more research. It is very common for papers to proclaim a point of view only to find that the work cannot be duplicated or there are problems with method or analysis. Being wrong is part and parcel of research, that's why we do research in the first place, to seek truth. I am sure you can fault Block and Block for being biased (making the truth harder to find) but if so, prove it in a scientific fashion and not by rhetoric which tends to prove nothing.
3.26.2006 12:29pm
Kovarsky (mail):


Enough already with the asinine "left wing needs to stignatize" garbage. They did a study, no matter how many times you want to put it in quotation marks. Given limited resources I don't think they were free to construct a nationwide sample that would allow them to track data for 20 years.

Nobody set out to do anything. That's the way the study was conducted, and that's what the data said. Nobody should think conservatives are intellectually inferior based on the data. As Block points out, the data is the data - all it says is that in the sample the researchers chose, instances of the LIB/CON variable tending towards the right wing of the spectrum predict decreased confidence as a 3 year old. Make of that data what you will. I don't think it's particularly representative, but it says what it says.

The sample almost certainly skewed the results, and the study should be better about explaining that bias. As I've argued before, a better way to have conducted this study would have been to have flipped the axes and make LIB/CON the dependent variable. But none of your "criticism" is thoughtful. It's a nasty little rush to judgment, balanced precariously all this fantastic claim that the data set was not constructed because of financial and geographic limitations (which it almost certainly was) and that it was constructed - deliberately - because Block knew the results of a 20 year lognitudinal study beforehand and was just "going through the motions" to make the "whiny" claim look credible.

Jim has done some great blogging on this, but Malkin's has been horrible. It's clear she doesn't know the first thing about statistics, and I can't blame Block for dismissing her. The study has flaws, some of them serious, and those should be exposed in a thoughtful, academic way - not as some cartoon in Malkin's echo chamber.
3.26.2006 12:30pm
Mahan Atma (mail):
"Not only is this statement obviously correct, it is also likely relevant to the study in question."

No, it's obviously incorrect because the authors of the study aren't trying to say anything about people who weren't in the study! And if there were any any U-shaped relationships within the sample, the correlation coeffcients for them would have been 0, not 0.4 or whatever.

I read the original article, and the authors were pretty careful to say their analysis was limited to the handful of people they studied. It's you all who insist on interpreting the study to be a generalization of anybody outside of it.
3.26.2006 12:33pm
Zach (mail):
More important, the analyses were within the sample... I suppose one cannot expect hasty and untrained reporters to be familiar with the logic of research.

How familiar should they be with the logic of evasion? It seems to me that applying the logic within the sample vs the population at large turns on exactly the points which Malkin (and now Lindgren) stresses. So why throw in the gratuitous insult?

The question of how representative the sample was is not only relevant, it's obviously relevant, and Block does science no favors when he acts as though only a cretin would fail to see why he's obviously right.

It seems clear to me that this child study was not intended to measure political alignment, and that political alignment was just an interesting trend that popped out of the data. If you were intending to measure how political alignment developed, there's just no way that you would choose Berkeley as the sole sample area. Any researcher would point out the same, common sense arguments for why the sample was likely to be unrepresentative that Malkin and Lindgren do. So why is it beyond the pale to apply that reasoning to a study which has already been published?

The second nursery schoool deliberately excluded all parents associated with the University, to increase the diversity of our subjects. 25% of the initial sample were black.

Well that's fantastic. I eagerly anticipate the next study showing that black children grow up to be black. Instead of saying what you did to increase diversity, how about showing some actual numbers about how diverse your sample actually was?
3.26.2006 12:34pm
Mahan Atma (mail):
"The question of how representative the sample was is not only relevant, it's obviously relevant..."

Please explain: How is representativeness "obviously relevant" if the study doesn't draw any conclusions about people who weren't included in the study?

Suppose I'm walking along the beach, and I pick up a couple seashells. I then say, "Look at these two shells, they're a nice color of pink, aren't they?"

Does it make sense to say, "Obviously they are not pink, because you haven't looked at any of the other shells!"
3.26.2006 12:43pm
Huh:
So if conservatives don't use psychological studies to stigmatize dissent, how are they doing it?
3.26.2006 12:47pm
Kovarsky (mail):
The question of how representative the sample was is not only relevant, it's obviously relevant, and Block does science no favors when he acts as though only a cretin would fail to see why he's obviously right.

Huh? Have you read the study? He doesn't make any claims about the broader population. So it's relevant to what? Of course it's relevant if you're seeking to extrapolate from the study and, as Jim pretty amusingly points out through his anecdote, that process of extrapolation is what drives research in the first place, doesn't it? But the critical difference is whether the audience does the extrapolation or the scientist does. In this case, Block leaves that extrapolative exercise to us, as he should. And we have carefully pointed out that its real-world use is limited, as we should.

Everyone would benefit from being careful about what is being said here - Block's "relevance" point was the representativeness argument doesn't undermine one iota what the data in the sample showed. He's right about that and I don't think there's any serious dispute about it. What you seem to be disputing is the extrapolation point, which is one that Block simply does not make.

I can certainly understand Block's frustration here, having measured with a micrometer only to have the statistically impaired malkin come in and swing with an axe.
3.26.2006 12:57pm
Kovarsky (mail):
So if conservatives don't use psychological studies to stigmatize dissent, how are they doing it?

I believe the preferred method involves bumper-sticker insinuation that liberals are unpatriotic.
3.26.2006 1:00pm
Mahan Atma (mail):
"So if conservatives don't use psychological studies to stigmatize dissent, how are they doing it?"

Honestly. Conservatives who compare this study to Soviet-style elimination of dissent really are being whiny.

The Republicans control all three branches of government right now, and they're mighty good at stigmatizing dissent. And speaking of name-calling, I can distinctly remember all the names I was called (many of which I couldn't print here, but "traitor" being the most common) when I dared to suggest three years ago that invading Iraq might not be in this country's best interest.
3.26.2006 1:04pm
Zach (mail):
Please explain: How is representativeness "obviously relevant" if the study doesn't draw any conclusions about people who weren't included in the study?

Every published study is supposed to meet standards of reproducibility, relevance, and interest. If Block is honestly not claiming that these results are applicable to anybody not in the study, then he should refrain from publishing, or fold the political alignment study into the larger study as peripheral information about the population.

How exactly would you reproduce a study that only applies to its sample? Run time backwards and test the kids again? What's the relevance of a single sample of a couple hundred kids in Berkeley, California?

Saying the study only applies to the population it sampled (More important, the analyses were within the sample.) is just a maddeningly obtuse statement to make in science. It's saying that the study isn't science at all. I mean, nobody's claiming that Block faked the study, or that he didn't measure the things he says he did, or that he measured the data on some other group and applied it to this one. If he collected the data, then it describes the group he collected the data from. That's what data is. The whole debate is on whether the study says anything about the world we live in.
3.26.2006 1:05pm
Kovarsky (mail):
Saying the study only applies to the population it sampled (More important, the analyses were within the sample.) is just a maddeningly obtuse statement to make in science. It's saying that the study isn't science at all.

That's just incorrect. And read what Block said. He wasn't saying the STUDY was irrelevant. He was saying Malkin's CRITICISM (which did involve representativeness) was irrelevant to whether the study was conducted with statistical rigor, which it was.
3.26.2006 1:11pm
Mahan Atma (mail):
<i>"Every published study is supposed to meet standards of reproducibility, relevance, and interest."</i>

Sorry, this is social science we're talking about, not physics. Social science is chock-full of irreproducible studies.

I seem to remember one that was touted quite heavily on this very site some time ago, dealing with affirmative action in law schools.
3.26.2006 1:16pm
Federal Dog:
I again invite certain posters to take a breath and reread my remarks. I merely suggested that Block's abuse of psychological diagnosis as a political weapon bears similarities to old Soviet practice (which extended far beyond extreme examples of imprisonment and torture to achieve pervasive social control beyond gulag walls).


Given the illogical and unduly emotional reactions to my simple question, it clearly hit a nerve. I wonder why that is. Couple that reaction with the fact that no one can identify any right-of-center thinkers who have attempted similar political abuse of psychological diagnosis: The "study's" claims about conservatives look increasingly ironic.
3.26.2006 1:32pm
David Matthews (mail):
First off, I just read through the conclusions, and I'll have a closer look at the study itself later, but to say that Block is not trying to extend his results to others outside of his study is just flat wrong. He, in fact, speculates (rather wildly, in my opinion) on, for example:

"Why will the psychological characteristics of over-controllers often influence them to
gravitate toward the politically conservative? As suggested here by our findings and earlier
intimated in previous political writings, timorous conservatives of either gender tend to be
easily rattled by uncertainty...."

Obviously, he feels that his results about conservatives or liberals and their personality characteristics apply to conservatives (in general and outside the study) and liberals (in general and outside the study.)

Hell, he even believes his results shed light on the current sad state (political power-wise) of the left in American politics:

"Ironically, the sheer variety of changes and
improvements suggested by the liberal-minded under-controller may explain the diffuseness,
and subsequent ineffectiveness, of liberals in politics where a collective single-mindedness
of purpose so often is required."

So, clearly, he thinks his results have relevance outside of the folks he studied in Berkeley.

The question of the relevance of the representativeness of the sample hinges not on whether Block believes that his study sheds light on general conservative/liberal personality traits (he clearly does), but on whether the Berkeley context skews these traits (he says it doesn't, or he doesn't see why it should.) Actually, though, most relevant to the representativeness question, is whether the correlation between particular traits when young and conservative/liberal affiliation when older is influenced by the highly charged left-leaning political environment of Berkeley itself? That is, does it take a special personality type ("over-controlling," for example) to develop a political view highly out of step with one's cultural environment, and would we see different results in a location like, for example, Provo, Utah?
3.26.2006 1:35pm
Average Joe (mail):
And Zach writes:

creating those publicatIt seems clear to me that this child study was not intended to measure political alignment, and that political alignment was just an interesting trend that popped out of the data.

to which I say "Bingo!" Having worked for some years as a scientist, I know that if you do alot of work to obtain data for a study, then you will want to get as many publications from the data as possible. Having many publications are good for your career and professional reputation and ideally requires studying the data deeply and thoroughly. I suspect that the authors were studying their large, laborously obtained data set, found a correlation that seemed interesting and provacative, and wrote it up for publication. Naively, you might expect a referee to have required the authors to put some sort of caveat about the sample in the paper, but referees are typically other busy scientists who volunteer their time and expertise to referee papers. They probably noted that the methodology seemed correct and let the paper be published with the idea that any further critiques of the paper would appear in the literature.
3.26.2006 1:43pm
Mahan Atma (mail):
<i>"He, in fact, speculates (rather wildly, in my opinion) on, for example:</i>"

Nice editing you have there. You seem to have omitted the very obvious qualifying statement he makes just before the quotes you cited:

"Invoking our own theoretical parlance, <b>and with regard to the present sample</b>..."
3.26.2006 1:49pm
Mahan Atma (mail):
"He, in fact, speculates (rather wildly, in my opinion) on, for example:"

Nice editing you have there. You seem to have omitted the very obvious qualifying statement he makes just before the quotes you cited:

"Invoking our own theoretical parlance, and with regard to the present sample..."
3.26.2006 1:50pm
lucia (mail) (www):
Mahan Atma
The proper scientific response is simply to say: This study applies to a small handful of people who grew up in Berkeley; it's basically meaningless to anybody else. Period, end of story.
I agree this is a proper scientific response; the results describe characteristics of a group of 95 or so people. Period, end of story.

The natural follow on question is: Why bother to perform or publish the study? Is there something remotely interesting about this particular group of 95 people?
3.26.2006 1:56pm
Zach (mail):
He was saying Malkin's CRITICISM (which did involve representativeness) was irrelevant to whether the study was conducted with statistical rigor, which it was.

Then at this point, there are no issues remaining in dispute. 1)Block conducted a study which measured several aspects of childhood development. 2) Block conducted (or had conducted) a statistical analysis of the study. 3) The statistical analysis showed certain correlations between attitudes and political affiliation for the sample investigated, using the techniques employed. 4) The study was not designed well enough to actually show that such correlations exist in the real world.

Assuming all of these points to be true, I would argue that the study is not publishable. The fourth point is simply the most important from a scientific perspective. If you publish a study claiming to show that whiny kids grow up to be conservative, that's making a claim about the real world, and the study just wasn't put together well enough to do that.

I'm sure that Block wrote nasty emails to every single site that read his study or press acounts of it and concluded that he was making a claim about real conservatives in the real world. They had the underlying science wrong, whereas Malkin had it exactly right. Perhaps we could make some sort of central repository where people could forward the nasty emails they received from Block. Perhaps we could have a separate section where newspapers run corrections noting that the results were valid only for a small nonrepresentative sample of a couple hundred kids.
3.26.2006 1:59pm
David Matthews (mail):
Damn, some people have difficulty with reading comprehension.

"with regard to our present sample" modifies what follows it in that paragraph, i.e.
"during their earlier nursery school years, female Conservatives may be viewed as trending substantially
toward over-control (Block, 2002; Block &Block, 1951, 1952; Block &Block, 1980b), as tending toward uncertainty, constriction, and compliance, becoming—usually—followers rather than leaders in their social settings. Reciprocally, our female Liberals when in
nursery school appear inclined toward under-control, manifesting an independence of evaluations, an expressiveness, and a relative unconstraint by others...."

In contrast, what I quoted addresses the larger (next paragraph) question, "why....?" And he is here applying his results to a GENERAL principle, specifically claiming that his study supports the conclusions of other studies, i.e., about conservatives/liberals IN GENERAL, and speculating as to CAUSES of a GENERAL trend -- NOT Berkeley-specific or "these subjects"-specific. Further, if he thinks that this study only applies to the <100 people surveyed, what on earth would he be doing speculating about how his conclusions reflect on the (national or world-wide) state of the political left? To wit:

"Ironically, the sheer variety of changes and improvements suggested by the liberal-minded under-controller may explain the diffuseness, and subsequent ineffectiveness, of liberals in politics where a collective single-mindedness of purpose so often is required." (This clearly does not apply to the political situation in Berkeley, where the liberals seem to be fairly effective.)

No, in fact, the statement that people seem to be misunderstanding:

"However, and of course, any sample bias carries no implication whatsoever regarding analyses of individual differences conducted within the sample."

specifically does NOT mean that he is drawing no conclusions about liberal/conservatived differences IN GENERAL. What it means is that, since he is not implying that "80% of the population at large...." it doesn't matter whether he drew from the liberal end of the pool. He's comparing traits of the liberals he did find with those of the conservatives he did find in his sample, and applying these DIFFERENCES to the population at large.

That is, his logic goes, if I dip into a bucket of red and green beans that's not well mixed, and I dip into a predominantly green part of the bucket, that fact does not matter, as long as I'm just measuring the lengths of red and green beans and finding that red beans tend to be longer. The fact that the bucket was not well-mixed according to color should not affect the relevance of the observation. However, as I stated above, the problem is one of hidden relationships. Perhaps, in the presence of a majority of green beans, red beans tend to be longer, but not otherwise. Similarly, perhaps in the context of Berkeley, we find particular (striking) character traits when young that are predictive of political persuasion when older, but these may not be the same, or as striking, in another location.
3.26.2006 2:15pm
David Beatty (mail):
The strength of the whiny/confident correlation to eventual political ideology is only r=.27. This means it accounts for only 7% of the total variance! The statistical correlation between the SAT and students freshman GPA is a whopping, comparatively, 12% or about r=.35 (it is popularly reported by the College Board itself as r=.47 and r=.48 equating to 22-23% variance)

Therefore, being whiny or being confident (things I am not sure are antipodal to begin with - this article is confidently whiny) as it correlates to liberal/conservative outcomes is only 7% predictive while your SAT/GPA correlation is 12% predictive.

While the Toronto Star article also correctly states that 7% is a rather significant predictive value in social science it is not even close to predicting ideology. Just as my high SAT score didn't predict the dismal freshman grades I received.

Remember also that correlational studies do not represent, to any degree, causality or antecedant/consequent relationships. There can be no functional assessment within a correlational study for measurements like this so we're only talking about two separate things that happen to co-occur.

The problem people have when they aren't familiar with correlational research is that they frequently assume covariance as synonymous with causality. This is not the case and you cannot make a case for causality unless you are performing experimental controlled research.

Two common misconceptions:
1. Smoking causes Cancer -- FALSE, smoking covaries with cancer and it is one of the strongest correlated values researchers have.
2. Low levels of calcium causes osteoporosis -- FALSE, calcium levels also covary with the incidence rates of osteoporosis and is a strongly correlated value but recent research into viewing violent media and violence by young children has a stronger correlate than the calcium/osteoporosis connection


Finally, as for results not being generalizable to the entire population: it should be understood that that doesn't mean it is not valid research or that it doesn't pertain to some people. If you are a dead on match for the participents in this study by socioeconomic class, educational background, parenting style, location, and you were raised in a similar time period, this research claims, at most, a rate of predictability at 7%.
3.26.2006 2:19pm
Mahan Atma (mail):
"The natural follow on question is: Why bother to perform or publish the study? Is there something remotely interesting about this particular group of 95 people?"

Again, this is social science we're talking about, and the study is very similar to a great deal of the work that's done in social science. If your point is that a great deal of social science is very limited in its worth, I would agree with you.

That's not to say something isn't worth doing at all. For one thing, you can learn something about how to do studies, from a purely methodological standpoint. Solid, well-done social science is difficult to do, and as a science you need to learn how to walk before you can run.

Second, the study suggests a hypothesis that somebody else could actually test using a representative sample. If I was seeking funding for such a study, I'd point to these results to say "there might be something interesting going on, let's investigate it." It's in the nature of a "pilot study" in other words.
3.26.2006 2:29pm
Brian Macker (mail) (www):
1) I agree with Challenge on this point. It's the old "shrew Yankee" but "greedy Jew" fallacy.

2) I further think it is flawed because it over generalizes about political positions. People are far more complex than a simple one dimensional line scale. I think even two dimensional tests like the libertarians "World's Smallest Political Quiz" are not good enough. This quiz makes the point that it is not merely about left-right.

2) Our left-right scale is an temporary and parocial environmental accident. Other countries do not even split along the same lines. That is this is provably a split that has enormous environment factors, not genetic.

3) The study seems to be in conflict with one of the dogmas of the left, that it's "nurture over nature". Thus rendering any leftist that holds this belief and cites the study a hypocrite.
3.26.2006 2:33pm
Mahan Atma (mail):
"Further, if he thinks that this study only applies to the less than 100 people surveyed, what on earth would he be doing speculating about how his conclusions reflect on the (national or world-wide) state of the political left?"

He's doing just that — speculating. Is any such speculation unsupported by his study? Yes, I would agree with that.

But that's different than saying these are conclusions he drew from the study. He actually limits his conclusions to the sample itself. I think that's clear by the qualification I cited above, as well as the following statements where he admits the limits of the sample.

By taking those sentences out of context, you ignored the qualifications he made both before and afterwards.
3.26.2006 2:36pm
David Beatty (mail):
"The natural follow on question is: Why bother to perform or publish the study? Is there something remotely interesting about this particular group of 95 people?"


95 people is actually a relatively huge social sciences study.

Clinical Psychology experimental studies are considered pretty good if you have about n=30 and research on methylphenidate treatment in children frequently have participant sizes of less than 20 and still make definitive statements.

Getting a lot of participants to get involved in research, with low levels of attrition (leaving the research), is the key to what is known as "power."

Power is a statistical analysis tool which is tied to sample size.

With correlational rather than experimental designs it's a little different. But with 95 people you have a relatively large social science sample.

Of course, because it is not an experimental design there are a literally hundreds of confounding variables (extraneous things that can affect what you're studying).

So everything must be taken with a grain of salt and you have to remember more than anything else, we're not talking causality and we are not saying every whiny child will be conservative and every confident child will be a liberal. If you are whiny there is a 7% chance and if you are confident there's a 7% chance you'll turn out liberal. Of course, a lot of people question the representative nature of the sample and that makes the generalizable effect a lot less than 7%.



http://www.stat.uiowa.edu/~rlenth/Power/
3.26.2006 2:38pm
Mahan Atma (mail):
"(This clearly does not apply to the political situation in Berkeley, where the liberals seem to be fairly effective.)"

You obviously haven't live in Berkeley in recent years!
3.26.2006 2:39pm
Mahan Atma (mail):
"Power is a statistical analysis tool which is tied to sample size."

Statistical power has nothing to do with this study, because there's nothing even remotely random about the sample.

Unfortunately for the author, he makes the same mistake talking about power, and presenting statistical significance numbers. It's a common error that shows up all the time in the social sciences, because few social scientists actually have a grasp of the probability theory underlying the statistics.

The bottom line is that this isn't a random sample, so all the statistical inference machinery is completely irrelevant to this study.
3.26.2006 2:42pm
pmorem (mail):
I'm not sure why this "study" was necessary. What really motivated it? The level of effort to "dehumanize the other" is starting to really disturb me.

This study seems to want to say that even reeducation camps won't work and that eugenics will be required.
3.26.2006 2:45pm
David Beatty (mail):
Mahan, yes it was a non-probability sampling. I'm sorry, you're correct in that respect.
3.26.2006 2:48pm
David Beatty (mail):

"I'm not sure why this "study" was necessary. What really motivated it? The level of effort to "dehumanize the other" is starting to really disturb me.

This study seems to want to say that even reeducation camps won't work and that eugenics will be required."


Actually, as I understand it, this is only a byproduct of longitudinal research. The study was not designed to find this specifically. I believe it was only a tag-along significant covariance that was noticed in their cross correlation of variables. But I may very well be wrong.
3.26.2006 2:50pm
David Matthews (mail):
David Beatty:

I draw your attention to the following:

"Given widespread inattention to the sum and interaction of ever-present attenuation effects, the frequent discordancy of distribution shapes, the competing presence of multiple but unemphasized influencing factors, and the unacknowledged frequent usage of homogenous samples, it is not surprising that obtained correlations may not impress uncontextualizing researchers. We suggest that, given the above recognitions, the relationships reported here may be viewed as worthy of respect."

I believe the author would be calling you (has called you?) an "uncontextualizing researcher." Them's fightin' words where I come from. ;-)

Mahan Atma:

Block's "three considerations," far from admitting the limitations of his study, in fact attempt to anticipate potential criticisms, and insist that his results are worthy of GENERAL application, which I believe is open to serious question. The closest he comes is admitting "[t]he sample, born in the late 1960s and achieving young adulthood about 1990, grew up in Berkeley and Oakland, an enveloping cultural context appreciably different from much of America—a factor that should be taken into account." He then proceeds to nowhere take this into account, and, to the contrary, to keep insisting, by comparing his results to that of other surveys, that it, in fact, had little bearing on the results.

As to Berkeley, I haven't been near there since 1968, when my dad took us "over to see the hippies," which was a serious disappointment -- apparently there was something going on somewhere else, and all the colorful characters were out of town. Are you implying that the liberals aren't even doing well politically in Berkeley? Or was it my use of the word "effective?" (I meant politically effective, but I was just assuming.)
3.26.2006 2:59pm
Mahan Atma (mail):
"Can you name any right-of-center figures who have used "psychological studies" to stigmatize political dissent as intellectually deficient or psychiatrically impaired?"

Conservative scientists have much bigger fish to fry, what with trying to prove global warming doesn't exist and cooking up all sorts of other numbers to accommodate the Bush Administration's agenda.

Stigmatizing political dissent is more effectively done through other channels, apparently.
3.26.2006 3:01pm
Mahan Atma (mail):
"Or was it my use of the word 'effective?' "

It was that.

I've lived in and aroudn Berkeley for about 15 years now. You can't get people here to agree on what color the sky is, let alone what to do with Shattuck Avenue!
3.26.2006 3:04pm
David Matthews (mail):
Mahan Atma:

"The bottom line is that this isn't a random sample, so all the statistical inference machinery is completely irrelevant to this study."

OK, I just read that post. You're correct on all counts. I won't argue with you any more, since you're probably also correct on the other things....

You're just far more charitable than I was being about the author's acknowledgement of his study's limitations.
3.26.2006 3:10pm
David Beatty (mail):

I believe the author would be calling you (has called you?) an "uncontextualizing researcher." Them's fightin' words where I come from. ;-)


Yes the author may but my fact still stands that at most the ability to generalize the sample to the population would be at r=.27. 7% predictive value. "At most" means that in all likelyhood, even if the sample were heterogeneous and representative the most predictive it could be is no more than the sample itself. So beyond that you have to consider the likelihood that the sample is completely statistically representative without + or - percentage points is foolish. Even your gallop polls say "margin of error + or - 3 percent"
And those are pollings of usually more like 1000 people, not 95.

I'll hedge my claims on a margin of error call.
3.26.2006 3:17pm
lucia (mail) (www):
That's not to say something isn't worth doing at all. For one thing, you can learn something about how to do studies, from a purely methodological standpoint. Solid, well-done social science is difficult to do, and as a science you need to learn how to walk before you can run.

I didn't mean to say it wasn't worth doing at all and I understand that social science experiments are difficult to do. (So are experiments in the physical sciences, but that's for another discussion.)

Whether difficult or not, to be worth doing, the study must be reported in a way that permits readers to begin to guess how it might generalize to other situations, or in this case, other groups of people. It is natural to asks: Is this finding likely to apply broadly over a political range that spans "right of Louis the XIV" to "Left of Karl Marx"? Or does it apply to a range fromm "Left of Karl Marx" to "Just to the right of Karl Marx".

People are asking this, and it's not a sign of their ignorance. Rather, it is a sign they understand that scientists are expected to describe the characteristics of data.

It may be that the social sciences may be as flawed as you claim and they are unaware of this. Still, I would think even in the social sciences, people would understand they need to actually describe their sample. When reporting a corralation between "X" and "Y", you are expected to describe the average value of "X" and variation of that variable. Likewise, they must explain that for "Y".


People are asking the Blocks whether or not there were any conservative in the group; this amounts to asking "How is Y distributed". The data that must have been collected to calculate the correlations they published. Consquently, should be easy to answer this.

The Block appears to be reluctant to answer this question, and is instead, posting to blogs explaining his results are limited to his sample -- whose characteristics he does not seem wish to describe even when asked.

So, to extend my previous question:

If the results apply only to those 95 people in the experiment, or the researchers will not describe the measured characteristics of the people therby making it impossible for readers to begin to guess which groups this result may describe, how is the final result interesting? Why did the reviewers permit publication?
3.26.2006 3:21pm
Mahan Atma (mail):
"You're just far more charitable than I was being about the author's acknowledgement of his study's limitations."

I had a career in social science before I was a lawyer, so I suppose I have more sympathy for them.

If you put a caveat in every paper you submitted that said "Oh and by the way, our sample isn't representative, so this paper is pretty much meaningless" you'd never get published, and you have no career.

So it's pretty much SOP for the author to toss in a few paragraphs in a "Discussion" section to speculate on how the research is relevant to reality in some way.

These people have to do something to earn a paycheck. And sometimes, in the right circumstances, you actually get a study that provides genuine insight into human nature despite having methodological flaws.

I don't believe this study is one of them though...

Go read about John Snow's study of cholera for some high quality science despite a lack of representative samples and the like:

Snow on Cholera
3.26.2006 3:26pm
A. Zarkov (mail):
One cannot even begin to take this study seriously until it has been replicated, and replicated on a population outside of Berkeley. Would we find the same thing in NY or Kansas? I suspect you won't see this done.
3.26.2006 3:26pm
David Beatty (mail):

One cannot even begin to take this study seriously until it has been replicated, and replicated on a population outside of Berkeley. Would we find the same thing in NY or Kansas? I suspect you won't see this done.


Well Zarkov, give someone 20 years and they might. Your criticism, for the present, is an argument that cannot be contradicted but if we were to hold your argument true in all cases, original research would never really be significant.
3.26.2006 3:38pm
Mahan Atma (mail):
"One cannot even begin to take this study seriously until it has been replicated, and replicated on a population outside of Berkeley."

I could be wrong, but I'd be surprised to see any social scientist of note who actually would take the study very seriously. Social scientists certainly tend to lean to the left, but we're not complete morons.

I sure it got more press than it deserved strictly because of the humor angle.
3.26.2006 3:45pm
David Matthews (mail):
"Snow on Cholera"

What an excellent resource, thanks! I've added it to my list of background resources for the Introductory Statistics class I teach. And now I'll "waste" another evening reading about this, instead of, well, whatever else I was going to do....
3.26.2006 3:46pm
Mahan Atma (mail):
^^^ Another teaching resource for you, in case you don't already know of it:

The Statistics text by Freedman, Pisani and Purves. Hands down the best introduction to statistics there is.

I had the pleasure of working for two of them during grad school, and Freedman was on my dissertation committee.
3.26.2006 3:59pm
Zach (mail):
One cannot even begin to take this study seriously until it has been replicated,

I'll take this as a peg to hedge my earlier claim that the study wasn't publishable. The sheer difficulty, expense, and time of doing a study that would actually measure political development with any kind of reliability is large enough that even marginal "free" results should probably be published, if only to stimulate better studies.

In other words, if you're going to go to all the expense and trouble of taking huge reams of data, you should probably err on the side of publishing whatever you find in the data. The public's paying for it all anyway, so the public should be able to see the results.
3.26.2006 4:01pm
Kovarsky (mail):
Zach writes:

<b>I'm sure that Block wrote nasty emails to every single site that read his study or press acounts of it and concluded that he was making a claim about real conservatives in the real world. They had the underlying science wrong, whereas Malkin had it exactly right. Perhaps we could make some sort of central repository where people could forward the nasty emails they received from Block. Perhaps we could have a separate section where newspapers run corrections noting that the results were valid only for a small nonrepresentative sample of a couple hundred kids.</b>

That's selective memory if I'd ever seen it. The SECOND Malkin got ahold of that poster that said he knew some of the study participants and that they all went to the same school, she immediately inferred from that post that ALL the study participants went to that school (even though that's not what the post said) and made that claim over and over again, all over the place. She has been somewhat embarrassingly been revealed to have been incorrect in that inference.

The study isn't that great, but Malkin's response to it was not careful or scientific. Please don't lay waste to all credible criticism by defending Malkin on this one.

Also, it would be advisable to distinguish between a criticism involving statistical bias and one involving statistical signfiicance. N=94 is certainly statistically siginifcant for at least some attributes. The sample is likely biased, but it would be better if the people claiming to be scientists didn't deliberately blur these two issues.
3.26.2006 4:43pm
Kovarsky (mail):
I wrote:

She has been somewhat embarrassingly been revealed to have been incorrect in that inference.
***

Wow, that's a horrible sentence. I was watching the end of the UConn-GMU game regulation. By the way, I can't believe they just hit that layup.
3.26.2006 4:49pm
David Matthews (mail):
"N=94 is certainly statistically siginifcant for at least some attributes."

"N=x" does not, by itself, create stastistical significance. How those "N" were drawn is the essential, integral part of whether the results from a particular sample (of whatever size) carries any statistical significance at all. The notion that a sample was drawn in a faulty manner (statistical bias) but that, since there were 94 faulty draws, something of statistical significance still must exist, is appealing, but false.
3.26.2006 5:05pm
Kovarsky (mail):
David

"N=94 is certainly statistically siginifcant for at least some attributes."

"N=x" does not, by itself, create stastistical significance.


Um, that's why I said (in the excerpt you emphasize, in fact) that the degree of statistical significance depends on the attribute being measured.

You continue,

The notion that a sample was drawn in a faulty manner (statistical bias) but that, since there were 94 faulty draws, something of statistical significance still must exist, is appealing, but false.

It's a good thing I never made that claim then.
3.26.2006 5:14pm
Mike G in Corvallis (mail):
I think that an underappreciated aspect of this study is the question of how the heck did this get published in a peer-reviewed journal? Both the methodology and the significance of the research have been called into question; the paper itself ought to resolve these issues. It doesn't. Do we have another Social Text fiasco here?
3.27.2006 2:47am
David Matthews (mail):
OK, let's be plain:

"N=94 is certainly statistically siginifcant for at least some attributes."

No, it's not. N=94 is meaningless. For what attribute would it have meaning, if the sampling is biased?

Lack of bias is a prerequisite for statistical significance. That's why, if the sample is biased, there's no need to discuss significance. There isn't any significance.
3.27.2006 8:47am
te (mail):
Dateline - Blogosphere

New study reveals that self-identified conservatives spend too much time fretting about trivial pop-psychology studies.

Developing . . .
3.27.2006 12:49pm
Kovarsky (mail):
David,

Enough already. I sense that you know what you're talking about, so I don't understand why you're challenging me on this. N=94 could be statistically signficant if you were measuring:

likelihood of getting married for the general population

likelihood of getting a high school degree for the general population

likelihood of graudating from college for the general population

likelihood of leaving california for the berkeley population

Etc. etc. and so on and so forth. Absence of bias is just not a prerequisite for statistical significance, no matter how many times you say it. Maybe you're operating with the common meaning of the terms, and that is confusing you. But if I measure a million IQ data points, my results, which statistically significant may be statistically biased based on the selection of my sample set.

If I were measuring the likelhood of rolling a hard eight with loaded dice, if I conduct the experiment 1000 times, i have a statistically significant sample for how frequently i'm likely to get a hard eight with those dice, although that sample is biased by the dice being loaded.

Please stop this.
3.27.2006 2:37pm
lucia (mail) (www):
Kovarsky,

You are pretty much wrong or at least mixed up and changing definitions willy nilly. Your examples are inapt likely because you aren't thinking this all the way through.

Let's say we use your roll the die 1000 times metaphor. If, Block were studying a particular loaded die, then, it appears he only rolled it once. (The students are "the die"; they aren't a subset of another group. He only measured the characteritics of this groups once.) Statistical significance of results: Zero.

If Block was trying to study the behavior of all possible dies in the universe and the dies are the individual students themselves, then it appears he ordered his 100 or so dies a single gambling house. In this case, the statistical significance of his results when applied to describing the behavior of all dies in the universe cannot be computed. Computation required the dies be drawn in an unbiased fashion.

If his goal is to study the characteristics of dies drawn from some particular source, then it's possible he can calculate a level of statistical significance. But, in that case, his value is still flawed. Why? He hasn't defined how one can recognize which dies come from that particular source and which dies do not come from that source. Once again: He cannot compute a confidence interval because the larger population is undefined!
3.27.2006 5:23pm
Kovarsky (mail):
Lucia,

I think I made it clear at least 900 times that I was talking, each of those 900 times, about the attributes of the sample set.

I also find it somewhat interesting that you are lecturing me about this, as I used to model and execute advanced statistical analyses for major pharmaceutical companies and television networks, among others. So you might want to rethink your condescending "you're not thinking this through." In fact, if you are going to be condescending about a probability experiment, you should probably learn to spell the plural of die.

Samples are never perfectly representative of populations. Nonetheless, sample populations - and the broader populations they are modeled to predict - have certain attributes that occur in such a frequency that we're comfortable saying that measured results are "statistically significant." Whether or not the attribute being measured is "statistically significant" depends on the nature and instance of that attribute. The results might be flawed, but that doesn't mean that the experimental results are not "statistically significant."

I think that you perceive me to be resisting whatever political inference you want to draw about the study. All I said was that N=94 could be a statistically significant data set depending on what attribute you are measuring. If I am measuring, for example, the average number of years an upper middle class berkely resident played varsity sports, I think N=94 would be plenty large.
3.27.2006 6:06pm
David Matthews (mail):
"All I said was that N=94 could be a statistically significant data set depending on what attribute you are measuring."

No, what you said was that N=94 is CERTAINLY significant. That's where the trouble came in.

BTW, your dice example isn't the best, because what you are doing, in it, is limiting your population to a specific pair of dice, then ASSUMING that your rolls are representative (unbiased) for that pair of dice, then finding a significant result. If, for example, you were rolling the loaded dice by always setting one down with a "1" facing up, you wouldn't even be finding out anything significant about that particular pair of dice (except that the other one doesn't have a "7").

The only methods with which I'm familiar (and not very; from your background you probably know them better than I) that can find statistical significance in the presence of sampling bias involve making assumptions about the nature and extent of the bias (I attended a talk by Nobel Laureate James Heckman where he spoke on this; since it was designed for non-PhD's like myself it was watered down considerably) to compensate for the (otherwise absolutely essential) prerequisite of a sample that is unbiased WITHIN THE POPULATION IT IS SUPPOSEDLY DESCRIBING.

What set me off, in your comment, was the word "certainly." I get people coming to me with all sorts of messed up data that they've collected, and they believe that, simply by virtue of their sample size, they ought to be able to conclude SOMETHING. Well, if we agree that "you found out something about the people you surveyed (and nobody else)," is SOMETHING, then it's usually (but not even always then) true.

Now I'll stop. (I wasn't trying to be obstinate or obtuse, really.)
3.27.2006 6:39pm
David Matthews (mail):
(Darn, I left out the most important words again)

"No, what you said was that N=94 is CERTAINLY significant [for something]."

Please insert words in brackets.

(It may well be significant for absolutely nothing, was the point of what follows.)

Anyway, now I'll really stop ;-)
3.27.2006 6:45pm
Kovarsky (mail):
David,

I'm not a phd in statistics either.

Almost every sample is statistically significant for some prediction, if that prediction is defined at a sufficient degree of granularity. The number of times Lee (that's me) stands up to go to the bathroom on each of the past 25 Monday mornings is statistically predictive of how many times I'm likely to go to the bathroom next Monday AM, but defined at that level of granularity, who cares?

At the opposite end of the spectrum resides some experiment with 10 million data points, collected from a perfectly controlled sample. Think FDA testing of drugs here.

Most experiments lie somewhere between these two poles - at one end requiring the predicted behavior to be defined so narrowly that nobody cares about the data or at the other a fairly robust experiment. Whether N=94 is "significant" depends on what type of behavior you are trying to predict. If you are predicting the likelihood of shark attacks (which you would expect in the general population to occur with a frequency of .000001%), then N=94 is going to tell you absolutely nothing. You're either going to have the number of positive experiments (shark attacks) equal 1 or 0, and whatever observed rate is going to be an utterly useless number. but that's because you've chosen to measure such an unlikely outcome.

i don't have any real problem with the idea that this study doesn't really tell us very much about the things that politicos would like it to say - certainly liberals want to say that this shows conservatives are lifetime assholes and certainly conservatives are acting overly defensive on that basis. but there's just no reason - and i believe even Jim has acknowledged this - to write this study off as some piece of bogus science. i don't think it says what some liberals want it to say, or maybe even what the experiment's designers want it to say. but that's rarely the case in scientific studies, and it's certainly nothing new to the social sciences.
3.27.2006 7:13pm
Kovarsky (mail):
by the way, david,

The only methods with which I'm familiar (and not very; from your background you probably know them better than I) that can find statistical significance in the presence of sampling bias involve making assumptions about the nature and extent of the bias (I attended a talk by Nobel Laureate James Heckman where he spoke on this; since it was designed for non-PhD's like myself it was watered down considerably) to compensate for the (otherwise absolutely essential) prerequisite of a sample that is unbiased WITHIN THE POPULATION IT IS SUPPOSEDLY DESCRIBING.

that is certainly the method we used for modeling web-surfing patterns. you end up making somewhat circular assumptions about the likelihood of an event in the general population in order to construct confidence intervals around observed behavior in a sample. it's very imperfect, of course....
3.27.2006 7:21pm
David Matthews (mail):
"Almost every sample is statistically significant for some prediction, if that prediction is defined at a sufficient degree of granularity."

I think I finally figured out that's what you were saying. I've usually got my math down pretty good, but when it comes to English, I'm a bit slow, at times.
3.27.2006 9:08pm
lucia (mail) (www):
Kovarsky,
<i>So you might want to rethink your condescending "you're not thinking this through."</i>

I find it interesting that your rebuttal to my suggestion you are not thinking this through is an appeal to authority. That is always very impressive. [vbg]

Since you like appeals to authority, might I suggest you read this chapter from some MIT lecture notes on confidence intervals. www.math.umt.edu/wilson/Math444/LectureNotes/Chapter19.pdf

The notes don't discuss bias at any length, but you will note the very last sentence states:

"If we don't know the bias or the precision, we can't form confidence intervals"

As you likely know, you cannot determine a stastitical significance if you are unable to formulate confidence intervals. I'm sure you will recall this, as, if you think back, you will recall the horror of deriving the formulas for stastical significance which first required deriving the relations for the confidence intervals.


If you read the MIT lecture notes, and think a bit (as I previously suggested), you will remember the thing you claim to be able to calculate without regard to the bias is called the precision or repeatability. It is not the confidence interval. (Or at least, it is not the confidence interval as defined in statistical texts.)

The precision is involved in determining the statistical significance, but the full computation also requires some statistical statements about the probability of bias error.

On another item: You seem to be fixated on the N=94. No one prior to you was suggesting the problems with the study were related to a small sample size and no one is claiming it now. They were discussing bias.

When I tell you this statement is wrong:

<i>All I said was that N=94 could be a statistically significant data set depending on what attribute you are measuring.</i>

I mean it must be corrected to read:

If the samples are <b>unbiased</b>, N=94 could be a statistically significant data set depending on what attribute you are measuring.

The problem with your claim is forgot things must be unbiased.
3.27.2006 11:22pm
Kovarsky (mail):
lucia this is absurd. i don't care about N=94. you've got some axe to grind about an inference from this study that i didn't make. by the way, that looks like a very nice class you teach or take or found on the internet. somebody said that this study is so stupid because N=94 and that the sample could never tell you anything about any potential general population sampled. you can gerry-rig confidence intervals if you take some minor licenses (some fairly safe assumptions about standard deviation, etc) with known attributes of the larger population. i merely pointed out that you could say certain things about the sample with some statistical confidence if you were measuring certain types of attributes. obviously - as i pointed out over and over and over and over and over again - i selected examples that involved "upper middle class kids" from berkeley. that's the population for which i suggested the sample might have below the threshold of real world bias that would allow you to say something meaningfully predictive. unless you are claiming that a sample can be perfectly predictive of the general population it samples - which is not a realistic proposition - when we talk about a sample being biased we are talking about relative degrees of bias. but you know that. i'm not interested in comparing the size of our respective protractors with eachother. i'm sorry you find this study to be such a statistical outrage. i take it you generally see much better in the fat part of the social sciences.
3.28.2006 1:11am