I have now read three pieces to come out of the Block & Block Berkeley longitudinal study besides the one on political orientation, and I am generally very impressed both with the thoroughness and the quality of the work. They did a staggering amount of work and collected a wealth of data.

A. Is the sample representative?

I, nonetheless, think there are still sampling and measurement problems with the political orientation study. In that study Block admits that the sample is not representative on the main outcome variable—conservatism/liberalism—but he disagrees that this is a problem.

The debate continues at Michelle Malkin's site, where she excerpts part of an email sent by Jack Block. Block informs Malkin that one of the two nursery schools was indeed open to faculty & staff, but in the other, faculty & staff children were excluded. Presumably, this was school that the Blocks in 1980 described as a "parent cooperative," which was part of the UC-Berkeley Child Study Center, though it was administered by the Berkeley Public Schools. One might reasonably wonder about the political makeup of two schools, both part of the university's Child Study Center, one open to UC-Berkeley faculty & staff children, the other a "parent cooperative" in Berkeley, California.

Given that the question of the representativeness of the sample has arisen and that Block has admitted in the new article that the sample had "relatively few participants tilting toward conservatism," it would be good for Block to point people to any paper that gives the demographic breakdown of the education and SES of the parents. Chris from MixedMemory (who has written the best blog post I've read in support of the nursery school study), has suggested to me that he seems to recall that such SES information is in one of the studies to come out of the project. If not, perhaps Block can release that information on the parents of the 95 children.

Ultimately, it would be more important to know:

1. How many of the 95 subjects self-identified as conservatives, moderates, or liberals?

2. How many of the 95 subjects self-identified as Democrats, Republicans, or Independents, if that question was asked?

3. What was the party ID and political orientation of their parents in 1969-71, if that is known? (As part of the thoroughness with which the study was done, parents were studied in 1969-71 as well as children).

B. Does representativeness matter?

Block also makes this following dismissive argument in his email on Malkin's site:

More important, the analyses were within the sample. Logically, the Malkin analysis therefore is fundamentally irrelevant and inapplicable. I suppose one cannot expect hasty and untrained reporters to be familiar with the logic of research.

Here Jack Block goes too far. Only if the relationships that he is examining are straight line linear AND he has good measures of who is or is not conservative or liberal would it not matter whether the sample was representative.

I remember once suggesting something like Jack Block's argument to Norman Bradburn, a psychologist specializing in questionnaire framing and the former provost of the U. of Chicago and research director of the Natl. Opinion Research Center. I said that, as long as I was measuring some basic psychological process, it was not too important whether the sample was representative. Bradburn archly replied that, of course, representativeness did not matter, so long as I wasn't planning to generalize my results to try to shed light on what people outside my survey were like. I got Bradburn's point.

If there are really very few conservatives in the Blocks' sample (as he admits), then it wouldn't make sense to describe what conservatives think or even what the relatively conservative think. It would be better to describe the results as reflecting what liberals and moderates think, or to contrast what the "relatively liberal" and "relatively moderate" think.

Further, on some of the issues used to try to separate liberals from conservatives, it appears that moderates (not conservatives) would score at one extreme and liberals at the other, so it would matter very much to the results if the sample was substantially skewed to the right or substantially skewed to the left. When the relationships between two variables are U-shaped, it is important to have representative samples from all parts of the spectrum or it is possible to get major relationships backwards.

In short, representativeness does matter.

UPDATE: In the comments below, the question is raised whether the Blocks intended their results to be generalizable to the general public. Their nursery school article interestingly suggests that they put at least some stock in the representativeness of the sample and its possible application beyond its unique setting:

This configuration of personality characteristics, although methodologically based on quite different procedures, is especially reminiscent of earlier speculations by Fromm (1941), the Berkeley studies of the authoritarian personality (Adorno et al., 1950), Rokeach (1960), and Altemeyer (1981), among others. Providing additional conceptual and informational support for the present findings, is the attractive recent review by Jost et al. (2003b). The congruence between our findings regarding adult conservatives and prior empirically-based understandings attests to the general representativeness of the present sample as adults and, therefore, the likely veridicality of the unique nursery school results.

(By the way, "veridical" is defined as "1. Truthful; veracious . . . 2. Coinciding with future events or apparently unknowable present realities.")

Note that in this part of their article the Blocks do view their sample of adults as representative of the larger sample of adults in society. As for the relevance of having a representative sample, here the Blocks do not consider representativeness "fundamentally irrelevant and inapplicable," as Block does in his email quoted above on Malkin's site.

Although the Blocks think that the "representativeness of the present sample as adults" is important enough to mention because it leads to the likely "veridicality" of their observations of nursery school students, it is not entirely clear what they mean by "veridicality" in this context. They could (and probably do) mean that their results are likely to be generalizable to other observations of nursery school kids in other eras and places, or they could mean something less. In any event, they don't consider "representativeness" "irrelevant and inapplicable."