Exit polls:

This is a close race, and I'll certainly agree that Bush may well lose it. But the pro-Kerry inference that some people are drawing from 50-49 exit polls strikes me as mighty odd.

I suspect the mathematical margin of error on those polls is at least +/-3%. Add to that the margin of error that flows from the possible differences between the late afternoon vote and the earlier vote, or between those who like to talk to pollsters and those who don't, and the 1% difference is way, way within the margin of error. Even if you are looking for 70% confidence rather than 95% confidence, the exit polls can't give you that.

People desperately want to know the future now, even if the future is only several hours away. But let's not kid ourselves into thinking that these are anything more than wild conjectures. And, yes, I'd have said exactly the same thing if Bush were up by 1% in those exit polls.

UPDATE: The same applies to 51-49 exit polls as well as the 50-49s.

FURTHER UPDATE: Exit polls also don't include absentee ballots.

Reporting all the information about exit polls:

A Slate article argues that it was right to report the exit polls, on the theory that this way voters get to know what the journalists know. Maybe so; I have no fixed opinion on the matter.

But if you report 50-49 exit polls, and then report 51-49 numbers with subheads such as "Mucho flattering to Kerry," wouldn't it be more helpful to also report the following, from the National Election Pool page?

The margin of error for a 95% confidence interval is about . . . +/-4% for a typical state exit poll. . . . Other nonsampling factors may increase the total error.

You might also thrown in this from the Mystery Pollster (thanks to Douglas Johnson for the pointer), if you think he's right — or come up with a better analysis yourself, if you think he isn't quite right:

Even if comparable to the final numbers — which they are decidedly not — the mid-day leaked numbers would have much greater error, perhaps +/- 7% or more.

Even if you think 95% is a more demanding confidence interval, and you think 68% confidence is fine, you'd still have a margin of error half of those mentioned above. And this is the purely mathematical margin of error, i.e., accounting only for random variation, and not for the other problems that bedevil polls; you might also warn people of other possible causes of error, such as differences between early voters and late voters, or between those who talk to pollsters and those who don't, or between the true vote and the vote reported to the pollster.

So if you're really trying to inform your readers, it seems to me the story ought not be:

Updated Late Afternoon Numbers Mucho flattering to Kerry; plus Nader makes an appearance. . . . Updated Tuesday, Nov. 2, 2004, at 4:28 PM PT

Florida Kerry 51 Bush 49

Ohio Kerry 51 Bush 49 . . .

Rather, they should be (using the Mystery Pollster's 7% level, though if you prefer to use 6% the picture doesn't look much different):

Updated Late Afternoon Numbers Mucho flattering to Kerry; plus Nader makes an appearance. . . . Updated Tuesday, Nov. 2, 2004, at 4:28 PM PT

[Margin of error: +/-7% if you want to be 95% confident in the results, +/-3.5% if you want to be 68% confident; and note that the margin of error may be even greater if early voters differ from late voters, or if voters for one candidate are more likely to talk to exit pollsters than voters for the other.]

Florida Kerry 51 Bush 49

Ohio Kerry 51 Bush 49 . . .

And if you realize that, even at the 68% confidence level, the results are Kerry 47.5-54.5 vs. Bush 45.5-52.5 — a statistical tie even at this very low level of confidence — should the headline really be "Mucho flattering to Kerry"?

(Note that some of the other pro-Kerry predictions in the Slate-reported exit polls, such as those for Pennsylvania and New York, proved to be right as to the bottom line — not coincidentally, these were mostly those in which the gap was indeed pretty large, and not just a couple of percent.)

UPDATE: Martin Plissner, also in Slate, likewise has a criticism of paying much attention to exit polls that are within the margin of error.

Exit polls:

Slate Press Box — subtitled "media criticisms" -- reports:

To hear [Joe] Lenski of Edison Media talk about it, the whole election brouhaha of 2004 can be blamed on the people who leaked the exit poll information and the outlets (Slate, drudgereport.com, wonkette.com, dailykos.com, mydd.com, et al.) that tossed the raw data out for consumption.

"I'm not designing polls for some blogger who doesn't even understand how to read the data," Lenski told the Los Angles Times yesterday. "It's like if you were graded by your readers on the first draft of your article."

Yet it is Lenski and the networks who are at fault for not telling viewers — and bloggers — the deeper meaning of exit poll data. The business of calling an election is as much an art as it is a science, and they've not been candid about that. . . .

That may well be right — but wouldn't one also say that "it is those who post exit poll results, including Slate and in particular Pressbox, who are at fault for not telling readers the deeper meaning of exit poll data," such as for instance

1. the +/-4% announced mathematical margin of error (95% confidence interval, half that if you want 68% confidence) for state exit polls,

2. the likelihood that the margin is higher earlier in the day, when the exit poll isn't complete, and

3. the possibility that the true of margin of error is higher still, because of different response rates among different people, and other factors?

Even if item 3 is obvious — and I suspect that many readers don't find it obvious — the precise mathematical margin of error is generally not well-known to readers. Shouldn't media outlets that report polls have a responsibility to give their readers the relevant information needed to figure out the poll's limitations, just as the pollsters have a responsibility to give the media and the public the relevant information needed to figure out the poll's limitations?

UPDATE: My friend Gil Milbauer points out that, at the bottom of their exit poll item, Slate did write:

These early exit-poll numbers do not divine the name of the winner. Instead, regard these numbers as a sportswriter does the line scores from the fourth inning of a baseball game. The leading team might win the game, but then again it might not. But having the early data in front of him helps the sportswriter plot the story he thinks he'll need to write at game's end.

But the analogy to reporting the fourth inning scores only goes so far -- at least reports on the fourth inning scores are precise reports of the current score (though not the ultimate score), with no margin of error. Here, there are no precise results even as of the fourth inning; and 51-49% is not an accurate way of reporting even an intermediate result that's actually 51-49% plus or minus 6%.