Exit Polls: This is from The Campaign Spot:
We all remember how far off the exit polls were in 2004.

But we don't recall how far off some of the leaked exit poll results were in 2006. In the races were expected blowouts — Ohio, Pennsylvania, Rhode Island — the exit polls were pretty accurate. But on some of the close ones — the ones where race-watchers want reliable data the most — they managed to botch it pretty badly.

Virginia Exit Poll Result: D: 52 R: 47.

Actual result: Webb 49.59 percent, Allen 49.2 percent.

The exit poll margin was 5 percent; the actual margin was less than one percent.

Montana Exit Poll Result: D: 53 R: 46.

Actual result: Tester 49.16, Burns 48.29.

The exit poll margin was 7 percent; the actual margin was less than one percent.

Arizona Exit Poll Result: R: 50 D: 46

Actual result: Kyl 53, Pederson 44.

The exit poll margin was 2 percent, the actual margin was 9 percent.

Because they correctly predicted the ultimate winner, no one remembers these poll results as being egregiously off-base. But a few thousand votes here and there, and they would have had the wrong winner in Virginia and Montana.

Ten Reasons Why You Should Ignore Exit Polls: From FiveThirtyEight:
1. Exit polls have a much larger intrinsic margin for error than regular polls. This is because of what are known as cluster sampling techniques. Exit polls are not conducted at all precincts, but only at some fraction thereof. Although these precincts are selected at random and are supposed to be reflective of their states as a whole, this introduces another opportunity for error to occur (say, for instance, that a particular precinct has been canvassed especially heavily by one of the campaigns). This makes the margins for error somewhere between 50-90% higher than they would be for comparable telephone surveys.

2. Exit polls have consistently overstated the Democratic share of the vote. Many of you will recall this happening in 2004, when leaked exit polls suggested that John Kerry would have a much better day than he actually had. But this phenomenon was hardly unique to 2004. In 2000, for instance, exit polls had Al Gore winning states like Alabama and Georgia (!). If you go back and watch The War Room, you'll find George Stephanopolous and James Carville gloating over exit polls showing Bill Clinton winning states like Indiana and Texas, which of course he did not win.

3. Exit polls were particularly bad in this year's primaries. They overstated Barack Obama's performance by an average of about 7 points.

4. Exit polls challenge the definition of a random sample. Although the exit polls have theoretically established procedures to collect a random sample -- essentially, having the interviewer approach every nth person who leaves the polling place -- in practice this is hard to execute at a busy polling place, particularly when the pollster may be standing many yards away from the polling place itself because of electioneering laws.

5. Democrats may be more likely to participate in exit polls. Related to items #1 and #4 above, Scott Rasmussen has found that Democrats supporters are more likely to agree to participate in exit polls, probably because they are more enthusiastic about this election.

6. Exit polls may have problems calibrating results from early voting. Contrary to the conventional wisdom, exit polls will attempt account for people who voted before election day in most (although not all) states by means of a random telephone sample of such voters. However, this requires the polling firms to guess at the ratio of early voters to regular ones, and sometimes they do not guess correctly. In Florida in 2000, for instance, there was a significant underestimation of the absentee vote, which that year was a substantially Republican vote, leading to an overestimation of Al Gore's share of the vote, and contributing to the infamous miscall of the state.

7. Exit polls may also miss late voters. By "late" voters I mean persons who come to their polling place in the last couple of hours of the day, after the exit polls are out of the field. Although there is no clear consensus about which types of voters tend to vote later rather than earlier, this adds another way in which the sample may be nonrandom, particularly in precincts with long lines or extended voting hours.

8. "Leaked" exit poll results may not be the genuine article. Sometimes, sources like Matt Drudge and Jim Geraghty have gotten their hands on the actual exit polls collected by the network pools. At other times, they may be reporting data from "first-wave" exit polls, which contain extremely small sample sizes and are not calibrated for their demographics. And at other places on the Internet (though likely not from Gergahty and Drudge, who actually have reasonably good track records), you may see numbers that are completely fabricated.

9. A high-turnout election may make demographic weighting difficult. Just as regular, telephone polls are having difficulty this cycle estimating turnout demographics -- will younger voters and minorities show up in greater numbers? -- the same challenges await exit pollsters. Remember, an exit poll is not a definitive record of what happened at the polling place; it is at best a random sampling.

10. You'll know the actual results soon enough anyway. Have patience, my friends, and consider yourselves lucky: in France, it is illegal to conduct a poll of any kind within 48 hours of the election. But exit polls are really more trouble than they're worth, at least as a predictive tool. An independent panel created by CNN in the wake of the Florida disaster in 2000 recommended that the network completely ignore exit polls when calling particular states. I suggest that you do the same.

Related Posts (on one page):

  1. Extreme Measures:
  2. Ten Reasons Why You Should Ignore Exit Polls:
  3. Exit Polls:
Comments

Extreme Measures: From the LA Times:
But behind the elaborate preparations and gung-ho attitude, the television networks are heading into the night with a sense of cautious restraint -- especially when it comes to exit polls -- all too aware of the implications of a botched call.

"We don't want ever a repeat of what happened in 2000," said Phil Alongi, executive producer of NBC's special events, referring to the networks' haste in awarding Florida to Al Gore, then giving the state to George W. Bush before realizing it was too close to call.

"We learned so many lessons across the board," he said. "One of the first: Get it right."

To do so, the networks now follow strict rules that govern projections, examining not only exit poll data but actual vote tabulation and turnout information. NBC -- which keeps its decision desk isolated from the calls made by competing networks -- will only call a winner once its statisticians conclude that the chance of an error is less than 1 in 200. And no calls will be made until all the polls have closed in a state.

Extreme measures are taken to ensure that early data from the exit poll does not leak out, as it did in 2004, when the first wave of surveys showing John Kerry in the lead rocketed through cyberspace.

For much of the day, only a small group will have access to the exit poll, which is being conducted by Edison Media Research and Mitofsky International for the National Election Pool, a consortium of the networks and the Associated Press. Three members from each outlet will be sequestered in an undisclosed location in New York, where they will analyze the results of questionnaires filled out by 100,000 voters nationwide. Their cellphones and BlackBerries will be taken away until 5 p.m. ET, when they will be allowed to share the data with their newsrooms.

It's the same procedure that was used in the 2006 midterm elections and effectively prevented the release of incomplete data, much to the relief of network executives.

"Exit poll information in the hands of trained professionals is perfectly fine," said Sam Feist, CNN's political director. "Exit poll information in the hand of the general public, who may not understand what it means or stands for, can be dangerous."

That's because exit polls are designed to provide a demographic portrait of voters, not to predict the winner of a close race. The early waves of data can be especially misleading because they do not necessarily reflect an accurate sample of the electorate.
Comments