Back in the early 1990s, Gary Mauser (Simon Fraser University, British Columbia) and I wrote an article, originally published in Political Communication & Persuasion, explaining why polls are sometimes inaccurate as measures of public opinion. The article is titled 'Sorry, Wrong Number': Why Media Polls on Gun Control are So Often Unreliable, and although the focus is on polls about gun control, the article observes some general problems with polling. If Senator Obama on election day significantly underperforms, or overperforms, what the polls predict, there will be many possible causes, other than the "Bradley Effect" or the "Reverse Bradley Effect." There are many factors, other than race-consciousness of the interviewees, which may cause a gap between opinion polls and actual votes.
Some Reasons Why Polls May Be Inaccurate
might be feverishly scratching around for some little straw of hope.
What you have from an election is a decision, not an opinion.
So apart from measuring what does not exist, it's measuring the wrong units.
Public opinion is a ephermeral as the news for a good reason; it takes newspapers to create its illusion.
We're supposed to be concerned about Obama's father now?
First, historically, the consensus on Presidential elections is almost always correct. Only 2 of the last 18 elections haven't been accurately called. If there were systematic errors, we wouldn't see that kind of historical accuracy. (Consider also the accuracy of the primary polls.)
Second, there are far more pollsters doing election polling than there are on other issues. Different pollsters use different methods (some call cell phone users, some are internet-only, they use different likely voter models, etc.). So if they all agree, they eliminate a lot of the possible sources of error.
Third, even if every poll is systematically missing some cleavage of voters, internal change in the poll is still significant. If Gallup misses youth voters, then their topline numbers will be inaccurate, but change in those numbers toward Obama still mean he's more likely to win the election.
Of course, given the Electoral system, national aggregates of polls are meaningless. A candidate could easily win, let's say Cali and NY by 90 %, and it makes no difference as compared to 51 %, in the Electroal counting.
Among other things, do we really think that in 2008, a significant number of people are telling annonymous pollsters on the phone -- or in some cases, automated phone services -- that they are voting for Obama when they are really going to vote for McCain, because they worry that the annonymous pollster or machine will think they are racist?
Plus, Obama is opening up a pretty wide lead. He's up by two touchdowns (53-39) in the newest CBS poll. And the state polling news is pretty bleak for McCain two.
Denial, anger, bargaining . . . I know they all come before acceptance, although I can't remember the order.
Of course, the system is only as good as the inputs -- 528 has the added advantage that all the information, including the polls and their weights, are provided in full detail.
Short answer -- there's a perfectly reasonably way to make reasonable national aggregates of data.
Another interesting thing is the probability of an electoral college/popular vote split. Again, the odds are way down now that the election looks to be a blowout but at one point it was also ~5%.
1) You explicitly limit your study to media polls, which you go on to demonstrate are often cranked out on compressed schedules, with attendant cut corners in terms of proper polling methodology. Although there are some media polls being conducted this cycle, the majority of national polling data (not by source, but by volume—e.g., the Rasmussen tracker and the Gallup tracker are both producing more data and interviewing more respondents than the CBS/NYT, CNN, Fox News, and ABC/WaPo once-every-two-week one-offs put together), and the vast majority of state polling data is being produced by either professional polling firms or educational/public interest groups that specialize in opinion polling. Especially in the case of professional pollsters, such as Rasmussen Reports and SurveyUSA, their reputation as accurate and reliable is the largest determinant of their success, and thus they have strong incentives to employ rigorous scientific polling methodology.
2) Regularly-released poll series run by professional or independent pollsters generally have a good deal of data from cross-tabulations available, either to the public or to subscribers. This allows polling experts in the media and the blogosphere (unfortunately, it's become painfully accurate this cycle that the smartest voices on this issue are being posted online—whether major media news orgs just don't hire polling experts or don't put them on TV, cable news coverage of polling suffers from a number of flaws) to point out sample problems which might undermine the accuracy of the poll's central findings. This alleviates, to some extent, the difficulty you describe in your answer with obtaining an accurate sample for the question you're trying to survey. The major caveat here is that the question of the composition of this year's electorate is still very much up in the air; I would say the general consensus is that it's likely to be younger, browner, and just generally more Democratic, but the degree to which we can count on that is certainly contested between polling experts. Nonetheless, professionally-run polls do have this major advantage over media polls: the latter are generally very jealous of their cross-tabs, although disclosure has actually been up over this election cycle and the last one (which obviously significantly postdates your research paper).
3) One of the major concerns you treat in your paper, question wording, is really not an issue at all with regard to Presidential trial heats. Question order could possibly be affecting results slightly, but I believe most pollsters ask the "top-line" question at the top of the call, or second after demographic questions, which is probably the best you can do in terms of not influencing respondents one way or the other. The somewhat related issue of pushing leaners and how the prompts for that push are worded is a thorny issue in and of itself, but once you get past the first debate voter preferences tend to stabilize quickly. "Leaned" preferences probably reflect the most likely vote choice for respondents who end up voting at all at this point in the election season.
4) You mentioned that your paper goes beyond the Bradley Effect and related phenomena, but in case any other readers are unaware, I would note that there is substantial evidence* that the Bradley Effect disappeared in the mid-1990s. Both of those links incorporate data from Obama's own performance during the primaries to show that there is simply no evidence that white voters are saying one thing to pollsters and doing another in the privacy of the voting booth. This also speaks to your point, made in the paper, that the perceived characteristics of the interviewer can influence responses; and it's worth noting that one pollster generating a lot of state polling this cycle, SurveyUSA, uses only robo-calling, which essentially eliminates that effect entirely.
5) It's true that polls are generally worse measures of public opinion than the officially acknowledged margin of error, but polls can show trends, and when they converge they are much more often right than wrong. Some pollsters are also measurably better than others, and it's possible to take that into account when weighing the various polls one sees generated each day. The superstar of adjusting polls for the broader universe of "X-factors" at play in politics is, of course, 538, and I would encourage anyone who hasn't visited the site yet to take a look. Nate Silver is fairly aboveboard with his methodology, and he does have a strong background in statistical analysis.
In conclusion, I think your concerns about public polling and its accuracy are overstated as applied to this election. While it is true that polls are not exact measures of public opinion, the general "consensus" among publicly-released polls of the past two weeks that John McCain trails by about 8 points nationally and by various margins in the key battleground states is most likely indicative of the fact that John McCain is in fact trailing by about 8 points nationally and by various margins in the key battleground states. Looking ahead, it seems overwhelmingly likely, based on the information available, that Obama will win the election unless McCain turns in a very dominating performance at tomorrow night's debate, or if some major non-campaign-related news event alters the race's dynamic during these last three weeks. Historical trends show it to be highly unlikely that McCain can make up this much ground, this late, through standard campaign tactics.
*I keep getting the comment rejected with an error when I attempt to link to the paper using standard HTML tags, so the link is: http://people.iq.harvard.edu/~dhopkins/wilder13.pdf
As far as 538's "accuracy", are you refering to the accuracy of their aggregation/weighting/de-biasing techniques or are you referring to their predictive power? The former is fair game but it seems unfair to judge them on the latter since they can only use the poll data that they are given.
He's advising Barack by ouija board.
based on the length of the screed that followed, i find this assertion dubious :)
The thing that bothers me about this is that a number of studies have found that significant numbers of voters at least partially base their voting preferences on whom they think will win: People like to be on the side of the winner. There is a real danger that a statistically insignificant polling lead may actually generate a self-fulfilling prophecy. This may give polling organizations a dangerously large role in affecting election outcomes.
Or enough right bias in the right-leaning polls that the race is not as close as current polls suggest. I suggest the biases in the aggregate polls cancel each other out and Obama's got a 6-8 point lead. It also looks like it's slowly expanding,
However, I don't trust polls at all. I've been following 538 and others, and one thing I notice is that 538 is VERY much more volatile in their swings, and always in favor of the Dems. I have heard someswhere the the owner of 538 is an extreme liberal, I don't find the cite right now. I discoutn them to some extent, as I do Zogby for the same reason.
The main guy from 538 is from Chicago. Hmmmm.... and at this link, provided by him,
http://www.newsweek.com/id/140469
he admits being an Obama supporter. Of course, if having a political opinoin disqualified pollsters, there would be no pollsters.
Another factor - I do believe that in the psychology of Republicans or Conservatives ( 2 different critters, of course ), there is more of a mindset to 'not reply to strangers on the phone asking you for personal opinions', whereas the mindset of liberals is the opposite - they are more prone ( IMO ) to want to scream their opinions at everyone around them.
All that being said - IMO polls or perhaps interesting ( I do follow them ), but provide very little information of any real use.
I will go out on a plank here and make a prediciton of my own - this year will see a continuation of the upturn in barely-pre and barely-post election legal action, with both sides contesting every little thing. And that the Dems will sue over EVERY loss they think they can possibly get overturned in court. They NEVER lost an election, they've only had some stolen from them. Just ask them :-)
Nate has been upfront about his support for Obama since the site was launched during the primaries. Nevertheless, 538 did manage to compile a fairly impressive track record during the primaries. Nate's open about his biases, he's open about his methodology, including the periodic changes has made, and I for one am very interested to see how his model performs against the election results.
I know that numbers sometimes scare "conservatives" when the numbers don't support the ideological outcome. 538 makes no qualms about its political affiliations. If you have a critique of their methodology, please offer it.
fwiw, I think the paper that Prof. Kopel wrote is a fair criticism of issue polls involving gun control, but I have my own bias as a supporter of gun rights. I'm curious if Prof. Kopel thinks his critique applies to larger polls with a more well developed methodology.
Hmm. On this day in 2004 that site predicted that Bush would get 284 electoral votes. He would up getting 286. They were wrong on Wisonsin and Nevada but pretty much had the other states nailed.
Click on the states themselves and the picture becomes clearer. The aggregate poll information was very accurate, although several states did swing considerably in the final few weeks prior to the election.
As for what electoral-vote thinks now? not good for McCain
McCain needs something major to change the election. Otherwise he's toast.
What do you know. Obama leads there too!
It has been suggested that this dynamic could be a huge hidden advantage for McCain in the battleground states, the reasoning being that Obama's popularity is highest among young adults, a demographic that has historically had low turnout on Election Day. Maybe this will pan out, maybe it won't. That said, it's a sad observation that McCain will probably need this dynamic to break in his favor - basically relying on the vagaries of chance and "unknown unknowns" - in order to pull off the win.
one thing is certain - when McCain loses, the right will scream 'ACORN !! ACORN !!'
To that, I say :
NUTS ! NUTS !!!!!
:-)
Why wouldn't this apply equally to supporters of the winning candidate that figure he can win well enough without their vote?.
Is it possible that Obama might legitimately win AND ACORN is corruping the process? Something like this happened in 1972.
By the way, one of the previous comments suggested that national results are "meaningless" because "a candidate could easily win, let's say Cali and NY by 90 %, and it makes no difference as compared to 51 %, in the Electroal counting." That's true, but, according to 538's analysis, this insight favors Obama, whose support is distributed much more "efficiently" for capturing Electoral College votes. 538 is currently estimating that it is more than 8 times more likely that Obama will lose the popular vote and win the election than it is that McCain will lose the popular vote and win the election.
Naturally. But that's not what we'll be hearing. We'll be hearing that ACORN corrupted the process and that's why Obama won.
By the way, is it possible that Bush legitimately won in 2004 AND there were a bunch of Diebold shenanigans that went unpunished?