Markos Moulitsos of the Daily Kos website has recently revealed that there was probably large-scale data fraud in a series of polls that the Daily Kos commissioned from the Research 2000 polling firm:
I have just published a report by three statistics wizards showing, quite convincingly, that the weekly Research 2000 State of the Nation poll we ran the past year and a half was likely bunk….
We contracted with Research 2000 to conduct polling and to provide us with the results of their surveys. Based on the report of the statisticians, it’s clear that we did not get what we paid for. We were defrauded by Research 2000, and while we don’t know if some or all of the data was fabricated or manipulated beyond recognition, we know we can’t trust it. Meanwhile, Research 2000 has refused to offer any explanation.
I mention this unfortunate incident because I wrote a blog post and a Forbes op ed where I analyzed data on political ignorance among Republican voters derived from a Research 2000 poll commissioned by Kos. In the post and the op ed, I questioned some of Kos’ interpretations of the data, noting some methodological problems and pointing out that ignorance and irrationality about political issues is common among voters across the political spectrum, not just Republicans.
Still, like Kos, I assumed that the data itself was real. It did not occur to me that a major polling firm would risk its reputation by engaging in what may well turn out to be outright fraud. Even if unscrupulous pollsters weren’t worried that fraud would be discovered by the general public, it was surely likely to be revealed by trained researchers, as happened here. And obviously the political campaigns, marketers, and others who hire pollsters are unlikely to go to to those with a reputation for making up bogus data. That said, Kos should probably have been more suspicious and so should I.
The specific Research 2000 poll I wrote about has not been analyzed or called into question by independent experts. So it’s possible that its findings were legitimate. Still, as Moulitsos points out, the firm’s misdeeds with respect to the State of the Nation surveys necessarily casts doubt on the rest of its work for Kos:
While the investigation didn’t look at all of Research 2000 polling conducted for us, fact is I no longer have any confidence in any of it, and neither should anyone else.
There is plenty of other evidence showing widespread political ignorance, serious irrationality, and conspiracy-mongering among voters across the political spectrum. However, that is no reason to rely on Research 2000 data that may be fraudulent.
I’m hardly a fan of Daily Kos and its politics. But they should be commended for admitting that they may have been duped by Research 2000 and for publicizing the evidence of this possible fraud. For my part, I should point out that I too made use of Research 2000 data that should now be regarded with great skepticism unless and until independent researchers confirm its validity.
UPDATE: I should emphasize that the problem here is different from the much more common scenario of questions that are poorly worded or even deliberately written in such a way as to produce a desired result. Rather, the problem is that Research 2000 seems to have made up or distorted much of the data tabulating how survey respondents answered the questions posed to them. The wording of polls is usually publicly available. Readers and researchers can judge for themselves whether it was well done or not. A poorly worded poll is therefore not fraudulent or deceptive, even if its results may give a distorted view of public opinion. The fraud that Research 2000 may have engaged in here is far more reprehensible.