In a recent Language Log post, Mark Liberman argues that surveys overestimate the extent of political ignorance. Unfortunately, his evidence is far from compelling.
He notes a few examples where scholars or reporters simply misstated the results of a particular survey. That surely happens. But it doesn’t account for more than a small fraction of the survey evidence finding widespread political ignorance.
Liberman also cites evidence that surveys based on “open-ended” questions sometimes overestimate ignorance because the coders are given bad instructions. An open-ended survey is one where the respondent is asked a question (e.g. – “Who is the Chief Justice of the United States”), and then must give an answer that he comes up with on his own, instead of choosing from a pre-set range of choices, as with a multiple choice question.
Open-ended questions do indeed have their flaws. But extensive political ignorance shows up in multiple-choice surveys too. For example, multiple choice surveys showed that only about 32% of the public knew that Paul Ryan was a member of the House of Representatives. These polls were taken before he was nominated for the vice presidency but after he had been a major figure in American politics for several years. Other multiple choice questions reveal massive ignorance about the distribution of federal spending. Back in 2009, a multiple-choice survey found that only 24% knew that “cap and trade” is an environmental program, even though it had just passed the House of Representatives (I cite the data in this article). And there’s many other examples where those came from.
Moreover, if open-ended survey items overstate ignorance, multiple-choice questions often understate it, because ignorant people will sometimes get the right answer by guessing. In an age of standardized testing, many people are used to the idea that they should guess on a survey question if they don’t know the right answer. And some prefer that option to admitting ignorance. If there are 4 options on a multiple-choice question, random guessing gives you a 25% chance of getting the right answer, and your odds go up if there are only 2 or 3 options.
Finally, it’s worth noting that a recent article by political scientists John Bullock and Robert Luskin shows that much of the criticism of open-ended questions is overstated. Even after correcting for various flaws pointed out by critics, they find that the evidence still shows a generally low level of knowledge.
In sum, both open-ended and multiple choice questions have their shortcomings. Survey researchers and people who cite their work should exercise due caution. But the overall picture painted even by multiple choice surveys is one where the public is ignorant about a wide range of basic facts about numerous political issues.