I watched several of the Sunday morning political talk shows yesterday. As usual they spent a lot of time discussing polls. Polls keep the talking heads in business, but they may not reflect reality. Of course, the biggest failure was Hillary Clinton’s loss to Trump in the 2016 presidential election. The American Society for Public Administration lists several of the most widely cited polls:
- Gallup Poll
- Harris Poll
- Pew Research Poll
- Quinnipiac University
- Roper Center
- Washington Post/ABC News Poll
- Marist Poll
- World Public Opinion Poll
The Pew Research Center examined the accuracy of its polling. It compared its polls with results from government data on the same topic, and found that the data tracked closely in many cases, but it warned that election polling is difficult. Polling on specific issues such as whether they are married or own their own home tracked more closely than election polling. The article recognizes the difficulty of polling in the world of social media. It says:
One limitation of this analysis is that the polling cited here comes from just one source, Pew Research Center’s American Trends Panel (ATP). But other survey panels that use the same general approach – by recruiting Americans offline and interviewing them online – provide data quality similar to the ATP.
Such surveys, however, represent just one part of the polling landscape. Many public opinion polls are still conducted by telephone using randomly-drawn samples or, even more common, are conducted online using opt-in samples. This substantial diversity in the polling field means that the results from this analysis do not necessarily hold true for any particular poll one might find. In the coming months, a Center report will provide more detail on how different types of online polls perform in this benchmarking assessment.
A University of California Berkeley study found that polls that touted a 95% confidence level actually only reached a 60% level.
Moore and Kotak obtained 1,400 polls conducted ahead of the general elections of 2008, 2012, and 2016, as well as the Democratic presidential primaries in Iowa and New Hampshire from 2008 and 2016 and the Republican primaries in the same states from 2012 and 2016. Because some polls asked about multiple candidates, the sample included results of over 5,000 surveys of how people said they’d vote on particular candidates, as well as the accompanying margins of error.
Analyzing the polls in seven-day batches, they found a steady decline in accuracy the farther from an election the poll was conducted, with only about half proving to be accurate 10 weeks before an election. This makes sense, since unforeseen events occur—such as former FBI director James Comey announcing an investigation into Clinton’s emails just a week before the 2016 presidential election. Yet most polls, even weeks out, reported the industry standard 95% confidence interval.
Polls give some indication of what is going on, but only in general terms. When polling about something that is evenly divided (like many elections) the polling is not reliable for predicting results. But it gives the network anchors and their expert guests something to talk about. As the Pew article suggests, the internet and social media have changed polling in ways that make it less reliable than the old method of calling people on the phone and interviewing them. Today, participants are more self-selected, choosing to participate, than they were in the old days.
In any case, the talking heads should talk about something other than the latest poll.