Polling Has Changed. The Way We Read (And Sometimes Misread) It Has Not.

Uncategorized

In the 15 years since I started writing MysteryPollster, pre-election polling has changed a lot, yet when it comes to what we expect from polls, much remains the same. We still assume polls can tell us “what’s going to happen,” when the most clear-cut finding right now is that we just don’t know. 

Yes, changing survey methods deserve close scrutiny and warrant caution, but if the primary election results take us by surprise, I’d wager a major culprit will be our long-held tendency to overlook voter uncertainty and assume too much precision in horse-race polling. 

Let’s start with what has changed. 

In 2004, most state pre-election polls were conducted by landline telephone because nearly all U.S. adults (92%) had a landline phone, and just 7% of voters were cell-phone-only. Today, more than half of Americans (57%) have dropped landlines in favor of mobile phones, and nearly all live-interviewer telephone surveys call both mobile and landline phones. 

In 2004, the few media polls conducted online were largely dismissed by mainstream news outlets because those polls typically sampled from “opt-in” panels rather than using more traditional random sampling that could theoretically reach the full population. Today, owing in part to much lower response rates that helped drive up the cost of telephone surveys, election polls are increasingly conducted online, especially at the national level. This past November and December, online surveys accounted for 40 of the 55 national Democratic primary polls compiled by FiveThirtyEight.

Media coverage of polling has also changed, mostly for the better. Fifteen years ago, blogs like mine were as close as we came to a social media conversation about pre-election polls. Few media pollsters discussed their methods outside of academic and professional journals or conferences. 

Today, many news sites provide highly sophisticated polling analysis, averages and charts, and many pollsters and academic experts routinely engage in social media discussions of their work. 

So the good news is that we have plentiful analysis on how the changes in survey methods might affect their accuracy (one great example is the recently published “Field Guide to Polling: 2020 Edition” from the Pew Research Center). 

But for all the changes, most of us who follow politics still crave the latest polls, especially around elections, as we always have. Consider the trend in Google searches for the word “poll,” which has shown the same seasonal pattern since 2004. Poll searches spike to their highest levels in October or early November of presidential election years. They also spike, but not quite as high, just before “off-year” general elections and in the period we are in right now: just before the early presidential primaries. 

Google Trends: Searches on “Poll”
Source: Google Trends

With that craving comes our long ingrained habit of “serially, myopically obsessing over each individual horse-race poll as it’s released,” as my former colleague Ariel Edwards Levy puts it. Since November, headline writers have catered to that appetite by trumpeting poll “surges”for Pete Buttigieg, Joe Biden, Bernie Sanders and just yesterday, Tom Steyer

Implicit in all this is the assumption that ample polling can measure the state of the race, “with, like, crazy precision” as Chris Cuomo recently put it on CNN. The problem is that polls have always been imprecise, at best, in predicting the ultimate outcome, especially in a wide-open, multi-candidate primary contest like the one we’re watching now. 

In the same CNN segment, Harry Enten explained that the median shift in Iowa for the winner, comparing poll averages a month out to the final result, has been nine percentage points over the last eight contests since 2000. The shifts were as big, according to Enten, as 20 percentage points in 2012 and 32 points in 2000. 

CNN’s Harry Enten shows historic shifts from Iowa Caucus poll averages to results,
source: Twitter.

Iowa is just the beginning, of course, but the early primaries help narrow the field and usually cause voter preferences to change further. Consider the 2004 Democratic primaries, which saw one of the biggest nationwide shifts of the last few decades. Support among Democrats for John Kerry truly surged from the single digits to near 50 percent during January after he unexpectedly won both Iowa and New Hampshire. 

Why so much potential volatility? The reason is something plainly evident in the current polling: Democratic voters are having a tough time choosing a favorite from a field of candidates they like. Yes, the vast majority will name one candidate as their first choice when asked to do so by a pollster. But on follow-up questions, massive numbers of the same voters say their minds are not made up:

Edwards-Levy has been regularly reporting results like these as “the one number you should be looking at.” She’s right about that. Democratic primary voters “are still very far from settling on their decisions,” she concluded in November. That’s still true now, and probably will be for another few months at least. 

Buckle up!

Typos corrected. Photo credit: Miryam León on Unsplash


Mark Blumenthal

Mark Blumenthal is the principal at MysteryPollster, LLC. With decades of experience in polling using traditional and innovative online methods, he is uniquely positioned to advise survey researchers, progressive organizations and candidates and the public at-large on how to adapt to polling’s ongoing reinvention. He was previously head of election polling at SurveyMonkey, senior polling editor for The Huffington Post, co-founder of Pollster.com and a long-time campaign consultant who conducted and analyzed political polls and focus groups for Democratic party candidates.