So WHY Were the Exit Polls “Wrong?”

December 27, 2004October 1, 2019 Mark Blumenthal24 Comments

This post is another summary that mostly ties together various items I’ve covered separately, but also adds some new material.

We know that the exit polls had an average error favoring John Kerry of 1.9% per precinct. What explains the error? There are many theories that are, in my view, far more plausible than the notion of widespread fraud or problems with the count. Unfortunately, we lack definitive proof of a cause, both because NEP has not released any of its internal analysis and because such proof is often elusive. Let’s review what we do know.

Warren Mitofsky and Joe Lenski , the researchers that conducted the exit polls on behalf of the National Election Pool (NEP), have so far been circumspect in there public comments. They have offered theories, but stopped short of claiming definitive proof for any explanation of the error favoring Kerry. Here is a sampling of their on-the-record speculation:

Kerry was ahead in a number of the — in a number of the states by margins that looked unreasonable to us. And we suspect that the reason, the main reason, was that the Kerry voters were more anxious to participate in our exit polls than the Bush voters…in an exit poll, everybody doesn’t agree to be interviewed. It’s voluntary, and the people refuse usually at about the same rate, regardless of who they support. When you have a very energized electorate, which contributed to the big turnout, sometimes the supporters of one candidate refuse at a greater rate than the supporters of the other candidate. (Warren Mitofsky on The News Hour, November 5, 2004)

In addition, some inquiry into what went wrong with the exit polls is also necessary. Thankfully, Lenski told me that such a probe is currently underway; there are many theories for why the polls might have skewed toward Kerry, Lenski said, but he’s not ready to conclude anything just yet. At some point, though, he said we’ll be able to find out what happened, and what the polls actually said. (Farhood Manjoo for Salon.com, November 12, 2004)

One thing [Warren Mitofsky] confirmed to me is that the average deviation to Kerry in the completed version of the exit poll is estimated at +1.9%. When asked if the full 1.9% deviation could be explained by non-response bias (Kerry voters being more likely to complete the exit poll than Bush voters), he said, "It’s my opinion, but I can’t prove it." He went on to say that it would be an impossible thing to "prove" categorically because there exist an infinite number of variables that could have a micro-impact on the exit poll which could combine for a statistically significant impact. These factors ranged from the weather to the distance from the polling place some of his poll takers were forced to stand. He is also trying to determine whether there is a statistically significant correlation between certain types of precincts and the non-response deviation. Again, right now he feels the most reasonable and logical explanation of the average 1.9% deviation for Kerry was non-response bias. (Blogger Chris Johnson of Mayflower Hill, November 17, 2004).

Whether the internal NEP analysis ever sees the light of day remains an open question. The networks have obviously resisted public disclosure to date. Four years ago, a Congressional investigation into the election night snafus helped motivate several networks to release internal reports. That pressure is lacking this year, so we will have to wait and see.

Until then, we can make some educated guesses about the analysis they have done or are doing. First, the NEP analysts can examine several potential sources of error to see if they contributed to any systematic bias for Kerry. By examining actual vote returns they can identify errors in:

The random samples of precincts
The hard counts of turnout obtained by interviewers
Data entry or tabulation
Telephone surveys of absentee voters (in 13 states listed here)
Absentee voting not covered by the exit polls (in 37 states and DC)

While all of these factors could have introduced error, with the possible exception of absentee ballots, it is hard to imagine how any could have contributed to a systematic bias favoring Kerry. Moreover, these problems are relatively easy to identify once the full count is available. As such, I’m assuming that if any of these factors could explain the errors favoring Kerry, we would have heard about it already.

The second step is to look at the error that remains, something the analysts refer to as "within precinct error." The most likely culprit is some combination of response and coverage error. Response refers to randomly selected voters who did not participate in the survey; coverage refers to votes refers to voters who were not included in the sample because the exited the polls when the interviewer was away or did not pass the interviewer when the exited.

The good news is that the exit pollsters have more tools as their disposal to help study response and coverage error than other survey researchers. Since interviewers are face to face with potential respondents, they keep a tally on the gender, race and approximate age of refusals. Most important, the NEP analysts can calculate the difference between the poll and the actual count within each precinct. They know quite a bit about each precinct: how it voted, the type of voting equipment used at the polling place, the number of exit doors at the polling place, whether the polling place officials were cooperative and how far the interviewer had to stand from the exit. They also know the age, gender and level of experience of the interviewer at each precinct. They can use all of these characteristics and more to see if any tend to explain the error in Kerry’s favor.

The bad news is that it is difficult to say much about those voters that refused to be interviewed, because…well…they were not interviewed. If the NEP analysts are lucky, they will be able to draw some inferences from the characteristics of the precincts where the error was greatest, but otherwise, explanations may be elusive, even with all the data available.

Those of us not privy to the internal investigation are left to speculate about the most plausible theories. Here are some educated guesses, but keep in mind, this is just hypothesis:

Bush voters were more reluctant to be interviewed – As summarized in an earlier post, Republicans and conservatives have long reported less trust of the national media and, as such, may be slightly less likely to want to participate in the exit polls. The NEP interviewers and materials prominently display their big network sponsorship.

Kerry voters were more likely to volunteer to be interviewed – Exit polls have a potential "back door" that other surveys lack. On telephone surveys, a respondent cannot possibly volunteer to be interviewed. In an exit poll, the interviewer is supposed to pick every third or fifth exiting voter, but others may still approach and express an interest in being interviewed. Interviewers are instructed to deny such requests, but only training and diligence of the interviewers will prevent deviation from the sampling procedure.

This year’s NEP exit poll interviewers were trained via telephone and most worked for just one day without supervision. With roughly 50 interviews per precinct and a 50% response rate, it would only take an average of one non-random "volunteer" respondent favoring Kerry per precinct to create a 2% error.

Now consider: The Democratic National Committee waged an apparently successful campaign to get the 5 million members on its email lists to vote for Kerry in unscientific online polls. A CBS News report found that John Kerry ran "about 20 points better" in non-scientific online polls than in traditional, random sample surveys (thanks to alert reader BB for the link). Is it possible that some of this enthusiasm to respond to online surveys carried over to the Election Day and make some partisan Democrats more apt to volunteer to take the exit polls?

Bush voters were more likely to avoid exit pollsters who were forced to stand among electioneering partisans – Overzealous election officials who force exit pollsters to stand 100 feet or more from the polling place consistently present exit pollsters their biggest logistical challenge. At that distance the interviewers cannot cover all exiting voters, and worse, often get trapped standing among electioneering partisans – a gaggle most voters try to avoid.

Now consider that several news accounts suggest that Democratic campaigns and groups like ACT and Moveon.org put far greater emphasis on Election Day visibility than their Republican counterparts. Matt Bai’s post-election piece for the New York Times Sunday Magazine noted the puzzlement of Democratic organizers that their "field offices weren’t detecting any sign of Bush canvassers on the streets or at the polls." Is it possible that exit poll interviewers found themselves frequently standing among Democratic partisans that exiting Republican voters might want to avoid?

Again, all of this is just speculation, and definitive proof may be elusive even to those with access to the raw data. However, some combination of the above most likely caused the exit poll errors that favored Kerry.

By comparison, the alternative explanation for the exit poll "discrepancy" – widespread nationwide voter fraud – is wildly implausible. Consider the preliminary finding that Warren Mitofsky shared with blogger Chris Johnson:

One possibility [Mitofsky] was able to rule out, though, is touch screen voting machines that don’t leave any paper trail being used to defraud the election. To prove this, he broke down precincts based on the type of voting machine that was used and compared the voting returns from those precincts with his own exit polls. None of the precincts with touch screen computers that don’t leave paper trails, or any other type of machine for that matter, had vote returns that deviated from his exit poll numbers once the average 1.9% non-response bias was taken into account.

In other words, the size of the "discrepancy" between the exit polls and the vote did not vary by the type of voting equipment used at the precinct. Now if you believe there were problems in the count that were limited to a few counties or precincts, than the exit poll "discrepancy" has little relevance. Even if such problems occurred, the NEP exit polls lacked the statistical power to detect small errors within individual counties or precincts.

However, if you believe the error in the exit polls presents evidence of widespread fraud, you need to explain how such a fraud could have been committed consistently across all types of voting equipment and in all the battleground states. You would also have to reconcile that theory with New Hampshire, where the exit polls overstated John Kerry’s support by 5%, yet a Ralph Nader sponsored recount found no noteworthy discrepancies. A Nader spokesman concluded, "it looks like a pretty accurate count here in New Hampshire."

None of this makes much sense. The more plausible explanation is that a problem evident to some degree in the exit polls every year since 1990 – a problem most likely caused by some combination of response and coverage error — simply got worse.

[FAQ on Exit Polls]

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.