Exit Polls: The NEP Report

A story by Jim Rutenberg in today’s New York Times finally reports the explanation of the exit poll mishap from the people who ran it, Warren Mitofsky and Joe Lenski of Mitofsky International & Edison Research. Rutenberg "obtained" a post-election report written by Mitofsky and Lenski for the subscribers of the exit poll conducted by the National Election Pool (NEP): the major networks and newspapers like the New York Times.

The Rutenberg piece is worth reading in full, but before doing so, consider again the different deliverables that NEP provided on Tuesday. First are the partial results reported in the middle of the afternoon and widely leaked on the Internet that showed Kerry doing a few points better in nearly every state poll than he did in the final result. Second are the numbers delivered for each state just before the polls closed, data that are typically weighted to match the day’s actual turnout and that facilitate election projections. Finally, NEP provides final data well after the polls close that they weight by the vote to match actual results. These data are not used for projections but for analysis and subgroup comparisons, so weighting by the actual vote helps make the final poll more accurate.

One question I have been asking here is whether the second deliverable – the data delivered for each state as the polls closed – had the same Kerry bias as the early reports. The Rutenberg piece implies that it did:

According to the report, the surveys had the biggest partisan skew since at least 1988, the earliest election the report tracked.

"We share all the members’ concerns about the inaccuracies in the projections produced by the early waves of exit poll data and we are personally miffed about the early results,” the report said.

Rutenberg goes on to confirm that the skew in the national survey persisted until the end of the day:

"The last wave of national exit polls we received, along with many other subscribers, showed Kerry winning the popular vote by 51 percent to 48 percent, if true, surely enough to carry the Electoral College,” Steve Coll, managing editor of The Washington Post, wrote in an online chat with readers Wednesday [emphasis added].

In an interview yesterday, Mr. Coll said his newspaper had to scramble to make last-minute changes to an article analyzing why voters voted the way they did that was based in part on the poll data when it was clear that no such victory for Mr. Kerry was possible.

"We think it wasn’t worth what we paid for it, that’s for sure,” Mr. Coll said of the survey data.

What reasons did the NEP officials offer for the skew?

In some cases, the [NEP] report said, survey takers could not get close enough to the polls to collect adequate samples of voters opinion. They were often stopped by legal barriers devised to keep people electioneering – not necessarily bona fide poll canvassers – away from voters.

The report also theorized that the poll results more frequently overstated support for Mr. Kerry than for President Bush because the Democratic nominee’s supporters were more open to pollsters.

Keep in mind that the exit pollsters can compare their results with reality all the way down to the precinct level. Speculation by NEP officials about a systematic non-response bias favoring Kerry is significant, because they are certainly in a position to confirm any such bias. Note also that Rutenberg’s story makes no mention of the earlier speculation that early voting by Democrats caused a mid-day skew that faded later in the day. That theory would explain a temporary problem observed only in mid-day samples, but not a bias that persisted until the polls closed.

So armed with this new information, let me try to review some of the questions I have received from readers:

Q: Are we SURE that we can eliminate the possibility that some kind of electoral fraud tipped votes from the Democratic to Republican columns? Is it possible that the exit poll was right and the vote count was wrong?

The Rutenberg article says the NEP report "debunked" the emerging conspiracy theories that "the exit polls are right and the vote count is wrong." The article did not elaborate. However, since 2000, the exit pollsters have tracked the type of voting equipment used at their sampled precincts. If the discrepancies could be explained, as some suggest, by precincts using the newer Diebold touch-screen voting machines, the exit pollsters could prove it. With their own reputations on the line, the NEP officials report no such evidence.

Q: Was there evidence of "a deliberate act of fraud and bias" on the part of exit pollsters, as Dick Morris implies, or that Democrats "had an election-day project of slamming the results" as Mike Barone argues? 

It is hard to take either suggestion seriously. The point of fraud by the exit pollsters would be what exactly? Putting their reputations at risk by deceiving their clients and the world just six hours before the real results are available? It makes no sense. And while I can imagine a campaign operative emailing fictional exit poll numbers to a blogger (something I suggested in point five of my "what you should know" post), the idea of a widespread campaign to manipulate the exit polls with pseudo-respondents is ludicrous. Again, what is the point? Were than many marginal voters really sitting by the computer deciding whether to vote on the basis of leaked exit polls? If Democrats were capable of pulling off such a massive, secret conspiracy to pad the exit polls by a point or two in every state, why not just put the same effort into mobilizing real voters?

Q: "Did the early exit polls showing Kerry ahead almost across the board actually spur pro-Bush voters to head to the polls?"

Sorry Mickey, but I doubt it. To know for sure, it would help to know how many voters were even aware on Election Day that Kerry led in early exit polls. I would be stunned if the number is greater than a few million, and virtually all were hard-core political junkies already certain to vote. The blogosphere has great reach, but not that great.

Q: How can we be sure that the post-election analysis from the exit polls is accurate?

This is the most important question of all. Since the final results are weighted by the actual vote returns (both at the precinct level and then overall) the results are certainly more accurate than those leaked on Election Day, and arguably more accurate than most political surveys. They are an unmatched resource, and I remain an exit poll fan despite the apparent glitch on Tuesday.

However, the fact that well educated and well-informed consumers of exit polls are asking this question should be a grave concern to those of us who value survey research. By ignoring the reality that partial exit poll would appear on the Internet, and doing nothing to help consumers data put that data into some context, we have seriously undermined public confidence in exit polls and surveys generally. We can do better.