Warren Mitofsky, one of the principal directors of the National Election Pool (NEP) exit poll, appeared last night on the Lehrer News Hour. The full transcript is not yet ready, but you can listen to the entire interview via real audio at this link.
Mitofsky is arguably the best source for information on this issue, and while the interview does not answer every question, it at least provides his explanation without any filter. It is worth listening to in full.
The money quote (my transcription):
Kerry was ahead in a number of the states in margins that looked unreasonable to us, and we suspect that the main reason was that the Kerry voters were more anxious to participate in our exit polls than the Bush voters. That wasn’t the case in every state. We had a few states that overstated the Republican margin, but for the most part it was Democratic overstatement for the reason I just gave you.
What Mitofsky is suggesting is extraordinary: If true, it is the first time in his exit polls and one of the first times I can remember for any political survey, that those who refused to participate in the survey were different enough (more supportive of George Bush) than those who were interviewed to make a material difference in the overall results.
If it proves true it raises an important question: Did similar non-response bias also occur on recent telephone surveys? More on this next week…
UPDATE: The full transcript of Warren Mitofsky’s interview is now online at this link.
Now that it’s clear it wasn’t early-polling bias, but was still there in the final exit polls: we *need* the data from the final exit polls. NEP suggests it was responder bias, and it wasn’t fraud, but they are just guessing. They have no data, or at least they present no data. And I doubt they, or any major media people, would have the cajones to suggest publicly that there was major fraud or even to look seriously into the possibility. Independent people need to be able to systematically compare the final exit polls to the reported poll results to see what patterns show up, if any. This needs to be in the public domain.
How can we get the final exit poll numbers released?
Wait a minute – when we look at the pre-election polls didn’t Bush supporters consistently register as more enthusiastic and committed to their candidate than Kerry supporters? How does that translate into Bush supporters being significantly less likely to want to be polled upon exiting the voting place. It just doesn’t make sense. Any thoughts?
The NEP report claims to have “debunked theories that the exit polls are right and the vote count is wrong.”
It is astounding they can come to this conclusion without providing a shred of evidence. What they’ve provided thus far amounts to sheer guesses as to the reasons for discrepancy, not a bit of proof. And since of course, they monitor exit polls, not voting machines, their firm position that the errors occurred in the exit polls would seem absurd.
From reading this and listening to the News Hour piece, they seem to be grasping at straws to try to explain away the situation. The only debunking they’ve managed is the theory that the incorrect exit polls were due to an early Democratic surge. They’ve now confirmed that the combined, end of day, final polls still had Kerry very much in the lead.
Some of their explanations for the mismatch are hard to seriously consider. Republicans, en mass, avoided exit pollers more than Democrats? Democratic districts, en mass, allowed exit pollers more access to voters than Republican districts? For that to be true, this trend would have to have occurred in over 16 states.
Before they throw out such explanations, they need at least some bit of proof. They need to do some polling, of their own exit poll workers. Poll them to see how close each were allowed to the polling places. Knock on doors around those districts, see if Republican voters really avoided the pollsters…
They certainly haven’t had the time to do so yet. It’s very possible they may no longer even have the budget to properly investigate this. But until they do, their explanations will remain absurd.
Explaining this massive discrepancy away with wild guesses is horribly irresponsible. And until a proper investigation occurs, nothing, not even massive voting fraud, can be so blithe discounted.
This makes perfect sense to me. I do research on survey data, mostly educational survey data, and non-participation by respondents is a huge problem. We constantly have to try to control for the loss of data.
We also have to manage respondent bias. For example, teachers often “want” to participate in surveys and we need to be careful not to oversample those teacher because their desire is an example of selection bias, and may influence the outcome.
So imagine you are in a state that is a battleground (like me). And there are a bunch of cars parked near the polling place with “Defoliate Bush” bumper stickers, and a few Moveon.org volunteers hanging around by the door. Or maybe there are some irritating GOP poll watchers, angering the Kerry voters and being obnoxious.
So, when you come out, are you likely to broadcast to anyone who can hear that you voted for Bush? I think there are lots of people who voted for him quite reluctantly and didn’t want to discuss it. No fraud.
There’s a reason that voting is a private activity–so you don’t have to tell people who you voted for. Apparently Bush voters embraced that in this election.
dr. cookes, I’ll concede that your suggestion may have held true in a few locations, but nationwide? And I certainly don’t buy the suggestion that Democratic supporters caused any more reluctance in voters talking with exit pollers than did Republican supporters. Supporters form both sides were very much in evidence at polling places in all the battleground states.
That aside, the real problem I have with the NEP explanation is that they provide absolutely no factual basis for their claim.
It may have happened they way they suggest. Perhaps there was a 16 state-wide trend of embarrassed and reluctant Bush voters. But the NEP lose a tremendous amount credibility by suggesting it as the only possible explanation, and without providing a shred of proof to back the claim.
Okay, I admit it’s a stretch.
What if the answer is all of the above? The NEP screwed up a little because its survey researchers didn’t get close enough. Bush voters were reluctant. And so on.
I’m in the middle of analyzing a huge dataset covering four years. Respondents completed the same survey several dozen times per year. The data looks different in Year One than in Year Four, and I’m starting to wonder if the respondents “learned” how to fill out the survey. That is, the act of completing the survey began to change the way they thought about it and reacted to it.
There is no way I can know for certain if this happened. But I am fairly certain it could happen and would affect my data.
Now what if voters were more plugged into the politics of this election, they were more blogged and more skeptical and more aware than in past elections. (I mean, the world is wired in ways it wasn’t a decade ago.) Could it be that voters have “learned” how to manage surveys? To answer or not answer, depending on their desired outcome? To get a proper sample, you have to assume that your sample does not have any intentions to see a particular outcome of the survey results. But what if they do?
Sade, I’m in a blue state and voted red. I’m in my 50s, female, not easily intimidated I deliberately did not have signs in my yard or stickers on my car. Why? Because there were numerous, authenticated stories about actual violence against e.g. GOP headquarters and of harassment of voters waiting in line for early polling in places like FL. I can hold my own, but life is too short to have to go get my car repaired because some mindless idiot trashed it, you know?
The left has been indulging itself for several years in vicious, mindless hate talk. It was fanned by the far left and many Europeans and it was *not* denounced by Democratic candidates or spokespersons. It is one reason that I, a moderate Dem for over 3 decades, voted for Bush/Cheney this year — a party that tacitly endorses such things does not deserve power, especially during a time of major geopolitical change.
I would not at all be surprised if the exit polls tapped people like me and they either lied or refused to answer.
I watched that newshour segment. Warren Mitofsky defended his poll and honestly didn’t seem to believe that the fault laid with it. He didn’t look like he really believed the “Kerry voters were more anxious to participate” excuse. My impression was he thought the exit polls were largely correct but, realistically knew he couldn’t come out and say that.
Mitofsky’s assertion is beyond “extraordinary”; it is absolutely absurd. We are supposed to believe that Kerry voters were sufficiently more “anxious” to speak to exit pollsters than were Bush voters to explain, even partially, a swing from +3 Kerry to +3 Bush? I think not. We are asked to believe that Kerry voters, disproportionally more urban and minority than Bush voters, and who on average waited considerably longer in line to vote, were so dramatically more willing to take the additional time to cooperate with a total stranger and fill out the exit poll form? Ask anyone who has ever done telephone sales or solicitations where they get more hangups from … “blue” urban areas or “red” rural and small-town areas? Does anyone really believe that a typical “middle American” is more likely to refuse to cooperate with an exit polltaker than your average New York City resident?
Obviously this was not the first time exit polling was conducted in a presidential election. If there was any evidence from prior experience to substantiate the assertion that an alleged greater willingness on the part of Democratic voters to fill out an exit poll could remotely explain a 6% swing between exit poll results and the “actual” results, people involved in conducting and analyzing those previous exit polls would be out in force making that claim. Quite tellingly, they are not!
Mitofsky’s assertion is beyond “extraordinary”; it is absolutely absurd. We are supposed to believe that Kerry voters were sufficiently more “anxious” to speak to exit pollsters than were Bush voters to explain, even partially, a swing from +3 Kerry to +3 Bush? I think not. We are asked to believe that Kerry voters, disproportionally more urban and minority than Bush voters, and who on average waited considerably longer in line to vote, were so dramatically more willing to take the additional time to cooperate with a total stranger and fill out the exit poll form? Ask anyone who has ever done telephone sales or solicitations where they get more hangups from … “blue” urban areas or “red” rural and small-town areas? Does anyone really believe that a typical “middle American” is more likely to refuse to cooperate with an exit polltaker than your average New York City resident?
Obviously this was not the first time exit polling was conducted in a presidential election. If there was any evidence from prior experience to substantiate the assertion that an alleged greater willingness on the part of Democratic voters to fill out an exit poll could remotely explain a 6% swing between exit poll results and the “actual” results, people involved in conducting and analyzing those previous exit polls would be out in force making that claim. Quite tellingly, they are not!
Aren’t exit polls given in the form of a written survey, handed back to the pollster immediatly?
It’s hard to see how people would be intimidated or fearful of reprisals from the other party’s supporters in this situation.
In particular, it is real hard to see the stereotypical gun toting redneck Republican fearing reprisals from any “pinko hippy” democrats who might be hanging around the parking lot.
I don’t see an easy explaination for this.
In the early 1990s I had some friends with access to the data call me with early exit poll results. Before the Internet and Drudge, this was the only way I could get them! Anyway, I noticed that results were invariably stronger for the challenger or the insurgent, depending on the race, than the final result. I concluded that supporters of these candidates were likelier to talk with exit pollsters than supporters of incumbents. So at least on an anecdotal level, Mitofsky’s explanation rings true to me.
Has anyone studied the bar graphs that Randi Rhodes discussed on her show yesterday (Air America) that seem to show remarkably more consistency between exit polls and final results in states using paper ballots than in states using electronic voting? I’d be interested to read a professional opinion about validity etc. http://www.therandirhodesshow.com/todays_show.html
If some people take the unadjusted exit poll results at face value; the adjustment makes it look like the Bush people stole the election by cancelling four points of the total vote from Latino Kerry supporters, in Florida and Texas. Why they would do this in Texas, where they were in a strong lead regardless, is difficult to explain. It would make sense, though, for the exit poll adjusters to compensate for oversampling of minority and Latino voters in particular, by subtracting the necessary points from the Latino Kerry supporters in texas and Florida. It makes them look less egregiously off base, so long as no one notices the legerdemain.
The Bush people probably DID steal the election, and you can find reporting by Thom Hartmann today on commondreams.org that explains how. A Congressional candidate in Florida claims to have evidence and also that it wasn’t the first time these database manipulation tactics were used.
http://dark1p.blogspot.com
Sigh.
To all you “the exit polls were right, Kerry wuz robbed” people:
“Cannon’s analysis doesn’t jibe with Slate’s exit-poll numbers. A comparison to the latest vote tallies shows Slate’s final exit numbers in the paper-trail states of New Hampshire (undervalued Bush by 5 percent) and Nevada (undervalued Bush by 3 percent) were less accurate than those in Ohio (2 percent off), Florida (3 percent), and New Mexico (2 percent). The other state Cannon lists, Illinois, won’t require a paper trail until 2006.” http://slate.msn.com/id/2109141/
I don’t know why the exit polls were wrong but they *were* wrong, and I don’t find it implausible that conservatives, having been told for years on talk radio, etc. that the “mainstream media” are biased against them (whether this perception is accurate or not is not the question here) might have been less likely than liberals to want to participate in the polls conducted by said media.
BTW, I am a Democrat. I am not despondent over Tuesday’s results. After all, all we need to do next time, after a three percent defeat in 2004, is to persuade about 1.5 percent of the voters to change their minds. It’s not impossible. Harder things have been done. But we will never do it if we persuade ourselves we don’t *have* to because we won anyway,and all we have to do is get rid of Diebold…
The Slate articles argues against eVoting fraud as opposed to paper ballots fraud. OK. And yes it might be plausible that GOP voters were under polled out of nonresponse. But note that Mystery Pollster calls this unprecedented in the history of exit polling.
And there are a few other possibilities:
Missed GOP turnout
Missed Latino GOP turnout
Vote spoilage: by definition a factor
Provisional Votes: by definition a factor
A more complete response by Exit Polling professionals is possible, and it is not unreasonable for us to ask in public.
I agree with those that suggest that problems in the exit polls are due to incorrect self-identification by Bush voters. I have believed for some time that *all* the polls (not just the “day-of” exit polls) underestimated Bush’s strength by 1-2 percent due to this “stealth vote”.
It’s all very well for the lefties here to pooh-pooh the idea: it’s very real. And–frankly–suggesting that it’s cowardly for people to be reticent about their voting preferences in an environment where the oppo acts like–well–MoveOn.org and ACT is…exactly the kind of behavior that makes otherwise impeccably liberal voters decide for Bush in the first place.
The exit poll sampling is done at 1,480 precincts, a probability sample based on turnout, demographics, and partisan makeup from past elections. The objective is to have a representative sample of precincts. Voters are randomly selected to be given a survey (every so many at a set number interval), but participation is voluntary. The exit poll accounts for absentee and early voting through telephone sampling in advance of the election.
This is a very sound methodology, but each step has some validity threats. For the sampling, people change and precincts change. Either due to migration (think Florida) or due to turnout changes that cause the makeup of the precinct to change in relation to historical patterns. For both the on-site survey and the telephone survey, there is a question of response bias, that is do the folks that respond differ in important respects from those that do not. There is also the possible interaction effect to be accounted for. Does the act of asking someone to participate in the poll cause some of them to respond differently.
I don’t know how I would handle the absentee/early voting any other way, but having two survey methods combined in a single sample complicates matters further. For example, Harris Interactive runs both an Internet and traditional telephone poll, but never combines their results. Same for other pollsters that are fielding multiple methods. However, the requirements of the exit poll require that these two survey methods be integrated into a single report.
I am not a public opinion specialist myself, but do sometimes use public opinion data and I am a social scientist who conducts quantitative research (also in education, Dr. Cookie). I find the exit poll data useful, but I personally don’t vest the data with the kind of authority others are doing. I think that the more unusual the election, the less well the exit polls will perform. And this was an unusual election in terms of intensity and turnout. I also think that it is risky when surveys are used for something other than their intended purpose. The election projection model is to use actual results to confirm the exit polls, not vice versa.
Can someone explain how the exit polls have ended up consistently wrong for the past three election cycles? I recall when they were deadly accurate down to the last percentage point.
The Michael Moore types claim that the exit polls show the real result, and Kerry really won. Oh, really? Okay, but these same exit polls were showing states like North and South Carolina “too close to call”. I don’t think anyone could credibly argue those states were ever close.
Even Kerry’s own pollster said his own internal numbers were showing Bush would win with 51% of the vote. Pew was showing the exact same thing.
I think the real problem with exit polls is they are so reliant on urban polling, where GOP strength continues to erode, while it continues to grow in the rural areas.
Some are suggesting that innate privacy, embarrassment, or reluctance caused an exit poll under-vote in conservative voters. I’ve never seen any evidence of that happening in any other exit polls. And I find it very hard to believe that Bush voters, in huge and disproportionate numbers, refused to participant in exit polls or lied on them.
This because all the exit polls I’ve seen are just as anonymous as the vote that immediately proceeds them. Like standard voting, the exit pollster hands the voter a ballot. The voter fills out the exit poll ballot and drops it in a closed box. By many measures, exit polls are a lot more anonymous than phone based polls. And while there is certainly some percentage of the population not willing to participant in any polls. I have never seen a precedent that this segment is higher in voters who have conservative leanings.
This theory suggests is that voters who cared enough to show up at the polls, voters who certainly weren’t reluctant or embarrassed or too private to actually vote, that these same voters refused or lied on another anonymous ballot just minutes or seconds after voting. And, that these voters making this decision were disproportionately Bush voters. Even if one is to believe this convoluted path led to a Bush undervote, there is no precedent or evidence to back the claim.
The larger problem with this theory is that it assumes the flaw was in the exit polls. This of course is a massive assumption, because we have no evidence supporting the claim.
I suspect this assumption is made because many simply cannot consider the other possibility. They believe that election fraud is something that just doesn’t happen in America. Well, with the current awful state of electronic voting machines, machines with no audit trails which run very insecure software, the possibilities for massive fraud certainly exist. And at this point in time, there is hardly less evidence supporting a fraud theory as any other theory.
Sure, the exit polls may have been flawed, there could have been something that caused them to miscount the actual vote on a conservative bias. But even assuming the flaw was in the exit polls themselves, the “Reluctant Bush Voter” theory is but one of many possible ways the exit polls could have been troubled. They could have been set up in wrong districts, the tabulation could have been incorrect.
I think this fast push to suggest this very specific flaw in the exit polls is just a quick and easy way to explain away this massive discrepancy. And a way to do so without having to consider a very troubling, alternate possibility.
Did Republicans steal the Florida election by rigging optical-scan voting machines?
Web pages are appearing alleging a massive, coordinated pattern of election fraud in Florida (for example, see Voting Fraud in the 2004 Election). What’s fueled these allegations is that, in certain Florida counties with large numbers of registered Dem…
There IS a precedent for an exit poll getting thrown off by the fact that one party’s voters were less enthusiastic about talking to interviewers: the 1992 British general election, in which there was an eight percentage point gap between the predictions based on exit polls (showing a Labour win) and the final voting tally (a decisive Conservative win). This is the worst known failure of a modern exit poll. See Crewe, Ivor (1993). “A nation of liars? Opinion polls and the 1992 election, Journal of the Market Research Society 35 (October, 1993).
Bryan, I wasn’t aware of that precedent, still, that was the very rarest of exceptions. As you quote
“The worst known failure of a modern exit poll”.
And so far, we’ve seen no evidence supporting theories that something similar happened in our recent polls. Or even that the exit polls were where the failure occurred. Blind assumptions that the failure must have occurred in the exit polls is the point I have the largest problem with.
Because, as you point out, the precedent would suggest massive exit poll failure to be a very low probability event. I believe some other possible reasons for the mismatch, all of them unlikely, need to be given serious consideration.
Mitofsky’s suggestion that eager Kerry voters influenced the exit poll seems unlikely. During the pre-election period, many people were making the opposite argument, that polls such as Gallup were oversampling Bush supporters because they were more willing to talk to pollsters. Why would these Bush people suddenly become shy on Election Day?
Hi TCS,
You wrote, “The election projection model is to use actual results to confirm the exit polls, not vice versa.”
As I’m sure you’re aware, the vote fraud detection model is to use the exit poll to validate the reported vote count.
Quite a paradox isn’t it?
1. Analyzed the Final Exit Poll and its methodology
2. If unexplained discrepancy, analyze Reported Vote counts and vote count methodology
1. Final Exit Poll Analysis
Of course, what most of us are asking for is professional debate/analysis on exit poll bias factors:
Missed GOP turnout
Missed Latino GOP turnout
Vote spoilage
Provisional Votes
Nonrespondent bias
2. If necessary, analyze Reported Results
Once a rigorous Final Exit Poll analysis is completed, if any discrepancy remains unexplained, a serious review of the vote count process is in order. Of course, it remains to be seen if election and poll professionals care about this sort of question enough to follow through.
I hate to bring it up, but I don’t think a national debate on the validity of the 2000 Florida election was completed at the time. This seems telling.
Essential Reading: this post from the Mystery Pollster
I’ve been literally living on two sites, the Mystery Pollster, Mark Blumenthal, one of the most clear-headed folks I’ve seen on this issue, and the Electoral Vote Predictor, a data-cruncher’s wet dream set up and run by Andrew Tanenbaum. Both of these …
TCS did make a point I overlooked. Exit Polls are not necessarily designed for Vote Fraud detection. Thus, they are unlikely to prove conclusive in this regard. Fair enough.
This brings up an intersting question. I wonder if the US would consider designing transparent Vote Fraud detection into the system. While I’m dreaming, I think we need to design away vote spoilage, dummy proof voting if you will. And move voting to Saturday.
Ahhh, what could be.
when do you assume Mitofsky’s suggestion as fact. He was trying to explain why exit polls didn’t match “final vote count”.
could it possibly be–hold on to your hats–FRAUD?
why not consider, based on evidence, fraud, which is much more probable than an educated, but still baseless guess?
But even if it is fraud – which it may be – do we not run the risk of endangering the Iraq elections?
If America is seen to be incapable of running its own elections–how can we administer to teh world.
A fair criticism of this viewpoint would be that four more years of Bush is worse than a failed Iraqi election. Further, that the Democratic party may tear itself to pieces trying to “restructure”, losing its own base while trying to appeal to fundamentalist Christian conservatives.
But considering things on a macro scale – is it not better to push for a hegemonous voting system run by a trusted independent organisation–perhaps Carter’s trust, or if this is seen as too Democratic, a group like fact check .org? They could run everything, have full control of voting stations, machines and so forth, and make sure that each voter has a numbered receipt (to avoid duplication) and a receipt is also dropped into a ballot box in case of system failure.
Perhaps going to war on this issue will pose problems for the next fifty years in world relations. Bush’s policies have already been largely carried through on a GLOBAL scale (the Iraq war) – all this energy spent on proving fraud may be better spent creating an airtight system so that certain Nebraska senators and Florida governors are never again re-elected.
It’s so difficult to prove fraud in a GOP dominated senate, house, supreme court, gubernatorial system and presidency – far easier to create an unimpeachable, and more importantly–unbreachable system to prevent another election Troy.
I wouldn;t be surprised about fraud – many of the GOP were brought up in the Nixon whitehouse. George HW Bush was allegedly involved in voter fraud and the Iran Contra scandal. Those who ridicule the suggestion should pay attention to history – fraud is a big deal in US politics. As a final example, consider the life-tenure of Hoover.
If we can’t prove it…we should invent a system to negate it–and leave it in the hands of the unimpeachable.
The theory, and it is a theory, that Republicans were more shy about revealing who they voted for than Democrats because of fear or lack of trust or whatever, can also be said of Democrats ~
I was a Kerry supporter but didn’t dare put a sign on my lawn ~ nor did I even discuss politics with friends or relatives who were Bush supporters for fear of being vilified, which happened once and was enough for me.
My girlfriend in WV who supported Kerry did put signs outside her house and had them taken away several times. Another friend, and Kerry supporter, also refrained from putting signs outside her house or on her car because of reports of ‘keying’ of Kerry supporters’ cars in her neighborhood and I could go on…
So, would all the fear that we Kerry supporters had, (btw, I have been called a ‘traitor’, ‘anti-American’ etc. simply because I have expressed disagreement with this president’s policies), not also cause Democrats to be reluctant to talk to exit pollers? Personally, I was afraid to talk to anyone outside the polling area and did not.
Mitofsky’s opinion is not information that is based on any credible data and is mere guesswork, it may or may not be true ~ there’s no way to verify it, so imo, it should be discarded and more provable evidence considered ~
Also, should we consider the political leanings of Mitofsky if we are to accept ‘theory’? Does anyone know whether he is a supporter of President Bush or not?
You must know that official US exit poll data was leaked on 12/31/4 & is available at:
scoop.co.nz/stories/pdfs/Mitofsky4zonedata/
Please pass it on.
You must know that official US exit poll data was leaked on 12/31/4 & is available at:
scoop.co.nz/stories/pdfs/Mitofsky4zonedata/
Please pass it on.