About those newly leaked exit poll numbers. Early this morning, Mickey Kaus linked to my earlier post the recently leaked exit poll documents and declared them a "smoking gun." When I first scanned the documents yesterday, I did not see the same bombshell. I still don’t but, let’s take a closer look.
First, let me clarify the source: Scoop, a left-of-center website in New Zealand that has been following the various vote fraud conspiracy theories, somehow obtained copies of the pdf reports that NEP provided for its national and regional exit polls. Since Scoop provided no state level data, it is likely that their source was a newspaper subscriber, not one of the NEP network partners. By the way, the "scoop" on the scoop numbers came from on New Year’s eve from the Blue Lemur, another "progressive" site.
The documents do tend to confirm what had been reported elsewhere (links to documents are in the bullet points that follow; all times are presumed to be EST):
- On Election Day, the "national" exit poll had Kerry ahead by three points (51% to 48%) at 3:59 PM and by the same margin (51% to 48%) at 7:33 PM when the polls were closing on the east coast. By 1:33 PM the following day, the completed, weighted-to-match-the-vote exit poll showed Bush leading (51% to 48%). These numbers had been previously reported by the Washington Post‘s Richard Morin and Steve Coll on November 3.
- The early samples included too many women: The percentage female fell from 58% at 3:59 PM to 54% at 7:33 PM, but this change alone did not alter the overall candidate standings (as the Simon/Baiman paper argues). By the next day, the sample was still 54% female, but the results among men and women were very different – Bush was 4 percentage points higher among men, 3 points higher among women.
- All of the three releases are marked as "weighted," but keep in mind: The first two releases were weighted only to bring their geographical distribution into line with hard counts of actual turnout. The last release would have been weighted so that it matched the official count (something I explained here).
- Keep in mind that the 7:33 PM sample from election night was incomplete. It had 11,027 interviews, but the next day NEP reported 13,660. The missing 2,633 interviews, presumably coming mostly from states in the Midwest and West, amounted to 19% of the complete sample (The Simon/Baiman paper includes what appears to be a later and more complete national survey report – more on that tomorrow).
The margin of error provided by NEP for the national exit poll was +/-1%. Thus, Kerry’s early lead and the overall differences between the 7:33 p.m. and 11/3 numbers were statistically significant. That the errors in the national poll were statistically significant, while similarly sized errors in state exit polls were not, owes to the much larger sample size of the national survey.
So is there anything truly new in these documents?
Perhaps not. At least not to me, and hopefully not to MP’s faithful daily readers. However, to political sophisticates like Kaus who do not share our odd obsession er…enthusiasm for the exit poll controversy, the official documents have more power than a few lines buried in an online chat. The last time most political junkies checked in on this story, it was a few days after the election and the "blame the bloggers" meme was in full force. You remember: The problem was not the exit polls or the way they were handled by the networks, but the foolish bloggers who blabbed about "early numbers" they did not understand. Well, these documents confirm something loyal MP readers have long known – the just before poll-closing numbers had the same errors.
Though my perspective is different, I am struck by how little guidance these cross-tabs provide about statistical significance. A newspaper subscriber like Richard Morin would have to do what I did above: Look up the sampling error provided by NEP in a separate table and apply it separately to each number. I cannot imagine that many editors or political writers went to that much trouble.
By comparison, I am told that NEP provided the network "decision desks" with printouts (or computer screens) that provided the exact confidence level for every vote estimate. That is, the estimate for each state included a percentage indicating the statistical certainty of the leading candidate winning the state. Networks would consider calling a state for a candidate only when that percentage went over 99.5% certainty. I had assumed the national poll reports provided to newspapers included similar reports of statistical significance. That they did not may explain why the newspapers that subscribed to the exit polls have been more willing to complain about the exit polls in public.
The continuing stonewall of secrecy that the networks have erected around the exit polls does not help. It is that secrecy, as much as anything else, that continues to fuel the more bizarre conspiracy theories floating around the blogosphere. I remain a skeptic of widespread fraud, but I cannot understand the continuing secrecy: Why did these documents have to be leaked by a left-wing web site in New Zealand? Why did NEP not release them in early November? Why did it take until late December for NEP to make the basic methodology statements the networks had on Election Day available online? And why so much reluctance to discuss, openly, what went wrong and why?
A bit more transparency from news organizations that trumpet our "right to know" would certainly help.
11/4 – Omitted formatting restored
Mark, are you aware of your own biases? They’re show rather clearly in places like this:
“the completed, weighted-to-match-the-vote exit poll”
They weren’t ‘weighted to match the vote’, they were *adulterated* to match the *claimed tallies*.
Absent empirical validation, the tallies represent only themselves. I really don’t think you can justify as good science your claim that they also represent ‘the vote’. Whether they represent the vote is the *issue*, and that’s not at all an unimportant point.
As always Mark, you have the scoop. I did not know that there was a difference in the tabs sent to the newspapers and the network subscribers. Useful info!
I do find it odd though that the national polls showed Kerry up 51-48 at 3:59, 7:33, and as confirmed by Morin to me, the final round (not sure if this was actually the 7:33 round though).
As indicated in Mitofsky and Edeleman (1995), for the 1992 elections, each round after 8pm showed a tighter margin (but still with a 1.6% bias toward Clinton). The text doesn’t seem to indicate that any of these rounds were weighted to the election tallies. It seems to attribute the discrepancy from successive earlier rounds to an incomplete sample.
One question… We know that these data are based on a sub-sample of the precinct samples. We also know that when these updates were phoned in, a count of the non-responses were phoned in. Do you think that the age, race, gender of the non-responses were also phoned in? Also, how do you think they accounted for the non-responses in their sub-sample? Did the sub-sample include the non-responses (e.g., if they took every nth survey, were blank surveys included so that it was equally likely that the non-responses were included in this sub-sample?)
Also, consider that the 2004 primary exit poll methods, conducted by Edison/Mitofsky, included the following methodology statement (as available via the Roper Center),
“The exit poll results are weighted to reflect the complexity of the sampling design. That is, the weighting takes into account the different probabilities of selecting a precinct and of selecting a voter within each precinct. For example, minority precincts that were selected at a higher rate receive a smaller weight than other precincts of the same size. Except for Iowa, there is also an adjustment for voters who were missed or refused to be interviewed, which is based on their observed age, race and sex.”
Do you think this “adjustment for voters who were missed or refused to be interviewed” was included in the leaked data (if of course this is how the survey was conducted this time)? Finally, if our “profiling” of voter intent of those missed or refused is incorrect, what does this say about potential for bias in an exit poll?
Many questions. Most simple methods questions. Why their secrecy? Getting information from these folks is like pulling teeth! But with all the poor statistical analysis of the flawed data currently in the public view, can you understand their reticence?
You’ve done some great work explaining the weaknesses of exit polls. Thanks. Two questions: One, since exit polls are essentially cluster samples at a precinct level, why couldn’t the raw data be used to point to potential fraud within a specific precinct when compared with tabluated results for that precinct? Two, do polls have any evidentiary weight in a court of law?
So is there going to be pressure on the networks to improve exit polling in 2006? so they actually get it right next time?
Warren Mitofsky has a history of opposing frank discussions of poll errors. I encountered his antipathy when I sent Everett Carl Ladd a draft of a meta-analysis of the 1996 presidential polls. I did so because Ladd had commented (in the Wall Street Journal) on the unidirectional character of the errors in those 1996 polls. I then subjected the data that Ladd had tabulated to what may have been the first meta-analysis of poll errors. It showed that the odds were 4,900 to 1 against the notion that these errors were the result of chance (see http://www.psych.purdue.edu/~codelab/PollOdds.html ).
Ladd was interested in publishing this contribution and so he circulated my draft among other members of the polling community. It was mentioned by John Fund in the Wall Street Journal and by Richard Morin in the Washington Post National Edition.
Mitofsky reacted with a campaign of intense vituperation and Ladd hesitated to proceed further. But, coming at the dawn of the web revolution, a revision has remained continuously available at the above listed URL and it has been regularly consulted ever since. And no subsequent election has exhibited egregious polling errors comparable to those of 1996.
At least one exit polling mystery solved. Why didn’t Mitofsky release the data? Go over and read Kaus. The findings have to be bad so Mitofsky has to be at fault. There can be no doubt so the poor son of a bitch is going to be flayed alive. Kaus (I’m sure on speed dial from John Ellis) has already started the process.
Mitofsky before release when he was saying that he knew there was a problem by mid-afternoon:
What a professional! Nobody better than Mitofsky! We have to trust Mitofsky!
Once we find out he didn’t realize an error by noon
Mitofsky is an idiot! Mitosky knows nothing about polling! Mitofsky is a cheap skate!
It sort of makes you sick. I hope MP sticks with his original feelings about Mitofsky.
One more thing. Stop saying conspiracy theories. That is what, we, as humans do, conspire with each other. That is our great talent as a species. If it weren’t for that Neandrathals would be in charge of the world (wait a minute – okay, strike that).
Gerald S. Wasserman, thanks for the source!
Since you say “And no subsequent election has exhibited egregious polling errors comparable to those of 1996” may I presume you have references to similar analysis of the 2000 and 2004 primary data? (what about other exit polls like the LA Times exit poll?)
It seems to me that even though exit poll data is available via the Roper Center, there would be lots of statistics grad students looking for a dissertation topic, eager to dive into it. There is really very little published on the subject, which I still find to be amazing considering the academic mantra – “publish or perrish.”
Blue22, I don’t know where, but I think MP has answered part of your first question (indirectly) about the individual precincts.
The polls are taken so that about 100 samples will be drawn throughout the day (set in advance, they try to approach every nth voter, but miss some and others refuse). The data leaked and linked to by MP, Kaus, and others includes a subsample of the precinct samples. (i.e., they only report about 50 of the 100 on election day). This seems to me to add error as it is a random sample of a random sample of a cluster sample (hence the design effect). Therefore, analysis of the full survey count per precinct should be more “accurate,” but remember, this is only 100 samples. An independent sample size of 100 assuming simple random sample has a margin of error of ~+/-10%. Also remember that not every precinct was sampled and therefore if you suspected a certain precinct of fraud, it may or may not have been in the sample.
Suppose though that someone identified a precinct in advance that could have fraud and decided to man that precinct with enough pollsters to approach a little more than 1000 voters, then you could get a MoE of about +/-3% for that precinct. Assuming no other non-sampling error, is +/-3% good enough to stand up in court?
Maybe a lawyer could answer this. But I don’t think we need a lawyer to tell us that +/-10% wouldn’t stand a chance.
Kaus on the Leaked Exit Data
UPDATE: Mystery Pollster is mildly suspicious of Mickey’s “smoking gun”…I was a bit confused as well, but MP offer’s this possible explanation…You mean all the high traffic bloggers, political pundits, and MSM journalists aren’t data geeks?
Rick Brady, thank you for the chance to clarify my remark. I should have said: “And no subsequent ‘national’ election has exhibited egregious polling errors comparable to those of 1996”. To be comparable, a set of national polls would have to be of comparable size; the work of 8 major polling organizations had been considered by Ladd in 1996. And their 8 errors would have to almost all go in one direction. And some of the errors would have to have been extremely large, near 10%. That has not happened since 1996; if it had, the spreadsheet I created would have allowed anyone to produce another meta-analysis quite readily.
Gerald, first question. Which Ladd 1996?
Ladd, E. S. 1996. The election polls: An American Waterloo. Chronicle of Higher Education, 22 November, p. A52.
Ladd, E. S. 1996. The turnout muddle. P.34 in America at the polls: 1996. Storrs, Conn.: Roper Center for Public Opinion Research.
Also, I’m still confused. How do you then justify this statement: “And no subsequent ‘national’ election has exhibited egregious polling errors comparable to those of 1996” if, as it appears, there has been no analysis of a national election?
The 2004 election saw many more GOP voters much more disrespectful of mainstream media than the 2000 election. You had the Rather debacle and the NY Times debacle, but also the steady growth of talk radio audiences and cable/Fox News. When they left the polling places and a “media” representative asked them how they voted, why wouldn’t they lie to screw him up? It’s not like all of them had to lie, but how about 6 in every 100? I was an avid Bush voter who was hoping to be accosted when leaving the polls so I could lie, but I realize I’m not typical… If Mitofsky people know their business, I think this must be the key to your exit poll puzzle.
MP, though I am not schooled in the intricacies of polling, I find you very informative. Thanks for sharing your knowledge with us in this forum.
The accuracy of the exit polls need not be argued in a vacuum.
The Mitofsky exit polls and the official tally are not the only data points that we have. The New Hampshire hand recount of largely Optiscan ballots was deemed accurate by all involved parties. NH had the largest exit poll discrepancy of any battleground state according to Freeman. Thus, one can conclude that physical ballots were stuffed in New Hamsphire or, more logically, that the exit polls were wrong.
Moving to Utah, the state with the twelfth largest discrepancy according to the Simon/Freeman data, the BYU exit polls agree with the official tally while the Mitofsky exit polls do not. (Kerry numbers: Official Tally 26.0%; Mitofsky 29.1% (both from MIT/Caltech Voting Project); BYU exit poll: 26.5% (from exitpoll.byu.edu). Thus, one can conclude that, in addition to the ballots being stuffed, the BYU exit polls were stuffed too, or one can conclude that the Mitofsky exit poll was wrong.
At this point, it should be patently obvious that the exit polls were wrong. Energy should be focused on analyzing why they are wrong to improve the process in the future rather than using the exit polls to allege fraud.
I did some quick math: if the 7:33 p.m. sample was correct and totalled 11,027 voters with a 51-48% Kerry lead, for the final results (with 13,660 voters) to show Bush leading 51-48%, Bush had to have won 64% of the votes in that last portion of the sample. Which on the face of it is monstrously out of line with both the actual election results and the previous exit polling. Would a pollster, the day after, “cook” his numbers so that they mirrored the actual vote percentages? If he wished to be rehired, at $10 million a pop, there’s a strong incentive for him to do so.
Hello, Mr. Blumenthal. I wonder why our government and press do not question exit polls in the Ukraine. Is exit polling more advanced there? Thank you.
Rick Brady, the information you want was given in: http://www.psych.purdue.edu/~codelab/PollOdds.html
Gerald S. Wasserman, thanks again, but it doesn’t answer all of my questions. Especially about how you justify the statement, “And no subsequent ‘national’ election has exhibited egregious polling errors comparable to those of 1996.”
One thing that I didn’t find in your paper is the exact date of each of those pre-election polls that you cite. If any of those polls were conducted more than a day or two before the election, I’d say their predictive value is about worthless. Polls are supposed to be a snapshot in time and any number of things could have changed the minds of voters in the last few days before an election.
Also, not in there is information about the assumptions used to assign undecided voters (if they were assigned).
These are important considerations I would like to think.
I’m really astonished by all the implicit ‘nothing to see, move along’ that’s going on here.
Would someone like to try making a case that it’s good science to silently ‘correct’ the exit polls with numbers from the tally?
How about a case for the accuracy of the tallies themselves?
Who benefits when people are given the ‘move along’ treatment when the stakes are high and unanswered questions abound? Who benefits from the obfuscation and damping-off of the issues?
Shouldn’t we expect impartiality and accurate representation of the facts from those who claim the mantle of science?
Mairead, “Would someone like to try making a case that it’s good science to silently ‘correct’ the exit polls with numbers from the tally?”
I guess the case is that: 1) it has always been done (that’s what I hear anyway); and 2) it goes to show that the pollsters are (and presumably have been) more confident in the election tally than they are of their polls.
Someone who used to comment on this blog described it like this to me in an e-mail: The election tally is “considered” to be like an atomic clock (infallibly accurate) and one can compare the exit polls to a VERY expensive Rolex Watch.
We notice that the Rolex Watch is way off from the atomic clock. In the past, the makers of the Rolex Watch simply reset it to the atomic clock and blamed the error on external forces that act on the watch and cannot be predicted or accounted for by the watch maker. “After all, the Rolex is no atomic clock” they say.
This time, the difference is rather large; much larger than normal.
Some say that this suggests that someone fudged the atomic clock when no one was looking (Freeman, Simon/Baiman). Others say the Rolex Watch was likely tinkered with to trick the atomic clock (Dick Morris/Jim Geharty). Still others seem to think that the Watch is fine; this time, as times in past, it was subjected to a force outside the control of the watchmaker (Mitofsky/Lenski).
I say, perhaps it is a combination of all three? Who knows?
So now the question is: Do we look at the Rolex Watch to determine that someone tinkered with the atomic clock?
Mairead, let me correct myself. According to Mitofsky and Edeleman’s (1995) review of the 1992 exit polls, the final exit was weighted to the election result. I can only guess that it was like this before and it will be like this in the future.
Larry,
Its good to lie to the exit pollster, that way no one will believe Bush was elected legitimately, instead they will believe the election was fixed.
This is all very interesting, but essentially useless until all of the raw data with the precinct identities is released. My own conjecture is that the nonresponse was asymmetrical by a factor larger than the models (whatever they are) predicted. If the pollsters did their jobs correctly, this should be detectable by the age/gender/race of the non-responders. I suspect that the NEP knows what the problem is, but they may never want to admit that the polls were wrong.
Yancey, re: “I suspect that the NEP knows what the problem is, but they may never want to admit that the polls were wrong.”
If you read MPs previous posts, you’ll find statements attributed to Mitofsky and Lenski that admit that the polls were wrong.
Also, I suspect many of the skeptics (not conspiracy theorists) will not have faith (heh) in the raw precinct data when it is finally at the Roper Center. Mark my words.
Rick, you seem to persistently respond to me in a way that’s not really a response.
I asked whether anyone wanted to try a defence based on scientific principles.
You gave me back one based on bureaucratic/tribal ones (‘it’s always been done that way’; ‘what was good enough for our ancestors is good enough for us’).
—————————-
I’ll ask again (of everyone, not of you, Rick): anyone want to try to justify the ‘correction’ practice based on scientific principles? Does anyone even think it’s *possible* to so justify it?
If it’s not possible to justify it, what does that suggest about the motives of the ‘scientific’ pollsters like Mitofsky et al.?
Mairead, the scientific case as to why the tallys are more believable than exit polls is simple.
A) widespread fraud with no whistle blowers is hard
to believe.
B) quite frankly the exit polls have such a large non-response that i believe it is unrealistic to expect the exit polls to be correct.
But that leaves a nagging question…why do they have no plans and fixing the non-response problem with the exit polls? this whole thing stinks.
Shutting my mouth…I can hear the crickets chirping…
Rick,
Which part of my post do you disagree with? I thought I was saying essentially the same thing as you?
Brian, I’m not sure ‘widespread’ would be the word I’d choose, tho technically it might be accurate.
How many fraudulent votes would it have taken in Ohio? An average of 12 per precinct, something like that? As Sjerp Zeldenrust remarked the other day: a few fraudulent absentee ballots, a few unfairly dumped provisionals, and a few counterfeit ballots replacing real ones and Bob’s your uncle. And there need be no conspiracy within the meaning of the Act.
Although I personally strongly suspect fraud because of the existing evidence, my scientific soul would be happy if people like Mark would simply play it straight and not use misleadingly soothing terminology. (I should probably declare my interest: I supported and voted for Dennis in the primaries, and for Cobb in the general. Kerry is not and would never be ‘my’ candidate. But democracy is.)
And I do completely agree with you that the whole thing stinks. From the poor-quality execution (compared to, e.g., Germany), through the silent adulteration of the results and unwillingness to fully disclose, to the widespread ‘nothing to see here’…it pongs like a two-week corpse.
Brian Dudley, I was responding to Mairead, because he brought up a good point. It seems I’m posting too much. I’m just a data geek who can’t seem to find anyone else who is interested in these things. I’ve got a bunch of e-mail lately from readers of my blog who say “Who cares about exit polls!” Well… I do… It’s my blog…
So – I abuse MP’s blog because it’s the best forum out here for me to hash out my thoughts. I should calm my tone at times, or “sleep it off” before posting, but by the time I realize I should have not posted something, the deed is done.
I’m a flawed man, I know, but hey…
I’m not allowed to post on DU (don’t meet their standards), but it would be fun to mix it up with the likes of TIA (who appears to have been allowed back). Maybe they will allow a crunchy conservative like myself to participate? Who knows, perhaps I should send an e-mail to the moderators and ask.
Mark may not involve himself in the comment section debates often, but I know he reads this stuff. From time to time, he even takes up a question in a post. I know he is busy and would then probably like to spend his energies posting, rather than responding to comments. Maybe also he just wishes some of us will just get frustrated by the silence and buzz off?
Perhaps we should keep directing our questions to Mark, hope he answers, but if not and if others want to jump on in, that could be okay as well? It’s only another opinion. Others can ignore it, reject it, challenge it, or embrace it. It’s up to them. That’s my two cents.
BTW – Mitofsky is not a jerk in my opinion, just perturbed. He tried his best to answer a few of my questions.
I had to read and re-read Kaus’ post to find out what the big deal was. To me, I thought the big deal was that it was confirmation that Kerry was up by 3% as of the 7:33 round. Were there subsequent rounds? I don’t really know. The whole gender focus was a bit odd though. I like how MP handled it – “not everyone is a nerd like us and have been following this story closely” (my paraphrase with artistic license)
For Rick Brady: Because most of the calls made by pollsters are not answered on the first attempt, polls necessarily involve a trade-off. A poll conducted over a short time frame will contain a bias towards the subgroup of respondents who are more likely to answer the phone on the first try while a poll that keeps calling back until more of the respondents have finally been contacted will naturally take more time. A poll that is both unbiased and last minute is like a unicorn — something that simply does not exist. Poll apologists have often exploited this dilemma by invoking a hypothetical last-minute surge to account for their errors. But such an apologia would only be plausible to someone who is unfamiliar with the technical realities.
So the 8 “Final Polls” tabulated by Ladd back in 1996 were not conducted on the evening before Election Day and neither he nor I ever said they were. He just called them “Final Polls”. I guess he assumed the meaning of the phrase would be apparent to those skilled in the art. The timing of 7 of these polls was recorded on the day after Election Day in an AP dispatch filed at 7:36 PM by Mike Mokrzycki entitled “A Look At Pre-election Polls”. It was what one would expect: these polls were conducted over a period of several days before the election with each pollster making a different choice of days. Similarly, each pollster naturally made a different judgment about the allocation of the undecided respondents. Each pollster also made numerous other technical judgments, any of which would provide a fertile field for post-hoc cherry picking. Not my cup of tea.
The analysis issue you keep raising is similar to the timing issue in that it also involves a fundamental characteristic that cannot be evaded: Statistical inferences involve a comparison of an observed effect with an error estimate. Most of what is called the analysis involves the calculation of the error estimate while the effect is usually obvious from mere observation. Thus Ladd observed a polling error effect, which he called to our attention. An inconclusive discussion followed which involved Frank Newport of Gallup and Humphrey Taylor of the Harris Poll. The discussion was inconclusive because it relied on the sampling error of a single poll to assess the collective error of a group of polls. My contribution lay in my recognition that this problem was ripe for the meta-analysis I provided in http://www.psych.purdue.edu/~codelab/PollOdds.html
Since that time, I have watched the final results reported by national pollsters. I have not observed a national polling failure that is comparable to the one that occurred in 1996. In the absence of a nontrivial observed effect, there is simply no point in doing the rest of the analysis: No observed effect means there is either no statistically significant effect or possibly, that there is only an effect that is so small that it is trivial.
Perhaps you think I have missed a comparable catastrophe. If so, just point it out. I will be happy to run the meta-analysis on those data. But frankly, after all this back and forth, I doubt that you will be able to produce such a case or you would probably have done so already.
Rick writes:
“Mark may not involve himself in the comment section debates often, but I know he reads this stuff. From time to time, he even takes up a question in a post. I know he is busy and would then probably like to spend his energies posting, rather than responding to comments. Maybe also he just wishes some of us will just get frustrated by the silence and buzz off?”
“Perhaps we should keep directing our questions to Mark, hope he answers, but if not and if others want to jump on in, that could be okay as well? It’s only another opinion. Others can ignore it, reject it, challenge it, or embrace it. It’s up to them. That’s my two cents.”
Just so you all know, yes, I do read this stuff, or at least I try to, though I may get to it a day or so after you post it. It all depends on my schedule.
Yes, as Rick suggests, time is the main reason I tend to avoid diving in to the debates in the comments section. I try to channel my energies into posting on the main blog, which I assume gets read by more folks.
But I want all of you to know that I very much value the questions — and even the criticism. Obviously, I work without an editor, and your comments help keep me on my toes and honest.
A suggestion (something that I’ll try to post on the main page): If you *really* want to get my attention with a question that you *really* want me to tackle on the main page, email me. If you’d like to let the other commenters take a shot at it first, post it here.
Either way, your contribution helps. I only wish I had more time to comment and post.
Mairead, there are over 10,000 precints in Ohio, even if you only needed 12 fraudulent votes per precint, this would have to happen in ALL 10,000 precints. How could this happen without a massive conspiracy? Since the exit polls were also wrong in states like SC and NH, it seems more reasonable that the exit polls were wrong, but that’s just me.
As for the weighting, as I understand it, first the polls are weighted to account for clustering. These are the exit polls that show Kerry winning the national vote. Then there is a further weighting to account for the actual vote (or the vote count as reported by all the precints if you want to remove bias). One reason for this is to make the results from the questionnaire more meaningful. Obviously if the reported vote count is not correct, this weighting is inaccurate. I don’t think this last weighting is done to hide anything.
Rick, I wouldn’t feel too bad about not being able to post at DU. I’m a Democrat and Kerry supporter and I find it tough to slog through all the conspiracy theories that get floated. Although it would be nice to get a few more rational people to respond to TIA.
Gerald S. Wasserman, thanks for the detailed comments! Time my friend…time… It’s not that I “can’t”; I don’t know! I haven’t even thought to look, I was just looking at your approach.
When I see analysis of a comparison of polls without dates and other information that “could” affect the outcome, I think an analysis is hiding something.
Or not hiding something; perhaps they are ignorant of how these things “could” affect the comparative analysis. This appears to be the case of a Real Clear Politics comparison of this year’s pre-election polls. They impugned the reputation of one polling organization for missing it “big time” in one state, but the poll they used to compare to the election result, although the final poll for that company in the state, was conducted 2 weeks before the election! To me that is REALLY stretching; especially, when an organization’s reputation is on the line. I would be livid if I ran the org.
After my experience with Tom Bevin and John McIntyre of RCP, I have learned to be a bit more skeptical of these things.
Mike Tocci responds: “Mairead, there are over 10,000 precints in Ohio, even if you only needed 12 fraudulent votes per precint, this would have to happen in ALL 10,000 precints. How could this happen without a massive conspiracy?”
erm, I did use the word ‘average’ in there, Mike, and appropriately, too. It was my way of pointing out that, spread across all opportunities for fraud, the mean amount of fraud needed per opportunity was small. I was not trying to suggest that there was a per-precinct quota!
Certainly if there *had* been such a quota, then that *would* have required an organised conspiracy, probably including a program manager, meetings, and progress reports.
But no such quota would have been required. People without scruples need not be told to act unscrupulously. All they need is a wink and a nod, and they go ahead. They work out how much they can do, and in what way, and they do it. Maybe all they do is leave the door unlocked so the ballots can be got at, relying on one or more of their ideological fellow-travellers to actually swap in the counterfeits. (Probably not 12 counterfeits, either.) Or they get *really* creative, in a stupid way, and claim that there’s an FBI warning that requires them to count behind closed doors. Or they pre-punch a couple dozen ballots such that a Kerry vote auto-invalidates the ballot. There are lots of ways to do the job.
The late Yitzhak Rabin, alav hashalom, was murdered at the behest of Likud. But the Likud leadership didn’t have to actually *conspire* with the killer. All they needed to do was talk endlessly about how he was betraying Israel. They knew some fellow-traveller would step forward and do the job. So what if he went down the plughole afterward? He’d have served their purpose.
As to NH showing the same effects as, e.g., Ohio–all that would mean is that the fraudsters couldn’t make it work. They guessed too low about how many they needed to ‘fix’–they missed it by 1.5%: Kerry still won by 1.3%. Note that the ‘correction’ was still enormously in Bush’s favor (9.5%).
I understand it’s on average, but as you remove precints from the list the average starts going up. How many precincts do you think were in on this? 100? 1000? Remember, this isn’t Florida in 2000, you need to find 120,000 votes for Kerry. The problems in Warren County, a few open doors, and a few dozen pre-punched ballots aren’t going to add up to 120,000 without a massive conspiracy.
I’m not sure what the Rabin comment has to do with this. Are you saying that there are Republicans all over Ohio that commited fraud on their own without direct orders? Actually you seem to be saying this happened all over the country? Was there similar fraud in every state that didn’t match the exit poll? This is what drives me to say that Kerry just lost, the conspiracy would otherwise be too massive with almost no evidence. I guess this can mean the conspiracy leaders are really good, or a simpler explanation is it didn’t happen.
Hi Mark
You are missing the smoking gun! It’s in the final corrected exit polls. The ones you have listed.
http://www.cnn.com/ELECTION/2004/pages/results/states/US/P/00/epolls.0.html
You missed the only question that tells if the poll are corrected correctly. There is only one question that you need to look at. What I am going to point out CLEARLY shows that CNN is putting out propaganda and not news. This is a situation in which the information that is put up is obviously unarguably misleading/wrong, and they must know that it is wrong. The pollers that collected it know it is wrong!
We all know that the early exit polls showed something different than the final outcome. I am NOT talking about the early exit polls these are the final/”corrected” exit polls. These are the numbers put up to dispel the “problems” with the early numbers and the ones they have used to tell us what we think. They are OBVIOUSLY misleading/wrong (see web page below). The PROOF is about 2/3’s maybe 3/4’s of the way down, the question is who did you vote for in 2000. The problem is obvious 37% Gore 43% Bush in 2000 that is WRONG. Again the people that did the polling know that’s not right, the people that posted it know it’s not right. They didn’t do any fancy weighting to adjust for that, as I’ll show below. It’s that simple. It’s more obvious than the type setting on the font of an old document. Anyone shown it should know it is BS. The only thing I wonder about is how that data got in the stats at all. Maybe someone on the inside is trying to say something?? I don’t know?
Do you understand how weighted averages work?? The sum of the fraction times the amount
Numbers from the data (see again see web poll data below go to the question who did you vote for in 2000). The 51/49% Bush win is what we are being told. If you plug in the straight numbers the data comes out as they what you to believe. This works with other number too so that shows no other factor are included:
.17*45+.37*10+.43*91+.03*21=51.11% Bush
.17*54+.37*90+.43*9+.03*71=48.48% Kerry
With equal amounts of Gore and Bush AS WE KNOW?? IT WAS.
.17*54+.40*90+.40*9+.03*71=50.91% Kerry
.17*45+.40*10+.40*91+.03*21=48.68% Bush
Again, any one who puts out a poll that doesn’t have equal amounts of Gore to Bush has to know the data collection is faulty, and that any information gleamed from it is also wrong. Also if 17% of the people that didn’t vote last time went 54% Kerry and 45% Bush and that is in a Bush favoring sample. The switch Bush to Kerry and Gore to Bush are about equal, looking at that there is really no way to say that Bush won.
Again check it for yourself, and think a bit.
http://www.cnn.com/ELECTION/2004/pages/results/states/US/P/00/epolls.0.html
Mike Tocci writes: “I’m not sure what the Rabin comment has to do with this. Are you saying that there are Republicans all over Ohio that commited fraud on their own without direct orders?”
No, what I’m saying is that no direct orders would have been needed. Maybe a few key people did get them, or got more than a wink and a nod. But the majority? No, not any more than people have to be told to do their job in an office.
The Daley machine in Chicago was internationally notorious for fraud, voting the graveyard being favorite. How many people do you suppose got direct orders? Maybe five would be my guess: you don’t get that far in the organisation if you need your (unofficial) job spelled out. “The Democrats are gonna steal the election!” Implication: we gotta thwart them by doing things that will counteract their plans. And if that means skullduggery, well, we’re on the side of God, so it’s okay. Serves those godless commies right!
When Coulter can talk about *exterminating* lefties and people nod their heads, do you really think those same people would shrink from a little ballot tampering? I don’t.
Coulter is a…okay…I’ll be nice…terribly misguided person in need of lots of help? How was that?
Rolf:
You seem to making a common mistake. I emailed Steven Freeman because he made the same mistake. The 43% for Bush and 37% for Gore are the numbers of voters that claimed to have voted for those candidates in 2000, not actual numbers. Why more people claimed to vote for Bush in 2000 than really did and fewer claimed to vote for Gore in 2000 than really did is a mystery I do not know the answer to.
There could be a skew on that set of questions or some respondents lied to the pollster. There are psychological assessments as to why people will sometimes claim to vote when they did not or claiming not to have voted instead of accurately stating they voted for a candidate they did not embrace.
Rolf is right, that voted for Bush (43), voted for Kerry (37), with the plurality of people who didn’t vote going for Kerry is a wowser and a half. I would really like to see MP address that.
Mairead wrote:
“Would someone like to try making a case that it’s good science to silently ‘correct’ the exit polls with numbers from the tally? How about a case for the accuracy of the tallies themselves? Who benefits when people are given the ‘move along’ treatment when the stakes are high and unanswered questions abound?”
Mairead:
I think your argument comes from not fully understanding the purpose of Exit Polls as the NEP ran them. Exit polls are concerned with getting both the “True Vote Count” AND “True Voter Profiles.”
But, I agree with you that NEP’s “silent” correction of exit polls is unjustified. Let them explain their methods and purposes. The NEP’s secrecy does not serve the public. As to who benefits from this obfuscation, I don’t know, but that is why I consider blogs, data geeks, and MP in particular heros. They aren’t afraid to investigate for the good of the public.
Back to what I think is the misunderstanding. What you are calling for is an Auditing function for the exit poll, an INDEPENDENT measurement of the “True Vote Count.” And, of course, the correction would play no part in gettting at the “True Vote Count” independently.
The corrections, however, are useful in trying to get at the “True Voter Profile,” another goal of exit polls.
I would argue the method has validity for that purpose, but less validity when the discrepancy is as great as it was this year.
Because the vote tally is a second, independently calculated data point estimating the “True Vote Count” it can be used to adjust the exit poll vote count and in so doing adjust the exit poll voter profile data to better match the “True Voter Profile.”
The reason this adjustment makes sense is because the vote tally is a “good” data point on the “True Vote Count;” obviously, compared to the exit poll, coverage is near 100% and response rate is near 100%. Coverage and response rate are the two main shortcomings in exit polls. So absent a certain threshold of systematic flaws in the vote tally, of course it is more accurate than the exit poll.
So, while the correction in no way gets closer to the “True Vote Count,” assuming both vote tally and exit poll were not systematically flawed (read election fraud or exit poll bias), this adjusted voter profile is more accurate than exit polls alone at giving the “True Voter Profile.” Granted, this may not interest you, and this I think is the jist of the misunderstanding.
Of course, compared to Germany, from what MP has explained, it appears BOTH exit polls and vote tallies in the US are woefully lacking. Election Fraud is practically an american tradition (Jim Crow laws, Kennedy’s Chicago Machine fraud, to name two historical instances). We don’t have professional election commissions to help mitigate the fraud, instead we have partisan election officials that arguably can be expected to in some circumstance engage in quasi-legal fraud. Also, exit polls are not treated as official election functions (denied reasonable access at some polls, lack of government support in providing voter demographics as Germany provides, workers not highly trained or respected by exiting voters, etc).
Since both exit polls and vote tallies can be considered somewhat suspect in the US, at what point does a discrepancy between the two become so great as to call into question both and thus invalidate adjustments of exit polls based upon vote tallies? Again this correction validity question only matters to those who are interested in the “True Voter Profile.”
Personally, I am really disappointed with American election systems, and I hope the technocrat class exerts their power in forcing an upgrade to a modern national voting system.
To put it another way:
Is it clear that exit polls are free of systematic bias and thus better than the vote tallies at getting at the “True Vote Count?” I think sadly, No. There is enough to suspect that exit poll bias is great enough that it cannot be trusted to give a good estimate of the “True Vote Count.”
So no matter the level of election fraud, the exit poll can’t be trusted either.
How do we get trustworthy exit polls that could be useful for Audit purposes? It seems the answer is do what the Germans are doing. I think it would be telling to find out whether the main goal of German exit polls is an audit function or voter profile function. I wouldn’t be surprised if it were the former, as that would be a greater motivator to get them right. And it seems the German’s have got them right.
Why are the final exit polls prior to adding in the election results not being considered? (Perhaps I missed something reading through here?)
http://media.washingtonpost.com/wp-srv/politics/elections/2004/graphics/exitPolls.html
I love Richard Wells answer to Rolf’s question because it shows how historians are going to have such a field day with this. The argument when Bush has lower exit poll numbers than actual tally is that people didn’t want to admit they were voting for Bush, or people voting for Bush weren’t responding to exit pollsters. But IN THE SAME POLL when there seem to be too many people who said they voted for Bush in 2000 it’s because people said they claimed they voted for him when they didn’t. This takes rationalization of data to meet political purposes to a whole new level.
Wilbur: The data relies on what people say, not what they do. So both conditions are possible at the same time. Without spending time determining why given numbers are generated, calculations based on the numbers can generate even stranger results.
I would not be surprised if some claimed to have voted for Bush in 2000 but really did not amongst the group of Kerry supporters this year. There seem to have been a number of people that claimed in pre-election editorials to have suddenly galvanised into a rejection of Bush (sometimes inconsistent with a person showing up at progressive fund raisers for a decade); I would expect that some might try to create an impression of a repudiation of Bush by his base in responding to exit polling.
That is my theory based on the exit polls. Sufficient people made inaccurate statement that when processed through the algorithms generated a total picture which was incorrect. This may happen in any election where a large portion of electorate is voting against a specific candidate and I doubt any exit polling technique can compensate for it.
Hi
Yes we could come up with some reasons although highly unlikely that might explain why 43% Bush 37% Gore in this case. When I say highly unlikely that is not my judgment that would be statistically highly unlikely. That is a 6% gap in a 13,660 sample. Lets think about it. If you know that the make up of a given town is 50% women and 50% men. Someone does a poll of 13,660 people and tries to tell you that it is 54% men and 46% you have to know that is wrong, there is something wrong with the poll and in that case it should be adjusted or discounted. Anyone out there want to crunch the numbers on the probability of getting a 54/46 outcome in that sample size. Now it could be they forgot which sex they are or they are afraid to admit what sex they are….. But all those arguments are just used to try to get you away from my point.
If a poll I tells you BUSH won in 2000 52% (BUSH) to 45%(GORE). There is really only one thing the data is telling you and that is, it wrong and the sample is bad. Please tell me you know that is wrong!!!! If that information which is obviously wrong is then posted and used to further a government position then that is propaganda my friend. And that is my point.
Yes it is scary and I will say no more about it.
Hi
Yes we could come up with some reasons although highly unlikely that might explain why 43% Bush 37% Gore in this case. When I say highly unlikely that is not my judgment that would be statistically highly unlikely. That is a 6% gap in a 13,660 sample. Lets think about it. If you know that the make up of a given town is 50% women and 50% men. Someone does a poll of 13,660 people and tries to tell you that it is 54% men and 46% you have to know that is wrong, there is something wrong with the poll and in that case it should be adjusted or discounted. Anyone out there want to crunch the numbers on the probability of getting a 54/46 outcome in that sample size. Now it could be they forgot which sex they are or they are afraid to admit what sex they are….. But all those arguments are just used to try to get you away from my point.
If a poll I tells you BUSH won in 2000 52% (BUSH) to 45%(GORE). There is really only one thing the data is telling you and that is, it wrong and the sample is bad. Please tell me you know that is wrong!!!! If that information which is obviously wrong is then posted and used to further a government position then that is propaganda my friend. And that is my point.
Yes it is scary and I will say no more about it.
Rolf: The data is bad. I believe it is because a number of respondents incorrectly informed the exit pollster as to what they did in 2000. Does not matter. A number of fraud proofs use those numbers of how voters in 2000 voted in 2004 to prove that Kerry won. But if the data being used in incorrect, the calculations on that data are also incorrect.
Rolf:
The Gore/Bush exit poll question and result may point to problems with the correction method.
But wouldn’t we also need to know what was the uncorrected Gore/Bush exit poll results?
This reminds me of a question I asked MP a while ago. How did the exit polls do in predicting all the other races it polled? State offices, proposition, etc.? This would greatly enhance our analysis of the exit poll discrepancy. This question has fallen out of the discussion, probably because we don’t have the data. Hopefully we will look at it at some point.
Mairead:
Please note that the “adulteration” process is troubling only in that the unadultered data is not available. Mark has called for the unadultered data to be released, so he understands your point.
Mark is not claiming that reweighting exit poll data to match the vote tally in any way proves the vote tally. The claim is that reweighting improves the voter profile.
Here’s is Mark’s explanation on this point:
http://www.mysterypollster.com/main/2004/11/the_difference_.html
widespread fraud with no whistle blowers is hard
OK, I don’t want to sound like a conspiracy theorist. I believe the error was much more likely to be in the exit polls than in the counts, for all the reasons Mark has given.
However, I do want to correct this persistent misperception. Because of the nature of many of the voting systems (both the balloting and the tabulation systems) there does not need to have been a widespread conspiracy to cause major vote fraud across multiple states.
It is technically possible for vote fraud to be introduced at the point of distribution for the software, and to be done in a way that is both indetectable after the fact and that requires collaboration with a very small number of people (possibly as few as one).
Now, it is true that each state has a unique mix of types of voting machines. So, if one type of machine were “fixed” this would not affect all states equally, and may leave some unaffected.
If/when the detailed final precinct-level exit poll data are made available there may be an opportunity to correlate the predictive accruacy of the exit poll data with the type of machine used. I say “may” because there may not be enough precincts of each major type of voting machine to provide a statistically significant sample.
I’m aware that some people have already conducted some studies along these lines, such as the one comparing the county-level results in Florida for screen vs. optical ballots. The problem with that was that the different machine manufacturers were grouped together, and the comparsion was not with exit polls but with census data and results of previous elections.
Richard,
Do you understand how many cognitive steps there are in what you are describing.
1) I don’t like Bush
leads to
2) I want people to think that Bush is being repudiated,
leads to
3) Bush will be repudiated if
a) He loses the election
b) it seems fewer people actually voted for Bush in 2000
leads to
4) I can accomplish both a (by voting for Kerry) and b (by claiming I voted for Gore even though I actually voted for Bush) leads to
5) I will lie and say I voted for Gore even though I voted for Kerry.
This is incredibly complicated and must be done in a split second and must be done by a number of people across the country on the same day without any prior discussion. The odds of something like this happening would, I believe, approach infinity (I know infinity is not a number but a concept – a little poetic license here.)
Wilbur: Not a planned action. Just a natural psychological tendency that could shift a few responses to the exit pollster.
To use a sports analogy, do you think as many people would have claimed trading for Kevin Brown was a good move for the New York Yankees at the end of the season as at the beginning? Some who supported that trade at the beginning of last year would call into sports radio and various forums declaring that they never liked the trade. People sometimes claim to have better foreknowledge than they really possess based on later outcomes.
I see three groups of possible incorrect respondents:
Those who voted for Bush in this election but did not vote 4 years ago. Embarassed about not voting, they claim to have voted for Bush 4 years ago.
Those who voted for Gore 4 years ago but didn’t like him. They don’t want to indicate they supported a failed candidate they did not like and respond to the questioner that they did not vote 4 years ago.
The group you seem not to believe could exist. But if people are willing to falsely claim to be Bush supporters that turned to Kerry in OpEd pieces, why can’t they do the same to an exit pollster?
If 10% of the respondents incorrectly responded to a question, that would produce the errors the exit polling saw. There may be other reasons for it but I seriously doubt that the real reason was an attempt to rewrite the 2000 election into a landslide for Bush. But it is still simply my pet theory, not holy writ.
I feel a need to protect baseball fans, even the ones that call into the radio. I actually have a stake in this because I actually drafted Brown in my fantasy baseball league (a good decision that became a bad decision). No baseball fan I knew would deny having been in favor of the Brown trade because Brown had a bad year. In retrospect they would say that the Brown trade was stupid and shouldn’t have been done, but if you cornered them even the most intense would admit they were in favor of the trade.
There are those who would admit that they were in favor of obtaining Brown and that it is still a good idea (the same people who were in favor of going to Iraq and still think it’s a good idea). These people would answer the pollsters correctly.
There are those who were not in favor of obtaining Brown and shout it from the rooftops.
There are people who wanted to obtain Brown but feel like lying about it better serves the purposes of making the Yankees a better team (perhaps it will help remove Cashman, the general manager). But as I outlined above this is preceded by a complex cognitive based narrative and is not a decision that would be made on the spur of the moment when suddenly being quesitoned by the pollster.
There are people who wanted to obtain Brown but their disappointment in how Brown behaved causes some form of psychosis that convinces them that they were actually against Brown. I’m not saying there aren’t baseball fans that are this psychotic, but they are few and far between (and they probably have other issues in their lives).
In short there is a difference between regret, lying for purpose, and some sort of psychosis. The Op-Eds you are talking about reflected regret, not psychosis.
If you take out psychosis there is no viable explanation for the exit poll findings.
Thanks for the evidence Rolf. We have lots of liberals saying “let’s ask people how they voted” and you have given us proof of exactly what will happen if we do–because lo and behold here we have an “exit poll” of the 2000 elections and what do we find out? Only .37/(.37+.45)=45% of actual 2004 voters voted for Gore in 2000, despite fraudulent claims he “won” 50%. Call the FBI! We have hard evidence of an unbelievable +5% voter “blue shift” on a national scale in 2000! Gore somehow took a 45% showing and made it 50%!
The facts are simply this about the 43% said they voted for Bush in 2000 and 37% voted for Gore in 2000:
People lie.
Any reputable poll that asked that question this year came up with very similar numbers. Heck, even go read a Democracy Corps (not exactly non-partisan, but fair) poll a month before the election. They showed that on average 5-7% of poll respondents said they voted for Bush in 2000 than Gore.
In 1974, when Nixon was in the middle of the Watergate scandal, how many people said they had voted for him in 1972? Not many.
It’s a simple answer to a rather stupid conspriacy theory.