Hello Exit Polls My Old Friend

Exit Polls Legacy blog posts

The release of several new papers on the 2004 exit poll controversy brings me back to this familiar topic.  The first paper, from a team of academics with considerable survey expertise, breaks no new ground but provides a good overall summary of the controversy.  The second, by frequent Mystery Pollster commenter Rick Brady, goes further, taking on those whose widely circulated Internet postings proclaim evidence of fraud in the exit polls.  The academic paper is an excellent overall primer on the issue, but Brady’s work breaks new ground. 

The first, a "working paper" released on March 11 by the Social Science Research Council (SSRC) of the National Research Commission on Elections and Voting, is noteworthy for the expertise of its authors.  Michael Traugott of the University of Michigan, Benjamin Highton of the University of California (Davis) and Henry Brady of the University of California (Berkeley) are political scientists with scores of journal articles on voting behavior and survey methodology to their names.  Traugott, the principal author, is a past president of the American Association for Public Opinion Research (AAPOR) and the co-author or editor (with Paul Lavrakas) of several books on survey methodology, including the very accessible Voters Guide to Election Polls (which MP would include on a list of recommended books, if he ever got around to putting such a list together.  Full disclosure:  twenty years ago, Traugott was the third reader on MP’s undergraduate honors thesis).

In short, these guys know what they are talking about. 

Yet for all the academic firepower that Traugott and his colleagues bring to the exit poll debate, they break little new ground.  They do present a balanced and thorough summary of the short history of the controversy and its key issues and include the most complete bibliography on the issue (including URL links) MP has seen to date.   If nothing else, the Traugott paper is an excellent starting point for anyone grappling with this issue for the first time

Traugott and his colleagues also make a very important point about the key issue that continues to frustrate those seeking an "explanation" from the exit pollsters for the discrepancy between the exit polls and the final results:  When it comes to "nonrespondents" — those who refuse to participate in a survey — "proof" is inherently elusive.  In reviewing the report from the National Election Pool (NEP) released earlier this year, they write:

[The report] is complicated in a way that many post-survey evaluations are by the fact that some information is essentially unknowable. This is especially true when one of the concerns is nonresponse, and there is no information from the nonrespondents to analyze. As a result, there are some sections of the report in which there is an extremely detailed level of disclosure about what the exit poll data show, but in other parts of the report there are only hypotheses about what might have been the cause for a particular observation. These hypotheses can guide future experiments in exit polling methodology or even direct changes in the methods, but they cannot explain in a strict causal sense what happened in the 2004 data collection (emphasis added, pp. 8-9).

A second paper, posted over the weekend by our friend Rick Brady of the blog Stones Cry Out, is a point-for-point rebuttal of the final version of Stephen Freeman’s well known paper, The Unexplained Exit Poll Discrepancy (MP reviewed the first version of the paper back in November).  Brady, who has been studying graduate level Statistics on the way to a Master’s Degree in Public Planning, assails every statistical weakness in Freeman’s thesis.  Many of the issues that Brady raises will be familiar to MP’s readers, but he does an excellent job putting it all together and raising some statistical issues not included in the Traugott paper. 

For MP, the most interesting aspect of Brady’s review is his discussion of a subsequent paper by a team of PhDs (including Freeman) affiliated with the organization US Count Votes.  Kathy Dopp, the President of US Count Votes (USCV), issued a public challenge "for any PhD level credentialled (sic) statistician who is affiliated with any university in America to find any statements in our ‘Response to Edison/Mitofsky Report’ that they believe are incorrect and publicly refute it."

Brady may be just a Master’s Degree candidate, but he steps up to the challenge, essentially picking up where the Traugott paper leaves off.  He observes:

The US Count Votes authors conclude that only one of two hypotheses are worthy of exploration: 1) the exit polls were subject to a consistent bias of unknown origin; or 2) the official vote count was corrupted. The question then becomes; did the NEP Report substantiate the first hypothesis? [p. 12]

Reviewing the NEP report, Brady concludes:

Given the number of NEP Report conclusions that included qualifiers such as "likely," "may," and "could," I understand how US Count Votes is concerned with the analysis. In effect, the NEP Report never (from what I can tell) rejected the null hypothesis in a classical sense. However, the contention that "[no] data in the report supports the hypothesis that Kerry voters were more likely than Bush voters to cooperate with pollsters" is not in the least bit accurate. The NEP Report presented volumes of information that most analysts agree "suggests" support for the hypothesis that differential non-response was the cause of the observed bias in the exit polls [pp. 13-14, emphasis added].

The paper has much more.  Brady has been a loyal FOMP (Friend of Mystery Pollster), so I may be accused of some bias on this score.   Yet I hope other prominent observers will agree:  Brady’s paper is a must read for those still genuinely weighing the arguments on the exit poll controversy. 

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.