Have the Exit Polls Been Wrong Before?

I have a short backlog of posts on the exit polls I’ve been working on this week, intended mostly to summarize information I’ve covered previously and make it more accessible via the FAQ. However, there is new information here, as well as in the posts that will follow.

One of the odd bits of received wisdom I keep hearing about the exit poll controversy is that up until this year, the exit polls were “always right.” If so then this year’s errors seem “implausible,” and wild conspiracy theories of a widespread fraud in the count somehow seem more credible. The problem with this reasoning is that exit polls similarly “wrong” before, though perhaps not to the same degree or consistency.

Here is the documentation on previous errors. First, from the Washington Post’s Richard Morin:

The networks’ 1992 national exit poll overstated Democrat Bill Clinton’s advantage by 2.5 percentage points, about the same as the Kerry skew

Warren Mitofsky, who ran the 2004 exit poll operation along with partner Joe Lenski, wrote the following in the Spring 2003 issue of Public Opinion Quarterly (p. 51):

An inspection of within-precinct error in the exit poll for senate and governor races in 1990, 1994 and 1998 shows an understatement of the Democratic candidate for 20 percent of the 180 polls in that time period and an overstatement 38 percent of the time…the most likely source of this error is differential non-response rates for Democrats and Republicans:

From the internal CNN report on the network’s performance on Election Night 2000 (p. 48 of pdf):

Warren Mitofsky and Joe Lenski, heads of the CNN/CBS Decision Team, told us in our January 26 interview with them that in VNS’s use of exit polls on Election Day 2000, the exit polls overstated the Gore vote in 22 states and overstated the Bush vote in 9 states. In the other 19 states, the polls matched actual results. There was a similar Democratic candidate overstatement in 1996 and a larger one in 1992.

In short, Mitofsky and Lenski have reported Democratic overstatements to some degree in every election since 1990. Moreover, all of Lenski and Mitofsky’s statements were on the record long before Election Day 2004.

Of course, those errors were apparently bigger and more consistent this year. According to an internal NEP report leaked to the New York Times, this year’s “surveys had the biggest partisan skew since at least 1988, the earliest election the report tracked.” However, in some states, the errors in 2000 were still quite large. Consider this comment from Joe Lenski to CNN on December 12, 2000 (p. 48 of pdf), describing the table also copied below: 

The second group contains five states that had stupendously bad exit poll estimates. Here is a comparison of the final best survey estimate at poll closing with the final actual results for these five states… As you can see the exit polls in these five states were off by between 7 and 16(!!!) [Emphasis in original]

The exit poll errors four years ago led Mitofsky to tell the CNN investigators, “The exit poll is a blunt instrument,” and Lenski to add, “the polls are getting less accurate” (p. 26 of pdf). They recommended “raising the bar” on projections made from exit polls: “The proposed changes result from a belief that exit polling is “less accurate than it was before” and that “we should take exit poll data with caution in making calls,” said Lenski” (p. 27).

All of this led the authors of the internal CNN report — Joan Konner, James Risser, and Ben Wattenberg – to conclude (p. 3, 7):

Exit polling is extremely valuable as a source of post-election information about the electorate. But it has lost much of the value it had for projecting election results in close elections…[Their recommendation to CNN:] Cease the use of exit polling to project or call winners of states. The 2000 election demonstrates the faults and dangers in exit polling. Even if exit polling is made more accurate, it will never be as accurate as a properly conducted actual vote count.

[FAQ on Exit Polls]