The “Smoking Gun” Part II

Exit Polls Legacy blog posts

Today’s must read for polling junkies is undoubtedly the exchange between Warren Mitofsky and Mickey Kaus (on the "smoking gun" item I blogged on yesterday). I cannot do it justice with a quick excerpt — you should definitely read it all — but here is the text of Mitofsky’s email reply to Kaus:

The so-called smoking gun you wrote about was in the hands of every subscriber to our national election poll throughout election day. What took you and the others two months to locate it? At least a dozen news organizations have had this smoking gun since 11/2.

Second, the complex displays you ridicule, which were the source used by the leakers for the numbers that got posted by bloggers on election day, are not the tables you and others discovered. I stand by my original statement. Had you asked me I would have told you as much.

Third, if you doubt that we warned the NEP members on election day why don’t you ask one of them? Or is ridicule with your eyes closed your preferred method of sounding smart?

And lastly, if my clients were as misinformed as you seem to think how come none of them announced an incorrect winner from the 120 races we covered that day? It seems that the only ones confused were the leakers and the bloggers. I guess I should include you in that list, but I’ll bet you don’t make mistakes. We have never claimed that all the exit polls were accurate.

Then again, neither is your reporting.

warren mitofsky

Ouch.  I definitely have some thoughts on this one, but unfortunately, my day job prevented me from writing more today.  I’ll try to update this post later tonight.  For now, read the Kaus item in full (and if any of this exchange needs additional "demystifying, please post a comment and I’ll try to clarify). 

———

UPDATE:  First, because at least one highly valued reader heard it differently than I intended, let me clarify what I meant above by "ouch." It was that pained feeling I had watching someone I admire – and I’m talking about Mitofsky here – do something so obviously inappropriate. It was the way I would imagine Ohio State fans must have felt 26 years ago watching their legendary coach Woody Hayes punch that defensive back receiver from Clemson.

I’ll come back to my reaction to Mitofsky’s email, but the heart of this exchange is a question I considered a few weeks ago, "were the exit polls really wrong?" Looking back, I realize that I would have been well served by a good editor on that post, because while I asked a provocative question, I never made it clear where I stood. Moreover, by putting quotation marks around the word wrong (and then using the same phrase as the title of my exit poll FAQ), one could conclude I saw nothing "wrong" at all.

The point I wanted to make then was the exit polls were obviously wrong in some ways, not so wrong in others.  Everyone, even Mitofsky, concedes that the just-before-poll closing exit polls had an average "error" (or, to some, a "discrepancy") of roughly 2% in Kerry’s favor compared to the actual count.

Where the exit polls were right – or at least, not quite "wrong:" The errors were too small to achieve statistical significance in all but a handful of statewide polls. They were not large enough to give Kerry a lead beyond sampling error in any states that he ultimately lost, and not large enough to result in any wrong calls on election night.

[In an update, Kaus suggests a failing I overlooked: Projections in states like South Carolina might have been called earlier on actual vote returns but for exit poll errors in Kerry’s favor that implied those races would be close. As he writes: "The purpose of exit polls is obviously not simply to prevent the announcement of an ‘incorrect winner.’ It’s also to allow the earlier announcement of the correct winner"].

Where the exit polls were obviously wrong: As the Washington Post‘s Richard Morin put it in November, the errors were "just enough to create an entirely wrong impression about the direction of the race in a number of key states and nationally." And Kaus is right — it wasn’t just bloggers, but sophisticated journalists and political insiders who reached the wrong conclusion looking at those numbers on Election Night.

Of course, supporting the official network projections is only one mission, and arguably the least important. The exit poll subscribers also pay to get (a) some early indication of the outcome on Election Day so they can plan their coverage and (b) data to support analytical stories written on Election Night that explain the outcome and characterize the race among demographic subgroups. Here the exit polls obviously failed. News organizations planned coverage on the assumption that Kerry would win. Some stories based on the early evening cross-tabs apparently had to be rewritten. As John Ellis — a former analyst for both NBC and Fox News — wrote on his blog shortly after the election:

The lost productivity at places like The New York Times and The Washington Post, where literally hundreds of reporters and editors spent the equivalent of an 8-hour work day writing and preparing fiction…all of the consumers of this content have to be asking themselves: "why in the world do we pay for this?"

So who is to blame for that wrong impression? That is the central argument between Mitofsky and Kaus and others. Was it Edison/Mitofsky for how it managed and disseminated the results? Should the networks have spent more to assure better interviewing and coverage? Were they both wrong to resist disclosure of basic methodological details that might have helped reporters and editors and even bloggers better understand the limits of exit polls? Should those editors, reporters, and bloggers have known better? I tend to exempt the consumers, but otherwise, I find it difficult to place all the blame in one place, especially given how little we really know about what went wrong and why.

Having said all this, I will admit that I may have a bit of a blind spot with respect to Warren Mitofsky. He is, deservedly, a living legend in the field of survey research. In the 1960s and 1970s, working with a small group of colleagues at CBS News, he helped invent not only the exit poll, but also the CBS likely voter model and a practical methodology for random digit dial (RDD) telephone surveys that remains in use in to this day. He also spearheaded creation of the disclosure standards that explain the ubiquity of the "margin of error" in news stories about polling.

Of course, he also has a notoriously thin skin about criticism. In this regard, unfortunately, the email to Kaus speaks for itself.

I am willing to cut Mitofsky some slack — at least until I know more — about the nuts and bolts of why the exit polls were off. However, I tend to agree with his critics in one respect:  The lack of transparency about basic methodology, the instinct to deny obvious problems and then blame the bloggers and his habit of lashing out in anger at criticism are at odds with someone of Mitofsky’s well deserved reputation and stature.

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.