Is RFK, Jr. Right About Exit Polls? – Part III

This post resumes my paragraph-by-paragraph review of the discussion of the exit polls in the article by Robert F. Kennedy, Jr. in Rolling Stone, "Was the 2004 Election Stolen?"   Part I looked at Kennedy’s claims that exit polls have been "exquisitely accurate" in the past.  Part II examined the claim that networks "scrubbed the offending results."  Passages from the article are in bold italics. 

On the evening of the vote, reporters at each of the major networks were briefed by pollsters at 7:54 p.m. Kerry, they were informed, had an insurmountable lead and would win by a rout: at least 309 electoral votes to Bush’s 174, with fifty-five too close to call.(28) In London, Prime Minister Tony Blair went to bed contemplating his relationship with President-elect Kerry.(29)

Of the many distortions in this article, this one is the most bizarre.  Perhaps it depends on the meaning of "briefed." I have checked with three individuals who worked within the NEP consortium on Election Night, and none remembers any such "briefing" by the exit pollsters anywhere near that hour.  One source told me flatly that Edison/Mitofsky conducted no conference call briefings in the early evening.  The data streamed constantly to "decision screens" of network analysts and, if necessary, they were able to communicate with the pollsters via instant message.

The eight o’clock hours was an important time, of course, because polls were about to close in fifteen states plus the District of Columbia. The networks would be projecting winners in many of those states – the ones where the outcome had never been in doubt – at the top of the hour on the basis of exit poll results.  However, the idea that the exit pollsters "informed" their NEP clients at that hour that Kerry "had an insurmountable lead and would win by a rout" defies all logic and available evidence.

First, if the networks believed Kerry had a lock on 309 electoral votes at 7:54 p.m., it certainly had no effect on the projections they made on the air.  In fact, none of the networks ever projected an overall winner.  The margins in Iowa, Wisconsin and New Mexico remained so close that the networks were not able to project a winner until after Kerry had conceded the next day.  And the networks split on Ohio and Nevada.  Only FOX and NBC called Ohio for Bush, but not until just before 1:00 a.m. and neither network projected Nevada.  CBS, ABC and CNN called Nevada for Bush at about 3:00 a.m., but never called a winner in Ohio.   Thus it almost goes without saying that back at 7:54 p.m., the states that would decide the Election remained too close to call — well within the margin of error –on all of the network’s exit poll based estimates and projections.

And if that’s not enough, the Edison/Mitofsky pollsters conducted a well-documented briefing at 4:30 p.m. very different from the one Kennedy claims occurred three hours later.  Warren Mitofsky discussed it a few days later on the News Hour, and the E/M report described it as follows:

On Election Day, at 4:30 PM ET, we convened a conference call with the Decision Teams of the NEP members and cautioned them that we expected sizeable errors in the exit polls in nine states; in seven states (Connecticut, Delaware, Minnesota, New Hampshire, Pennsylvania, Vermont and Virginia) we suspected that the exit poll estimates were overstatement [sic] of the vote for Kerry; in two states (South Dakota and West Virginia) we suspected an overstatement of the Bush vote. We made these warnings based upon the discrepancies between the exit polls and our prior estimates in these nine states. [p. 17 – Note:  the term "prior estimates" refers to a summary of pre-election polls results for each state collected by NEP].

And here is the back-story that provides some added confirmation:  The NEP "subscribers" — mostly newspapers including the New York Times and the Washington Post — were not included on that call and were quite livid as a result.  Edison-Mitofsky had to essentially apologize in their report (p. 17) for "not sharing with the subscribers our concerns about the accuracy of the exit poll estimates." 

There was however, one person — a blogger — who communicated the substance of that call to a much wider audience.  Her name was Ana Marie Cox, and at 5:40 p.m. on Election Day she posted the following message to Wonkette: 

Disclaims a birdie: "*** There appear be problems with exits in the following states that could be tipping numbers toward kerry: MN, NH, VT, PA, VA, CT, DE. described only as ‘serious’ issues we’re looking at. so i would not put too much faith in those results."

So there you have it.  For all the disparagement of bloggers that followed, those who read Wonkette on Election Day got better real-time information on the exit poll’s shortcomings than the paid subscribers at the New York Times and the Washington Post. 

As the last polling stations closed on the West Coast, exit polls showed Kerry ahead in ten of eleven battleground states — including commanding leads in Ohio and Florida — and winning by a million and a half votes nationally. The exit polls even showed Kerry breathing down Bush’s neck in supposed GOP strongholds Virginia and North Carolina.(30) Against these numbers, the statistical likelihood of Bush winning was less than one in 450,000.(31)

That reference to West Coast poll closings does add a dramatic touch to this story.  While no one denies that the Virginia and North Carolina exit polls showed discrepancies favoring Kerry, by the time "the last polling stations closed on the West Coast," George Bush had been the declared winner of those two states for more than two hours. 

And while everyone concedes that the discrepancy in Kerry’s favor on the national exit poll sample was statistically significant, Kennedy’s contention that the exit polls gave Kerry "commanding leads" in Ohio and Florida has generated a confusing bit of controversy over last few days

Writing on Salon,com, Farhad Manjoo linked to one of my blog posts from December 2004 and concluded:

Of the ten battleground states that the exit poll showed Kerry winning, he ultimately lost four — states that, you could say, cost him the election. These were Ohio, Iowa, Nevada and New Mexico. But in none of those states was Kerry’s lead outside the poll’s margin of error.  In other words, the poll results showed a race that was too close to call . . . As Mitofsky told me, television news networks, looking at the exit poll data, seemed to understand that Kerry did not top the margin of error, and so did not call these states for him [emphasis in original].

In a reply posted Wednesday on Salon.com, Kennedy answered back: 

With regard to his claim that the exit poll numbers were within the margin of error, Manjoo is referring to the "corrected" numbers that Mitofsky retroactively weighted to more closely resemble the certified vote tallies. Freeman and other statisticians, mathematicians and social scientists disturbed by the poll numbers rely instead on Mitofsky’s raw data. Those numbers, as I report, indicated that Kerry had an advantage outside the margin of error in Ohio, Florida, Nevada and New Mexico. He also had advantages within the margin of error in two other states: Iowa and Colorado, which also went to Bush in the final results. It’s hard to engage in an honest debate with someone who won’t debate the same set of numbers [emphasis in original].

One thing that truly mystified me was Kennedy’s assertion that the final exit polls showed Kerry "ahead" by any margin, commanding or not in Florida or Colorado.  Steven Freeman, whose forthcoming book Kennedy cites in Rolling Stone to support the "commanding lead" claim, had originally authored a paper touting numbers extrapolated from tabulations he believed to be "uncalibrated" and "based solely on subjects surveyed leaving the polling place" that had appeared CNN.com on November 3, 2004 at 12:21 a.m. Rather than showing a "commanding lead," Freeman’s numbers showed Bush very narrowly ahead of Kerry in Florida (49.8% to 49.7%) and Colorado (49.9% to 48.1% — see the Simon/Freeman data on p. 15).   

Far more important, the Edison/Mitofsky report included the "Call 3 Best Geo Estimator" data for each state.  Again, these are the estimates used to make election "calls" as the polls closed based solely on the exit poll results.  They were absolutely not "’corrected’…to more closely resemble the certified vote tallies," because no such tallies existed at the time.  And these estimates showed Bush ahead of Kerry by 1.1 points in Florida and 5.5 percentage points in Colorado (p. 21).

As long as we are on the subject, the Edison/Mitofsky report also includes the statistics necessary to do a far more precise calculation of the statistical significance of the projected Kerry leads in the states that Kerry ultimately lost.  The "t-scores" I calculate (0.4 for Iowa, 0.4 for Nevada, 1.4 for New Mexico and 1.7 for Ohio) indicate that Kerry’s leads in these states were not significant even at a 95% confidence level.  Note that Edison/Mitofsky required a much more demanding 99.5% confidence level, among other criteria, before recommending that the networks project a winner (see the discussion of "critical value" on p. 7). 

So what on earth was Kennedy talking about?  It took some digging to figure out, but Freeman has apparently discarded numbers from his original paper and now believes he has better data.  This handout, from an October presentation, is what he now claims represents "how random samples of 114,559 voters nationwide said they voted for as they walked out of the voting booth."   And it purports to show Kerry ahead "beyond the polling margin of error" in Florida, Ohio, New Mexico and Nevada and ahead within the margin in Colorado and Iowa. 

And where did these numbers come from?  It is not obvious, but the handout indicates that Freeman took the average "within precinct error" (WPE) from the Edison/Mitofsky report for each state (which he trusts, apparently, but renamed as "Precinct Level Disparity"), divides it in half, adds half of the average precinct error to Kerry’s official statewide vote, subtracts half the average precinct error from Bush’s official statewide total, and describes the result as indicating what the "exit polls projected" about each race.   

Now I can think of a whole host of reasons why that is a bad idea. The exit polls in 13 states — including Florida, Colorado, Iowa, Nevada and New Mexico — included telephone interviews of absentee voters for which WPE does not apply. 
The exit pollsters weighted these absentee samples to match their estimate of what absentee voters would contribute to the total vote: 27% in Florida, 30% in Iowa and 50% in both Nevada and Colorado.  Many states, such as Florida, included over-samples of Latino or African America precincts that could have a disproportionate effect on WPE.  Finally, WPE is calculated by comparing the official count for each precinct to the exit poll result, but 29% of the sampled precincts nationwide were at polling places where two or more precincts voted. 

Steven Freeman apparently believes that he has found a better way to use exit polls to determine which candidate won the election.   Note, however, that his novel extrapolation requires access to the official precinct level count — something that was obviously impossible to obtain before the polls closed.  Some observers might accept that Freeman’s approach represents an improvement on the projections developed and applied by all the statisticians and analysts at all of the networks.  But one thing is absolutely clear:  The numbers in Freeman’s handout are not the estimates that those analysts used on Election Night.

And there is also the issue of sampling error.  Even if we accept Freeman’s "projections," he provides absolutely no documentation or explanation in the handout for how he calculated statistical significance for these estimates. The statisticians I talked were absolutely stumped as to how Freeman concluded that the 2.6 percent "lead" he gives Kerry in Florida could be "beyond the polling margin of error" given his assumption of a 97.5% confidence level.

And one more thing: After telling us about Kerry’s supposedly "commanding lead" in Florida, Kennedy reverts in the very next paragraph to the numbers from Freeman’s original paper (that showed discrepancies between the exit poll and the vote in Ohio, Pennsylvania and Florida of 6.7, 6.5 and 4.9 respectively – I will take up that paragraph in Part IV).  These were the same numbers that I used to calculate margins of error back in 2004, and thus the same numbers that Kennedy condemned on Wednesday as "corrected" by Mitofsky "to more closely resemble the certified vote tallies" 

So perhaps before lecturing Farhad Manjoo and the readers of Salon about how "hard" it is "to engage in an honest debate with someone who won’t debate the same set of numbers," Robert Kennedy might want to decide which set of numbers he wants to debate. 

”Either the exit polls, by and large, are completely wrong,” a Fox News analyst declared, ”or George Bush loses.”(32)

But wait, did some anonymous network "analyst" betray a belief among the pollsters that their numbers pointed to a Kerry victory? 

Hardly.  The "Fox News analyst" in question was Susan Estrich, a Democrat and Michael Dukakis’s campaign manager in 1988 (the Fox clip was rebroadcast on CNN and quoted in the Washington Post).   Moreover, according to the Fox transcript (via Nexis), a few moments later Estrich added, "there are years in which exit polls have
been wrong."

Yes, the exit polls did significantly overstate Kerry’s vote in Ohio and nationwide and, yes, they did show Kerry with a statistically significant lead in the national popular vote. And, as the Washington Post‘s Richard Morin put it in right after the election, those discrepancies were "just enough to create an entirely wrong impression about the direction of the race in a number of key states and nationally."  Many other cable news talking heads were fooled, as were many reporters, editors, campaign staffers and millions of ordinary Americans.  That result is certainly a problem for the exit polls and the journalists that depend on them (a topic I considered in this post).

But the main point here is that the network analysts and exit pollsters were not fooled when they looked at their estimates on Election Night.  They missed no projections.  Robert Kennedy tries very hard to create the opposite impression, perhaps because the fact that the exit polls lacked the statistical power to project a winner makes it tough to show that the election was stolen. 

Continues with Part IV, hich discusses the larger issue – whether the statistically significant discrepancies between the exit polls and the vote count present any evidence of fraud.  I am grateful to Mark Lindeman and Elizabeth Liddle for their helpful comments on drafts of this post.