What the USCV Report Doesn’t Say (Part II)

Exit Polls Legacy blog posts

Mea Culpa Update – In Part I of this series I erred in describing an artifact in the data tabulation provided in the report provided by Edison Research and Mitofsky International.   The artifact (which was first described by others back in January) results, not from random sampling error, but from all other randomly distributed errors in the count data obtained by Edison-Mitofsky. 

Over the last week, I have seen evidence that such an artifact exists and behaves as I described in Part I.   I will have more to report on this issue very soon.  For now, I can only report that others are hard at work on this issue and their findings are very intriguing.  As a certain well known website sometimes says, “Developing…”

[Update 4/21 – The development I am referring to is the work ofย  DailyKos diarist (and occasional MP commenter) Febble,” summarized in this blog post (and a more formal draft academic paper).ย  She reaches essentially the same conclusion that I did in Part I:ย 

It would seem that the conclusion drawn in the [US Count Votes] report, that
the pattern observed requires “implausible” patterns of non-response,
and thus leaves the “Bush strongholds have more vote-vount corruption”
hypothesis as “more consistent with the data” is not justified.
The pattern instead is consistent with the [Edison-Mitofsky] hypothesis of
widespread “reluctant Bush responders” – whether or not the “political
company” was “mixed”. 

Febble’s paper is undergoing further informal “peer review” and — to her great credit — she is considering critiques and suggestions from the USCV authors.    I will have much more to say about this soon.  Stay tuned…]

In the meantime, let me continue with two different issues raised in the US Count Votes (USCV) report and some general conclusions about the appropriate standard for considering fraud and the exit polls.

4) All Machine Counts Suspect? – The USCV report makes much of a tabulation provided by Edison-Mitofsky showing very low apparent rates of error in precincts that used paper ballots.  Paper ballot precincts, they write, “showed a median within-precinct-error (WPE) of -0.9, consistent with chance while all other technologies were associates with high WPE discrepancies between election and exit poll results.”  Their Executive Summary makes the same point illustrated with the following chart that highlights the low WPE median in green.

While the report notes that the paper ballots were “used primarily in rural precincts,” and earlier in the report notes that such precincts amounts to “only 3% of sampled precincts altogether,” it fails to point out to readers why those two characteristics call the apparent contrast between paper and other ballots into question.

What the USCV report does not mention is the finding by the Edison-Mitofsky report that the results for WPE by machine type appear hopelessly confounded by the regional (urban or rural) distribution of voting equipment.  The E-M report includes a separate table (p. 40) that shows higher rates of WPE in urban areas for every type of voting equipment.  Virtually all of the paper ballot precincts (88% — 35 of 40) were in rural areas while two thirds of the machine count precincts (68% – 822 of 1209) were in urban areas.  E-M concludes:

These errors are not necessarily a function of the voting equipment.  They appear to be a function of the equipment’s location and the voter’s responses to the exit poll at precincts that use this equipment.  The value of WPE for the different types of equipment may be more a function of where the equipment is located that the equipment itself (p. 40). 

The USCV report points out that Edison/Mitofsky “fail to specify P-values, significance levels, or the statistical method by which they arrived at their conclusion that voting machine type is not related to WPE.”   Here they have a point.  The public would have been better served by a technical appendix that provided measures of significance.  However, public reports (as opposed to journal articles) frequently omit these details, and I have yet to hear a coherent theory for why Edison-Mitofsky would falsify this finding.

Nonetheless, USCV want us to consider that “errors in for all four automated voting systems could derive from errors in the election results.”  OK, let’s consider that theory for a moment.  If true, given the number of precincts involved, it implies a fraud extending to 97% of the precincts in the United States.  They do not say how that theory squares with the central contention of their report that “corruption of the official vote count occurred most freely in districts that were overwhelmingly Bush strongholds” (p. 11).  Their own estimates say the questionable strongholds are only 1.6% of precincts nationwide (p. 14, footnote).  Keep in mind that their Appendix B now concedes that pattern of WPE by precinct is consistent with “a pervasive and more or less constant bias in exit polls because of a differential response by party” in all but the “highly partisan Bush precincts” (p. 25).

Presumably, their theory of errors derived from “all four automated voting systems” would also include New Hampshire, the state with fourth highest average WPE in the country (-13.6), where most ballots were counted using optical scan technology.  In New Hampshire, Ralph Nader’s organization requested a recount in 11 wards, wards specifically selected because their “results seemed anomalous in their support for President Bush.”   The results?  According to a Nader press release

In the eleven wards recounted, only very minor discrepancies were found between the optical scan machine counts of the ballots and the recount. The discrepancies are similar to those found when hand-counted ballots are recounted.

A Nader spokesman concluded, “it looks like a pretty accurate count here in New Hampshire.”


5) More Accurate Projection of Senate Races?
– The USCV reports that in 32 states “exit polls were more accurate for Senate races than for the presidential race, including states where a Republican senator eventually won” (p. 16).  They provide some impressive statistical tests (“paired t-test, t(30) = -.248, p<.02) if outlier North Dakota is excluded”) but oddly omit the statistic on which those tests were based.

In this case, the statistically significant difference is nonetheless quite small.  The average within precinct error (WPE) in the Presidential race  in those states was -5.0 (in Kerry’s favor), the average WPE favoring the Senate Democrat was -3.6 (See Edison-Mitofsky, p. 20).   Thus, the difference is only  1.4 percentage points — and that’s a difference on the Senate and Presidential race margins.

The USCV authors are puzzled by this result since “historic data and the exit polls themselves indicate that the ticket-splitting is low.”  On this basis they conclude it “reasonable to expect that the same voters who voted for Kerry were the mainstay of support Democratic candidates” (p. 16).      

That expectation would be reasonable if there were no cross-over voting at all,  but the rates of crossover voting were more than adequate to explain a 1.4 percent difference on the margins.  By my own calculations, the average difference in the between the aggregate margins for President  and US Senate in those 32 states was 9 percentage points.  Nearly half of the states (14 of 32) showed differences on the margins of 20 percentage points or more.   

And those are only aggregate differences.  The exit polls themselves are the best available estimates of true crossover voting.  Even in Florida, where the aggregate difference between the Presidential and Senate margins was only 4.0 percentage point, 14% of those who cast a ballot for president were “ticket splitters,”  according to the NEP data.

Historically low or not, these rates of crossover voting are more than enough to allow for a difference of the margin of only 1.4 percentage points between Presidential and Senate votes. 

The consistency in the average WPE values for the Senate races is greater than the slight difference with the presidential race.  The exit polls gave erred for the Democratic candidate across the board, including races featuring a Democratic incumbent (-3.3), a Republican incumbent (-5.2) or an open seat (-2.2). 


Conclusion: The Burden of Proof?
 

In the comments section of Part I of this post, “Nashua Editor” (of the Nashua Advocate) asked an interesting question: 

Is it not statistically, procedurally, and (dare I say) scientifically sound to maintain skepticism over a scientific conclusion until that conclusion has been verified? Is a scientist, or exit-pollster, not called upon, in this scenario, to favor the U.S. Count Votes analysis until it is proven incorrect?

The question raises one of the things that most troubles me about the way much of the argument about the exit polls and vote fraud turn the scientific method on its head.   It is certainly  scientifically appropriate to maintain skepticism over a hypothesis until it has been verified and proven.  That is the essence of science.  What does not follow is why the USCV analysis should be favored until “proven incorrect.”  When the USCV report argues that “the burden of proof should be to show that election process is accurate and fair” (p. 22),  it implies a similar line of reasoning:   We should assume that the exit polls are evidence of fraud unless the pollsters can prove otherwise. 

Election officials may have a duty to maintain faith in the accuracy and fairness of the  election process, but what USCV proposes is not a reasonable scientific or legal standard for determining whether vote fraud occurred.  The question (or hypothesis) we have been considering since November is whether the exit polls are evidence of fraud or systematic error benefiting Bush.  In science we assume no effect, no difference or in this case no fraud until we have sufficient evidence to prove otherwise, to disprove the “null hypothesis.”  In law — and election fraud is most certainly a crime — the accused are innocent until proven guilty.  The “burden of proof” only shifts if and when the prosecutor offers sufficient evidence to convict. 

In this sense, good science is inherently skeptical.  “Bad science,” as the online Wikipedia points out, sometimes involves “misapplications of the tools of science to start with a preconceived belief and filter one’s observations so as to try to support that belief. Scientists should be self-critical and try to disprove their hypotheses by all available means.” 

Whatever its shortcomings, the Edison-Mitofsky report provided ample empirical evidence that:

  • The exit polls reported an average completion rate of 53%, which allowed much room for errors in the poll apart from statistical sampling error.
  • The exit polls have shown a consistent “bias” toward Democratic candidates for president since 1988, a bias that was nearly as strong in 1992 as in 2004. 
  • In 2004, the exit polls showed an overall bias toward both John Kerry and the Democratic candidates for Senate.
  • Errors were very large and more or less constant across all forms of automated voting equipment and tabulation in use in 97% of US precincts
  • The exit poll errors strongly correlated with measures of interviewer experience and the precinct level degree of difficulty of randomly selecting a sample of voters.  In other words, they were more consistent with problems affecting the poll than problems affecting the count.

Given this evidence, the hypothesis that the exit poll discrepancy was evidence of fraud (or at least systematic error in the count favoring Republicans) requires one to accept that:

  • Such fraud or systematic error has been ongoing and otherwise undetected since 1988.
  • The fraud or errors extend to all forms of automated voting equipment used in the U.S.
  • Greater than average errors in New Hampshire in 2004 somehow eluded a hand recount of the paper ballots used in optical scan voting in precincts specifically chosen because of suspected anomalies. 

Add it all up and “plausibility” argues that exit poll discrepancy is a problem with the poll, not a problem with the count.  Yes, Edison-Mitofsky proposed a theory to explain the discrepancy (Kerry voters were more willing to be interviewed than Bush voters) that they cannot conclusively prove.  However, the absence of such proof does not somehow equate to evidence of vote fraud, especially in light of the other empirical evidence. 

Of course, this conclusion does not preclude the possibility that some fraud was perpetuated somewhere.   Any small scale fraud would have not been large enough to be detected by the exit polls, however, even it if amounted to a shift of a percentage point or two in a statewide vote tally.  The sampling error in the exit polls makes them a “blunt instrument,” too blunt to detect such small discrepancies.   Again, the question we are considering is not whether fraud existed but whether the exit polls are evidence of fraud.   

You don’t need to take my word on it.  After the
release of the Edison-Mitofsky report, the  non-partisan social scientists in the National Research Commission on Elections and
Voting in
the NRCEV examined the exit poll data and concluded (p. 3):

Discrepancies between early exit poll results and
popular vote tallies in several states may have been due to a variety
of factors and did not constitute prima facie evidence for fraud in the current election.

[4/18 & 4/19 – Minor typos and grammer corrected, links added]

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.