Palestinian Exit Polls

Wednesday’s Palestinian elections once again highlight the shortcomings of exit polls as a tool for projecting election results.  One of the many myths perpetuated by the seemingly endless debate over the 2004 U.S. exit polls involves the supposed infallibility of exit polls, especially in Europe and other countries.  In reality, as noted here more than a year ago, an election information project funded by the United National and the US Agency for International Development concluded in 1999 that when it came to projecting winners, "the majority of exit polls carried out in European countries over the past years have been failures."  Yesterday’s Palestinian elections provide another example of this fallibility. 

Our friend Professor Franklin has been all over the Palestinian polling lately, and his blog is the must-read on this subject.  At least two organizations conducted pre-election polling in Palestine, and Franklin’s chart shows a wide lead by the ruling Fatah party that closed to roughly ten points as Election Day approached.

Yesterday, three different exit polls all indicated a narrow win by Fatah. Here is Franklin’s summary of the "party list ballot" vote estimates:

There were three exit polls… done for the Palestinian Legislative Council elections on January 25th, one by the Development Studies Programme at Bir Zeit University, another by the Palestinian Center for Policy and Survey Research (PSR) and the third poll was done by An-Najah University in Nablus.
The results for the party list ballot were:

• DSP/Bir Zeit: Fatah 46.4%, Hamas 39.5%
• PSR: Fatah 42%, Hamas 35%
• An-Najah: Fatah 46%, Hamas 40%

The exit pollsters used these results to estimate the share of parliamentary seats won by each party and each projected Fatah winning more seats than Hamas, although their estimates ranged between 58 and 63 seats for Fatah and 46 to 58 for Hamas. 

This morning brought very different news.  Hamas had taken 76 seats to 43 for Fatah.  Obviously the early forecasts of a Fatah victory were off the mark, but what is unclear – at least from the reports I have seen – whether the discrepancies were about the estimates of the party ballot or in the estimated allocation of parliamentary seats.

A brief article from the Associated Press reports that exit pollsters were "at a loss to explain their failure" yet unwilling to comment for the record.  The article goes on to speculate:

The discrepancy may have been caused by reluctance of voters to admit to pollsters that they were abandoning the ruling party. The polling errors appeared especially glaring in district races, where smaller numbers of voters were surveyed.

Half the seats in Wednesday’s parliamentary vote were chosen on a national list and the other half by districts. While the national voting appeared to be close, election officials said Hamas had won a large majority in the district races.

Hamas apparently took advantage of divisions in Fatah, which had fielded multiple candidates in many districts, splitting the Fatah vote while Hamas’ support remained united.

As my knowledge of Palestinian elections and exit poll methodologies is cursory at best, I defer to Franklin: 

The problem of estimating winners in multimember districts with from one to nine members (averaging 4.1) is a daunting problem for any exit poll, even ignoring any response bias problems…We’ll need the district level vote data to know how close these district races were– my guess is that many were way too close to possibly be called by an exit poll at the district level (where the margin of error would be quite substantial.) But until the CEC releases the preliminary counts, we can’t do more than speculate about this. (Plus we don’t have access to exit poll results at the district level, at least not yet.)
What will be telling is if the exit polls estimates of the party list shares for Hamas and Fatah were close to right but the district level results were poorly estimated.

Franklin, in turn, links to the cautions issued by UCSD Political Science Professor Matthew Shugart about the perils of Palestinian exit polling before the announcement of the contrary election results:

I would be really cautious with exit polls in an electoral system like this-even if it were a ‘normal’ environment in which people felt free to talk to people on the street asking them how they just voted. By that I mean that this electoral system-multi-seat plurality, plus list PR in parallel-means the pollster needs to know:

(1) whether the voter used all his/her votes in the nominal tier (the local multi-seat district);

(2) the identities of all the candidates he or she voted for;

(3) and the party list the voter checked.

That’s a lot of moving parts for each interviewee. And then the exit-polling company has to extrapolate from a sample and somehow generate a national allocation. That involves lots of assumptions about how completely other similar voters filled out their slate of candidates in the nominal tier. In general, multi-seat plurality races are very hard to predict because small vote shifts for individual candidates can make substantial differences in the outcome of the election in a district. It is not as though the outcome can be extrapolated just from knowing the party a voter preferred when the voter has more than one vote and can use all or none of them and spread them out on candidates of multiple parties or concentrate them all on one party.

UPDATE:  Shugart has more to say here.

UPDATE (1/28):  Franklin has two new posts up.  In the first, he goes into detail on what he describes as the near "impossible task" of using an exit poll to pick winners inthe many multi-member voting districts in the Palestinian elections.  In the second, he compares how well each exit poll did at projecting the "national list vote:" 

The three Palestinian Legislative Council election exit polls tended to
overestimate Fatah’s vote in the national party list voting but they
all seriously underestimated Hamas’ strength. The result is that all
got the leader wrong, and this was beyond the margin of error of two of
the three surveys (and at the extreme end of the MOE of the third, and
least precise, survey.) The bottom line: the preliminary vote count
produces results that fall outside the margin of error of the exit
polls.

He goes on to speculate about three potential reasons for the discrepancy, a greater reluctance of Hamas voters to participate, a greater reluctance of Hamas voters to reveal their preference and the difficulty in constructing the exit poll sample given the lack of previous comparable elections to use a model.   Here is his bottom line:

[T]he exit poll errors cannot be explained by random variability due to
sampling. Systematic response errors, turnout estimation, or
non-response are likely culprits in this case. In principle, an exit
poll should have been able to detect the Hamas lead. With the sample
designs used here, and their associated margins of error, it is
unlikely any of them could have concluded that Hamas’ lead was
statistically significant. But getting the direction right was a
possibility.

One refreshing aspect of these exit poll problems
is that they do not easily lend themselves to the conspiracy theory
interpretations common after the U.S. 2004 presidential elections. With
ballot counting conducted under the Palestinian Authority, any
fraudulent counting would seem more likely to favor Fatah than Hamas.
Sometimes the exit polls are just wrong. We should all remember that
lesson (and continue to strive to improve the science of exit polls.)

For the graphs, details, links and more, read it all.

PS:  As of last night "final" vote data were still not available.  Franklin’s analysis is based on the preliminary count which, I’m told, should be pretty close to final.