Likely Voters V: More Gallup

Legacy blog posts Likely Voters

In the last post I covered how the Gallup likely voter model works. In this post, I want to review criticisms of the model.

An Imperfect Predictor? One complaint about the Gallup model and its progeny is that they do not perfectly predict likely turnout. Some real voters get classified as “unlikely” – some non-voters are deemed “likely.” The creators of the original Gallup model did not promise that their model could make a 100% accurate classification only that selecting a subgroup of the likely voters sized to match the likely turnout level provides the most accurate read on the outcome. Keep in mind that although the mechanics of the model have been in use for more than 40 years, the methodologists that apply it review how well the model worked after every election. Since they typically ask vote questions of all registered voters, they can look back after each election and check whether alternative models would have predicted the outcome more accurately. If Gallup continues to stick with the model, it is because they believe it continues to work as well or better than any alternative.

As described in the last post, the original Gallup models were based on validation studies that obtained the actual vote history for respondents. This process was relatively easy when pollsters interviewed respondents in-person, as their names and addresses easily obtained. Conducting a validation study on a random digit dial (RDD) telephone interview, requires that the respondent provide their name and address to the pollster. And records are dispersed in thousands of clerks offices and databases across the country. So such studies are now rare and difficult.

In 1999, the Pew Research Center conducted such a validation study of the Gallup likely voter model, using polls taken during the Philadelphia mayor’s race. They were able to obtain actual voting records for 70% of their respondents. The Pew report is worth reading in full, but here are the two key findings. On the one hand, the likely voter model was far from perfect in predicting individual level voting: .

Using [the Gallup likely voter] index, the Center correctly predicted the voting behavior of 73% of registered voters…. The 73% accuracy rate means that 27% of respondents were wrongly classified — those who were determined as unlikely to vote but cast ballots (17%), or non-voters who were misclassified as likely to vote (10%).

On the other hand, the Pew report showed that the results of the likely voter sample came closer to predicting the very close outcome, as well as the preferences of those in the sample who actually voted, than the broader sample of registered voters.

A longer academic paper based on the study summed up the conventional wisdom accepted by most public pollsters: “Though it is impossible to accurately predict the behavior of all survey respondents, it is possible to accurately estimate the preferences of voters by identifying those most likely to vote.”

One important limitation: The Pew study involved a low-turnout, off-year mayoral election, where the difference between the size of the electorate and the pool of registered voters was large. In a high turnout presidential elections in which 80% or more of registered voters cast ballots, there is typically less difference between registered and likely voters.

Too Much Volatility? The most common complaint directed at Gallup’s likely voter model is that it seems to yield more volatile results than other polls. In 2000, Gallup’s daily tracking surveys showed dramatic swings. On October 2, for example, they reported a dead heat between George Bush and Al Gore among likely voters (45% to 45%). Four days later following the first debate, they had Gore suddenly ahead by 11 points (51% to 40%). Four days after that, Bush was ahead by eight (50% to 42% — see the chart prepared by Gallup). Other polls taken over the same period showed nowhere near as much change, and Gallup’s own registered voter samples were more stable.

While Gallup dropped the daily tracking program this year, they have continued to show more volatility than other surveys. For example, they had Bush ahead by fourteen points (54% to 40%) in mid-September, had Kerry ahead by a single point after the debates (49% to 48%) and now have Bush leading again by six (51% to 46%).

An article in the current issue of Public Opinion Quartely presents evidence that the volatility resulted mostly from changes in the composition of the Gallup likely electorate. In other words, the volatility resulted less from a changing opinions than from changes in the people that Gallup defined as a likely voters. Authors Robert Erikson, Costas Panagopolouos and Christopher Wlezien analyzed the raw Gallup data from 2000 available in the Roper Center Archives. They compared likely voter non-likely voters and found that trend lines moved in opposite directions over the course of the campaign. The concluded that “most of the change (certainly not all) recorded in the 2000 CNN/USA is an artifact of classification,” and that the shifts resulted from:

The frequent short-term changes in relative partisan excitement…At one time, Democratic voters may be excited and therefore appear more likely to vote than usual. The next period the Republicans may appear more excited and eager to vote. As Gallup’s likely voter screen absorbs these signals of partisan energy, the party with the surging interest gains in the likely voter vote. As compensation, the party with sagging interest must decline in the likely voter totals. [The full text is available here]

Although Gallup has not formally responded to the Erikson, et. al. study, the methodologists at Gallup do not quarrel with the basic finding. Gallup’s Jeff Jones recently told the New York Times:

We’re basically trying to get a read on the electorate as of the day that we’re polling,” said Jeffrey Jones, managing editor of the Gallup Poll, “not necessarily trying to predict what’s going to happen on Election Day itself.

Jones frames the key question perfectly. Most pollsters agree that a pre-election survey is no more than a snapshot of opinions of the moment, but what about the people in the sample? As Erikson put it in an email to me last week, do we want surveys to identify those who are likely to vote on Election Day or those who are likely to vote “if the election were held today?”

Gallup’s answer is to let the composition vary. My view, and the view of most of my colleagues who poll for political candidates, is that we need to impose controls to keep the composition of the likely voters as constant as possible. However, those controls require making subjective decisions about what the likely electorate will look like on Election Day. Some weight by party (like Zogby and others). Others stratify their sample regionally to match past vote returns (like Greenberg/Democracy Corps and Fox/Opinion Dynamics) – an approach I prefer. However, supporters of the Gallup model argue that both alternatives pose a greater risk of imposing past assumptions on an unknown future.

I think those compromises are worthy. but I am a producer. You are consumers. How would you answer Erikson’s question? If you have a strong feeling, enter a comment below.

I’ll take that up one last complaint about the Gallup model in the next post.

[Mispelling of Erikson corrected]

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.