Why & How Pollsters Weight, Part IV

Legacy blog posts Weighting by Party

In Part I of this thread, I listed pollsters that weight by demographics but not by party identification, including the ABC News/Washington Post poll. I recently learned that ABC does weight the likely voter numbers in their October tracking survey by party ID. Like the approaches described in the last post, the ABC model is something of a compromise and worthy of further discussion.

The reporting of the ABC/Washington Post daily tracking poll is a bit confusing. I have received several emails from readers asking about the following passage from the Washington Post’s methodology page (a similar blurb appears at the bottom of most ABC poll stories):

The Post and ABC News collect data jointly but are responsible for developing their own methods to identify likely voters. This may produce slightly different estimates of candidate support.

What does this mean? Let’s start at the beginning. The Post and ABC both share data collected and tabulated every night by the same interviewing facility. Although they start with the same data they weight it differently and apply different likely voter selection models. I assume the weighting procedures are different, even before they select likely voters, because ABC and the Post have reported slightly different percentages among registered voters for 7 of the last 9 days (and reported ranges of interview dates were identical). Tonight among registered voters, for example, the Post has Kerry head of Bush (48% to 46%), while ABC has Bush ahead by the opposite margin.

[As of this writing, I cannot say much more about the weighting procedures used by the Washington Post, except that their methodology page makes no reference to weighting by party. I have made an email inquiry of Richard Morin, the Post’s polling director, and will report back if and when I hear more.]

Based on their methodology page and an email exchange with ABC Polling Director Gary Langer, I can explain the weighting procedures used by ABC. They report all results using a rolling three-day average, but weight each night’s data separately. They weight the full sample of adults each night to match census demographic estimates (for gender, age, race and education). For the registered voter sample, ABC stops there, weighting only by demographics.

In the October tracking surveys, likely voters get another weight for party identification, using a method that literally splits the difference between the Zogby approach (weighting to previous exit poll results) and the purist approach (not weighting by party at all). On any given night, ABC’s party ID target for likely voters is the average of that night’s unweighted result for party ID and the average party identification result on exit polls for the last three presidential elections (roughly 39% Democrat, 35% Republican, 26% independent).

ABC’s Langer and Merkle lay out the rationale for this compromise on page two of their methodology brief. They see the merits of both arguments. On the one hand, “party ID has been remarkably stable in exit polls conducted in presidential elections since 1984.” On the other, “party ID can and does change, and that polls measuring the dynamics of the race – rather than simply attempting to predict its outcome – need to measure and report this change, not suppress it.” So they adopt a weighting scheme that allows some variation, while also imposing a control tied to past exit poll results. Langer and Merkle hasten to add that having gone to all this trouble four years ago, their party ID weighting “had essentially no effect on our estimate of vote preferences – no more than a single point on any given day.”

* * *
The issue of weighting by party is a prime example of the tension between science and art in political polling. When it comes to art, judgements are always subjective. Keeping that in mind, here is my take: I am most comfortable with a combination of the approaches of John Gorman and Peter Hart (of the Fox News and NBC/Wall Street Journal surveys respectively) described in the last post. Gorman’s approach of stratifying his samples by actual turnout statistics, despite the risk that past turnout is not a perfect guide to the future, forces a defensible regional consistency across surveys that makes weighting by party less necessary. If weighting is ever necessary, I prefer Peter Hart’s cautious, ad-hoc “dynamic weighting in reserve” approach.

One thing to keep in perspective: The debate over party identification is important, but those who weight by party have no magic “fix” to the sometimes random variation in surveys and those who avoid weighting are not overlooking some obvious methodological flaw.

Which leads to my last point: For the last month, Ruy Teixeira and his correspondent, Alan Abramowitz have been loudly urging pollsters to weight by party identification to correct arguably flaws they perceive in likely voter models. I am sympathetic to some of their critiques of likely voter screening. However, they are now attacking the “silly” ABC/Washington Post likely voter model and suggesting that their “registered voter results are probably a better indicator of the actual standing of the race.” Perhaps. But, as we now know, ABC weights its likely voter numbers by party, but not its samples of registered voters. So is the problem about the lack of weighting or the result?

[Alan Abramowitz responds here]

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.