Post/ABC Poll Bites Back

Legacy blog posts Polls in the News

Earlier this week, I started a discussion of the notion of using demographic and attitudinal data as “diagnostic” measures to assess political surveys.  Two recent trends make this sort of analysis both possible and important:  (a) Pollsters are starting to disclose more about the demographic and attitudinal composition of their samples and (b) political partisans, mostly in the blogosphere, are starting to dissect and criticize the demographic results of polls they find disagreeable.  Unfortunately, that criticism is often quite wrongheaded.

Today brings another example.  A poll released on Tuesday last night by the Washington Post and ABC News had some unwelcome news for President Bush.    Within hours, the blogger “Bulldogpundit” on AnkleBitingPundits posted a takedown: “Here We Go Again – Debunking Another Slanted Poll From The Washington Post.”  Both Instapundit and the National Review Online (NRO) linked to it; John Podhoretz of NRO called the critique “incredibly convincing” and the poll worthy of “shame.”  And according to AnkleBitingPundits (henceforth “ABP”), Tony Snow “favorably cited” the post “on his nationally syndicated show.”

MP is less than convinced.  Let’s take APB’s case point by point:

1. Party Leanings
– Go to page 20 of the results. The respondents tend to “think of
themselves” as follows: 30% Democrat; 31% Republican, and 34%
Independent. Sounds fair you say (even though the 2004 exit poll showed
R’s and D’s split at about 37%). Yeah, it sounds about right – till you
read the next question and you find that the respondents “lean” towards
the Democrats by a percentage of 48% to 34%, which confirms something I
long thought. People in polls who ID themselves as “Independent” are
mainly Democrats and liberals.

This observation prompted NRO’s Podhoretz to ask, “how on earth could the Post actually think a poll whose respondents lean 48 percent Democratic to 34 percent Republican would have any validity?”

First, a bit of explanation, as the Post‘s presentation of the classic party identification (in their PDF summary) appears to have created some confusion.  They first asked the classic party identification question (Q901), “generally speaking, do you usually think of yourself as a Democrat, a Republican or an Independent?”  Republicans had a one-point advantage (31% to 30%).  Independents got the classic follow-up (Q904): “Do you lean more towards the Democratic Party or the Republican Party?”  The results for Q904 as presented in the Post summary (48% Democrat, 34% Republican) were clearly computed among independents only.  The summary also included a result for “Leaned Party ID” (Q901/904), a combination of the two questions tabulated among all adults: 48% Democrat or lean Democrat, 45% Republican or lean Republican.

Are these results “slanted” toward the Democrats?  If anything, the opposite is true.  Consider first the results from the first part of the question.  The latest Post/ABC poll includes fewer Democrats and more Republicans than their most surveys:

[Note, the Post released party ID numbers for its two most recent surveys (here and here).  I obtained results for the September and December surveys from the Roper Center IPoll database, and estimated party ID for the October tracking surveys from a cross-tabulation presented by the ABC pollsters at the recent AAPOR conference (see discussion below).  I have not yet been able to obtain results for party ID for surveys done earlier in 2005 by the Post and ABC]. 

The pattern is similar when we compare the Post/ABC poll (30% Democrat, 31% Republican) to what other pollsters measured during 2004:

Finally, what about “leaned Party ID?” The three point Democratic advantage on the Post/ABC poll (48% to 45%) is exactly the same as Gallup’s average for 2004 (48% to 45%, my tabulation using their Gallup Brain archive) and slightly more Republican than the combined result from the Pew Research Center during 2004 (47% to 41%).  Both results were also based on samples of all U.S. adults. 

ABP continues:

2. Sample Group and Timing Of Poll –  First of all, the Post polls only “adults,” not  “registered” or even “likely” voters.  As you know 36% of the respondents aren’t even eligible to vote, and of those that are eligible, only 60% vote.

This is the one aspect of ABP’s criticism that gets some support elsewhere, yet the argument is a bit of a red herring.  It is true that the Post/ABC poll surveys only adults. So do the most recent polls taken by Gallup/CNN/USAToday, CBS/New York Times, NBC/Wall Street Journal, Pew, Time, Newsweek, Harris and the LA Times.  It is true that a large portion of the adults in these samples do not vote.  Pre-election surveys that aim to forecast an election or measure the opinions of likely voters – including virtually all done by private campaign pollsters of both parties – routinely screen for registered voters.  But public polls typically have a broader mission:  They measure the opinion of all Americans. 

Both the Washington Post story and the ABC News analysis consistently refer to their poll as representing “the American public” or “Americans.”   So the issue is more one of philosophy than slant.  Why should a poll of Americans exclude the views of non-voters? 

There’s more…

Next, 1/2 of the polling nights are considered weekend nights, and weekend polling is notoriously unreliable and favorable to Democrats.

MP has his own doubts about the reliability of weekend interviewing but was surprised to see evidence presented by the ABC pollsters at the recent AAPOR conference showing no systematic bias in partisanship for weekend interviews.  The ABC pollsters looked at their pre-election tracking surveys conducted between October 1 and November 1 of 2004, and compared 14,000 interviews conducted on weeknights (Sunday to Thursday) with 6,597 conducted on Fridays and Saturdays.  Party ID was 33% Dem, 30% GOP on weeknights, 32% Dem 30% GOP on weekends, a non-significant difference even with the massive sample size.

3. Age of Respondents – The poll also over samples the number of 18-29 year old voters, the age group that voted most for Kerry. In 2004, 17% of the electorate was between 18-29, and Kerry’s advantage among them was +9%. In the Post’s poll that age group was 21% of the sample.

Oversample 18-29 year olds?  Nonsense.  Again, the population of “adults” is not the same as voters.  The Post/ABC poll essentially matches the 2000 US Census, which shows 18-29 year olds as 22.2% of those over the age of 18. 

And finally…

4. Income Level of Respondents –  Next take into consideration the annual income of the Post poll’s respondents. In 2004, 45% of the electorate was making under $50K, and voted for Kerry 55-44%.
But in the Post’s poll, 55% of respondents make under $50K. That’s a
huge jump in likely Democratic voters among the Post poll’s respondents.

5. Religion – Next, let’s look at the religion of Post poll respondents. In 2004, 54% of voters described themselves as “Protestants” and voted overwhelmingly for Bush (+19%).
In the Post poll, only 47% of respondents were “Protestant”. Also, in
the Post poll 14% of respondents had “no” religion, while in 2004, only
10% of voters had “no” religion, and they voted overwhelmingly for
Kerry (+36%). Catholics are also underrepresented by 4% in the Post
poll, another group that went for Bush in 2004.

ABP persists in finding fault with a survey of adults for not matching an exit poll of voters.  Again, these populations are different.  MP has not searched for comparable survey results for religion and income, but gives this warning to those who do.  Questions about income and religion use different language that often yields inconsistent results.

The point here:  MP sees nothing wrong with scrutinizing a poll’s demographics, but one would expect allegations of a “slanted” or “flawed” poll to have some basis in reality.  These do not. 

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.