New Polls, Old Arguments

Legacy blog posts Likely Voters Weighting by Party

New polls are out this morning from both CBS (story, results) and CNN/USAToday/Gallup (CNN story & results, USAToday story) that show continuing low marks for President Bush and eroding views of the Iraq War.  These two new releases provide a good opportunity to review two issues that seem to come up again and again, particularly among partisans who seek to discredit the results. 

1) Despite what you may have read elsewhere, CBS News does not weight its polls by party identification.  Neither do Gallup, the Pew Research Center, Harris Interactive, Time, Newsweek, AP-IPSOS, Fox News, LA Times/Bloomberg or ABC News/Washington Post among others.**  These organizations do typically weight or statistically adjust their samples of adults so to match US census estimates of demographic characteristics like gender, age and race.  The procedure eliminates minor errors in the demographic representation of these samples due to either random sampling variation or response bias.  CBS explained the process here, MP explained it here (at the beginning of a long series on party ID weighting).

What seems to confuse nearly everyone is a table that CBS regularly provides with its releases that shows the number of weighted and unweighted interviews for the three party subgroups (Democrats, Republicans and all others) by which they typically tabulate their data.  Yes, weighting by demographics will typically change the party percentages slightly.  Typically, because telephone survey response rates are slightly lower in urban areas, the weighting typically corrects a slight "response bias" that leads to an undersampling of urban Americans.  These Americans tend to be slightly more Democratic in their politics, so the weighted numbers are typically a point or two less Republican than the less representative unweighted results. 

In releasing party cross-tabulations for every question, CBS provides us with an incredibly valuable tool not routinely available from other public pollsters.  In providing the counts for the weighted and unweighted subgroups, they also adhere scrupulously to both the letter and spirit of standards of disclosure set by National Council on Public Polls (NCPP).  NCPP requires the "size and description of the sub-sample, if the survey report relies primarily on less than the total sample." CBS News goes a bit farther and emulates the practice of academic journals of providing both weighted and unwieighted sample sizes because calculations of sampling error derive from unweighted counts. 

2) Yes, CBS interviews samples of U.S. adults.  So does Gallup.  So do the Pew Research Center, Harris Interactive, Time, Newsweek, AP-IPSOS, LA Times/Bloomberg, ABC News/Washington Post and NBC/Wall Street Journal, among others.

A number of blog sites now routinely seek to discredit results from these national surveys by comparing their party and demographic results to those obtained by the 2004 exit polls (see examples here, here, here and here).   They are quite proud of their "debunking," but the process is akin to carefully cataloging the differences between and apple and an orange and concluding that because an apple is not an orange it is therefore inedible "garbage."  No, surveys of all adults do not represent the demographics of voters, but no one claims they do.   Also, unlike my imperfect apples-to-oranges metaphor, samples of adults include all registered and likely voters.

[For the record, I was in error linking to the post by the blogger Seixon above, as he did not compare the CBS party numbers to exit polls but rather to results from other recent public polls.  My apologies to Seixon and others for the error and also for having been inattentive to the comments for the last 48 hours].   

Yes, most public pollsters will report results among "likely voters" on surveys conducted closer to an election, and most typically report vote preference questions among either registered voters at other times (see for example, questions 2 and 5 on the current Gallup survey).  Yes, samples or registered or likely voters are typically a point or two less favorable to Democrats than the somewhat broader samples of adults, but rarely more than that (see also those same Gallup questions). And yes, campaign pollsters (including MP) whose surveys guide internal campaign strategy typically screen to include only some definition of "likely" voters.

However, the notion that reporting results of adults is some sort of plot to "distort the political landscape to the left’s advantage" or that "what ‘Americans’ or ‘adults nationwide’ think, doesn’t matter one iota in politics, or the polling world" is ludicrous.  The Gallup organization — not exactly a liberal darling — has been reporting results of political surveys for "adults nationwide" since the 1930s.  The National Election Studies conducted by the University of Michigan have reported on all-adult samples since 1948.  This practice as emulated by most of the public pollsters follows the quaint notion that "public opinion" on politics and government encompasses all of the public, not just voters.  How shocking.

And as long as we are on the topic, keep in mind that pollsters disagree with each other (and sometimes even with themselves) on how to select or model "likely voters."  The campaign pollsters who use likely voter screens year round typically use a broader definition than the classic Gallup model, largely to maintain consistency of the sampled population from survey to survey.  At this point in the election cycle, public pollsters avoid such screens and campaign pollsters keep them loose because few of us believe we can predict the ultimate electorate this far out with great precision.  "The polls," Michael Barone noted just yesterday, "whatever their bad news for Republicans, offer few clues about who’s actually coming out to vote."

And even if we could precisely model "likely voters," which particular electorate should public pollsters model?  Keep in mind that we use public polls to make historical comparisons, not just those from survey to survey.  So should pollsters survey only self-described registered voters (typically 80% or so of adults)?  The 60% of adults that voted in 2004?  The 55% that voted in 2000?  Or the 40% that voted in 2002 or in off-year elections generally (who tend to be a bit older and more Democratic than those who vote in presidential years)?  These choices are obviously not self-evident.

Those who warn against treating the current round of polls as infallible predictors of an election that is still eight months away are on solid ground.  However, the efforts of rabid partisans to discredit the current round of public polls as "dirty" or "garbage" are unfair, unfounded and just plain wrong. 

**The ABC News/Washington Post survey does sometimes weight by party in tabulating results among likely voters in the daily tracking surveys it typically conducts in October of presidential election years.

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.