Oh Those Pesky Filtered Questions

Legacy blog posts Polls in the News

Josh Marshall flags an error in Wednesday’s Washington Post poll story on Social Security (conducted jointly with ABC).  The story correctly reported that “The president also has at least general support from 53 percent of the public for the concept of letting people control some of their contributions to invest in the market.”  The problem came in the paragraph that followed: 

It is on the specifics that Bush faces problems. Support dropped to an even split when people were told that the cost of the transition to a new program could reach $2 trillion over time, as some forecasts project.

Marshall initially reported on an earlier version of the story which cited specific numbers showing that support “drops to 46% to 47%…when a price tag is put to the plan.”  Those are the numbers you get when you access the “complete data” for this question on the Post’s website.  However, as Josh noted, that page shows the question was asked only of those who support the stock market option.  So the 46% support with a $2 trillion in borrowing was among the 53% that initially supported the program.   

Again, as Marshall and his alert readers noticed, the correct characterization is that support for the President’s plan fell to only 24% (46% of 53%), and opposition increased to 69%, when respondents learned of the plan’s $2 trillion price tag.  ABC’s release and especially its compilation of full results (which shows the “net” calculation combining the two questions) make this crystal clear: 

Note that another question in the survey requires similar math.  The Post story reported correctly that:
62 percent said they would not participate in such a program if it meant their retirement income would go up or down depending on the performance of their stock picks — which is the essence of Bush’s plan.

Thirty seven percent (37%) said they would participate.  The ABC release mentions a follow-up question (that the Post omits):

Among the minority who say they would participate, eight in 10 say they’d invest “some” or a “just a little” of their Social Security funds in the market. Just 19 percent say they’d put in all or most of their available assets [emphasis added].

The full results among the 37% that answered the question:  7% said they would put “all” their social security money in the stock market, 11% “most,” 57% “some” and 23 “just a little.”   Thus, another entirely accurate characterization of the same numbers: Only 7% of Americans say they would invest “all or “most” of their Social Security funds in the stock market if it meant their retirement benefits  would go up or down with the market.

As Josh might say, that’s a pretty small number. 

All of this suggests a few important lessons: 

Always check who was asked the question and how the pollster calculated the percentage (e.g. what was the “base?”).  Follow-up questions like the ones asked above are very common in opinion surveys, as is the sort of mischaracterization on the Post story. 

My best tip to readers is to do what pollsters do. If you can, read the full questionnaire with results filled in before you read the poll story.  Unfortunately, this is not always possible, as many media organizations do not release full results even on the web, but many do.  Here is a list of organizations that typically release “filled-in” questionnaires or the equivalent (the links take you to archives of past data, otherwise links to complete data are usually included in stories posted online.  RealClearPolitics is another good source for links to complete releases):

It is tempting to take this opportunity to share my pet peeves about the way some organizations provide these releases only to paid subscribers or about the way virtually all withhold basic demographic results and screen questions.  However, it’s the holiday season, so I’ll put it this way:  I wish every public survey organization would resolve to emulate in 2005 the releases of the New York Times poll (try this link, or if that doesn’t work, the “multimedia interactive feature” link in the upper right column on this page).  They include the verbatim text of every question, including demographics and complete time series data when available. 

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.