When Respondents Lie

Legacy blog posts Measurement Issues

Yesterday, I talked to two junior high school students doing a school project on political polling.    One of their questions was, "Do people tell the truth when they answer poll questions?"  The answer is, they usually do though there may be times when they do not, especially when the question asks about something that might create embarrassment or what social scientists call "social discomfort."   If you did not vote, for example, you might be reluctant to admit that to a stranger. 

Today’s column from the Wall Street Journal’s Carl Bialik (aka "The Numbers Guy") provides another highly pertinent example (link is free to all):  A survey by Gallup showed that 33% reported giving donations that averaged $279 per household.  Bailik did the math and found that would add up to $10 billion contributed by US households as of January 9.  He also cited official estimates of the total donated by private sources at well under $1 billion. 

The culprit?  Social discomfort:

People tend to fudge when they feel social pressure to answer questions a certain way — in this case, by saying they’ve given to a good cause.

"The interview is a social experience," says Jeffrey M. Jones, managing editor of the Gallup Poll. A USA Today/Gallup survey was the most widely cited source of the one-third statistic in news articles. "As would be the case in a cocktail party or at a job interview, you want to give a good impression of yourself. Even though you can’t see the other person, and may never talk to them again, you want them to think well of you."…

Surveys about charity aren’t the only ones skewed by social pressure. Negative pressure about drug use or sexual behavior can suppress responses on those topics. Conversely, questions about behavior deemed positive, such as voting, tend to elicit false positive responses. Mr. Jones cites a Gallup poll conducted between November 19 and 21 in which 84% of respondents said they had voted a few weeks before. Actual turnout among eligible (though not necessary registered) voters was 60.7%.

Bialik also has a good review of some other methodological explanation for why an initial survey had 45% reporting tsunami relief donations and the second survey a week later showing self-reported donations at 33%, even though both surveys asked the same question with identical language.      

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.