Another Online Poll on Online Activity

Internet Polls Legacy blog posts Sampling Error

Last week’s "Numbers Guy" column by Carl Bialik of the Wall Street Journal Online looked at another online survey about an online activity, in this case about online alcohol sales to teenagers.  Bialik noticed something in the release that the other outlets should have at least checked.  The "online" sample was not drawn from all teenagers, but rather from teenagers who had volunteered to participate in online surveys.  It was a non-random online survey of an online activity, something with the potential to create a significant bias. 

I will let Bialik take it from there: 

People who agree to participate in online surveys are, by definition, Internet users, something that not all teens are. (Also, people who actually take the time to complete such surveys may be more likely to be active, or heavy, Internet users.) It’s safe to say that kids who use the Internet regularly are more likely to shop online than those who don’t. Teenage Research Unlimited told me it weighted the survey results to adjust for age, sex, ethnicity and geography of respondents, but had no way to adjust for degree of Internet usage.

Regardless, the survey found that, after weighting, just 2.1% of the 1,001 respondents bought alcohol online — compared with 56% who had consumed alcohol. Making the questionable assumption that their sample was representative of all Americans aged 14 to 20 with access to the Internet — and not just those with the time and inclination to participate in online surveys — the researchers concluded that 551,000 were buying alcohol online.

Bialik goes on to raise even more fundamental problems with the release from the survey sponsor — the Wine and Spirits Wholesalers of America — a group that according to Bialik, "has long fought efforts to expand online sales of alcohol."  For one, their headline claims that "Millions of Kids Buy Internet Alcohol," while even the questionable survey estimate adds up to just 551,000. 

One point not raised in Bialik’s excellent piece is that the survey reports a margin of error ("plus or minus three percentage points").  The margin of error is a measure of random sampling error, which applies only where there is a random sample.  This survey was based on a sample drawn from a volunteer panel, not a random sample survey.  I have raised this issue before (here and here).   Yes, random sample surveys face challenges of their own, but if a sound statistical basis exists for reporting a "margin of error" for non-probability samples, I am not yet aware of it. 

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.