Junk Polls: People’s Choice Awards

JunkPolls Legacy blog posts

And now for something completely different….

Just for a change of pace, let’s shift from a topic as serious as election reform to something as trivial as the People’s Choice Awards. I do so, not to condemn this crown jewel of American pop culture, but to kick-off what I hope to be a running series on Mystery Pollster on the misuse of junk polls.

Last Sunday night, the People’s Choice Awards were presented on live national television. In a surprise twist, Michael Moore’s film "Fahrenheit 9/11" won the award for favorite movie, while Mel Gibson’s The Passion of the Christ" won in the category of favorite drama. What caught MP’s eye was this wrinkle reported by the website GoldDerby.com:

Controversy will certainly erupt after the victories of both films when critics ask: Do the People’s Choice Awards REALLY reflect the views of the American public? Arguably, they did so in the past when winners were determined by a Gallup Poll survey, but voting was switched this year to less expensive — and less scientific — Internet balloting that’s easily manipulated by the zealous political and religious supporters. "Tinseltown has been buzzing about organized campaigns on behalf of Moore’s Bush-bashing ‘Fahrenheit 9/11,’" reports today’s New York Post and gossipmeisters say the same is true about Mel Gibson’s disciples crusading fervently for "Passion of the Christ."

Generally, when the news media describe a survey as "scientific" they mean that it was based on a random sample that aims to represent some larger population. A better word might be "projectable" – a survey or poll based on random sampling (sometimes called "probability sampling") allows for statistical estimates of some larger population. Non-random "votes" may have entertainment value, but they do not allow for such projectable estimates, especially when the voters select themselves. They cannot project estimate the views of some larger population. (The National Council on Public Polls has a list of questions journalists should ask to determine the merits of online polls. It discusses these issues in more detail).

Now, MP has no quarrel with the People’s Choice Awards nor other popular online or call-in "votes" like the All Star balloting for professional baseball and basketball, American Idol or the myriad "non-scientific" online polls that appear daily. MP himself ran such a reader "survey" just before the election (the oh-so serious term is "convenience sample"), that told him only about the roughly 3,000 readers who filled it out, not the much larger number who browsed the site in that period but did not bother to fill out the survey (actually, MP considered the most significant finding that 3,000 readers were willing to complete such a survey at all, but that’s another story. MP further notes that he still owes his readers a summary of those results).

The problem with these "unscientific" surveys is that someone inevitably makes the mistake of treating them as if they were projective random samples. Case in point: Michael Moore, whose soundbite appeared earlier this week in a story by Sandra Hughes on the CBS Evening News (my transcription):

[Hughes:] Filmmaker Michael Moore says the win may be just what he needs to convince Academy Awards voters.  [Moore:] "It’s safe to vote for this film, because the People’s Choice is a poll of red state and blue state America."

Not exactly. It was a vote in which anyone living in red state or blue state America, or anywhere in the world for that matter, could choose to participate. It was not, by any stretch of the imagination, a representative survey of Americans, especially if Moore waged a campaign to get his fans to vote for his film. (Moore’s newfound faith in polls is heartening, especially since he was trashing political polls as recently as September: "You are being snookered if you believe any of these polls").

As a public service, MP will continue to report examples of the misuse of "unscientific" polls, trivial and serious. Please email me your nominations for Junk Poll of the…hmm..Week? Month? We’ll have to see how it goes.

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.