And now for something completely different….
Just for a change of pace, let’s shift from a topic as serious as election reform to something as trivial as the People’s Choice Awards. I do so, not to condemn this crown jewel of American pop culture, but to kick-off what I hope to be a running series on Mystery Pollster on the misuse of junk polls.
Last Sunday night, the People’s Choice Awards were presented on live national television. In a surprise twist, Michael Moore’s film "Fahrenheit 9/11" won the award for favorite movie, while Mel Gibson’s The Passion of the Christ" won in the category of favorite drama. What caught MP’s eye was this wrinkle reported by the website GoldDerby.com:
Controversy will certainly erupt after the victories of both films when critics ask: Do the People’s Choice Awards REALLY reflect the views of the American public? Arguably, they did so in the past when winners were determined by a Gallup Poll survey, but voting was switched this year to less expensive — and less scientific — Internet balloting that’s easily manipulated by the zealous political and religious supporters. "Tinseltown has been buzzing about organized campaigns on behalf of Moore’s Bush-bashing ‘Fahrenheit 9/11,’" reports today’s New York Post and gossipmeisters say the same is true about Mel Gibson’s disciples crusading fervently for "Passion of the Christ."
Generally, when the news media describe a survey as "scientific" they mean that it was based on a random sample that aims to represent some larger population. A better word might be "projectable" – a survey or poll based on random sampling (sometimes called "probability sampling") allows for statistical estimates of some larger population. Non-random "votes" may have entertainment value, but they do not allow for such projectable estimates, especially when the voters select themselves. They cannot project estimate the views of some larger population. (The National Council on Public Polls has a list of questions journalists should ask to determine the merits of online polls. It discusses these issues in more detail).
Now, MP has no quarrel with the People’s Choice Awards nor other popular online or call-in "votes" like the All Star balloting for professional baseball and basketball, American Idol or the myriad "non-scientific" online polls that appear daily. MP himself ran such a reader "survey" just before the election (the oh-so serious term is "convenience sample"), that told him only about the roughly 3,000 readers who filled it out, not the much larger number who browsed the site in that period but did not bother to fill out the survey (actually, MP considered the most significant finding that 3,000 readers were willing to complete such a survey at all, but that’s another story. MP further notes that he still owes his readers a summary of those results).
The problem with these "unscientific" surveys is that someone inevitably makes the mistake of treating them as if they were projective random samples. Case in point: Michael Moore, whose soundbite appeared earlier this week in a story by Sandra Hughes on the CBS Evening News (my transcription):
[Hughes:] Filmmaker Michael Moore says the win may be just what he needs to convince Academy Awards voters. [Moore:] "It’s safe to vote for this film, because the People’s Choice is a poll of red state and blue state America."
Not exactly. It was a vote in which anyone living in red state or blue state America, or anywhere in the world for that matter, could choose to participate. It was not, by any stretch of the imagination, a representative survey of Americans, especially if Moore waged a campaign to get his fans to vote for his film. (Moore’s newfound faith in polls is heartening, especially since he was trashing political polls as recently as September: "You are being snookered if you believe any of these polls").
As a public service, MP will continue to report examples of the misuse of "unscientific" polls, trivial and serious. Please email me your nominations for Junk Poll of the…hmm..Week? Month? We’ll have to see how it goes.
Mark, did you read the chapter in Survey Nonresponse (Groves, et al, 2002) about non-response in Web Surveys? Especially the part about the “validation problem.” The conclusion of the chapter?
“The nonresponse process in Web surveys is much more complex than for other survey modes.”
Would you consider the Koufax award nomination a Junk poll? I hope you don’t strip me of my Whizbang Median New Blog of 2004 award! Kevin and the Whizbang folks had many encounters with the “validation problem” – mostly from KOS readers who were sending in bots to stuff the ballot.
Also, what about your poll of readers? Junk poll?
Fun topic – good break!
Rick –
A vote is a vote. Maybe “junk poll” is the wrong title for this topic, which is really about the characterization. For example, it’s junk if someone says the Koufax awards represent “blog readers.” No…just those that vote.
And yes, the MP reader survey was “junk” in that it was not representative of anything except the readers who chose to fill it out. Still useful (to me), just not projective.
I was just playing… good post.
Is there a way to make junk polls projectable? For example, let’s say you collected some demographic information about the respondents, then created a smaller sample from that group that is more reflective of the population at large.
Wouldn’t that make these polls more accurate? The only problem I see is that people might lie about their background or vote twice. Or is there something wrong with that method, because the selection process is made after you’ve collected their responses?
how do i vote for jennifer love hewitt for best actress