Reader BK emailed with a question about a survey of Muslims in the United Kingdom conducted for the London Daily Telegraph by the company YouGov just after the first London bombings. It asked respondents: "Do you think the bombing attacks in London on July 7 were justified or not?" Our reader was horrified that 6% of the British Muslims selected the answer "on balance justified" (11% answered "on balance not justified," 77% answered "not at all justified" and 6% were not sure). BK asks, "Is this a well designed survey? Do you see these results as strong?"
The short answer is that MP is uncertain. The longer answer provides a good opportunity to talk about the ongoing debate over Internet based surveys.
YouGov conducts its surveys over the Internet, using a panel of respondents that agree to be interviewed at regular intervals for some financial incentive. According to British polling blogger Anthony Wells, the company has "used advertising and in some cases even specialist recruitment consultants to try and build a panel that reflects all areas of society." When they conduct a survey, YouGov draws a random sample of its volunteers and weights the results to match the known demographics of the population of Great Britain.
The key issue is that YouGov’s panel is not a probability sample. Probability sampling is the basis for "scientific" polling. Draw a true random sample where every member of the population has an equal probability of being selected (or at least a known probability) and, the results of the survey can be considered projective of the larger population within a statistical "margin of error." But the YouGov panel (like those used in the US by Harris Interactive and Zogby) are not random samples at all. They involve hundreds of thousands of volunteers that "opt-in" to the panel through various sources, most often ads placed on web sites.
Conventional survey researchers argue that polls based on non-probability samples cannot be considered "scientific," that they have no sound theoretical basis. YouGov, and the other online pollsters, argue that the challenges now facing telephone surveys — especially the response rates that typically plunge below 30% on US surveys sponsored by the news media — undermine the theoretical basis conventional polling. They argue that with statistical adjustments, their non-probability samples will yield results that are just as reliable as those obtained with conventional methods.
This debate rages among survey researchers. YouGov has produced notable successes in Britain (see Anthony Wells’ summary), but was way off in its polls of the US presidential election last year. Their "final prediction" had Kerry beating Bush by three points (50% to 47%), an error comparable to that experienced by the exit pollsters [For a more detailed discussion of this debate, see the paper by the noted academics Morris Fiorina and Jon Krosnick, posted on the YouGov/Economist web site].
All things being equal, MP trusts polls that start as probability samples over those that do not. In the case of the survey of British Muslims, however, all things are not equal. Muslims — at 2.7% of the British population — qualify as what pollsters call a "rare population." That means that trying to survey British Muslims with a standard random digit dial (RDD) probability sample is prohibitively expensive. To get a sample of 500, the pollster would need to reach a sample of over 18,500 adults and then hand up on all but the 500 Muslims. YouGov simply sent an email to those in its panel they had already identified as Muslims, and then weighted the results obtained from the 526 that responded "to reflect the Muslim population of Great Britain by age, gender and country of birth."
Of course, an Internet Panel is not the only way to survey a rare population. Interestingly, the British survey organization MORI also conducted a survey of British Muslims in mid July. The MORI online summary says their survey was "conducted on-street and in-home among British Muslims aged 16+" and then, like the YouGov survey, "weighted by age, gender and work status to reflect the profile of Muslims in Britain according to 2001 Census Data." Presumably, MORI sent interviewers to heavily Muslim neighborhoods where they went door-to-door or stood on street corners, using some sort of random method to select respondents. Thus, MORI conducted a probability sample, but the same is only representative of the neighborhoods and street corners they sample from.
MP will not speculate as to which approach is superior. Neither produces a true probability sample of all British muslims, although no such sample is feasible in this situation. The two surveys asked very different questions, so we cannot compare the results looking for differences.
One possible advantage of the YouGov Internet poll is that it might have less "measurement error" on a question like the one that troubled our reader. Ironically, Jonah Goldberg, writing at the NRO’s The Corner wondered about that:
Presumably people who declined to answer or people who shaved their responses did so in order to downplay or conceal their sympathies. I suppose it’s possible that some folks felt pressure from family members to sound more militant thant they are, but I’d have to guess this poll underestimates the problem.
Actually, the fact that it was conducted online probably mitigated the sort of "underpolling" that Goldberg worried about. Consider: We know that respondents will often give a less than truthful answer when the truth might create some social discomfort between the respondent and the interviewer. Of course, the YouGov poll did not involve an interviewer. Respondents replied by computer. So in this case, it is not hard to imagine a British Muslim who believed the attacks to be "justified" responding more truthfully to an impersonal web questionnaire than to a person on the other end of the telephone.
So how reliable is the YouGov survey and what can we make of the results? MP will leave it up to readers to decide, but urges caution. The results are interesting and could not have been obtained by other means, but both the YouGov and MORI polls depart from true probability samples. As such, the results may represent the views of all British Muslims.
Or they may not.
PS: Some additional commentary on this poll by Anthony Wells, Gerry Daly and John O’Sullivan.
MP writes: “… but was way off in its polls of the US presidential election last year. Their “final prediction” had Kerry beating Bush by three points (50% to 47%)…”
Off by 2 pts on kerry and 4 points on bush is considered “way off”?
Blumenthal on the Recent YouGuv British Muslim Survey
At the link. I do not disagree with a single point he makes.
Mark also is kind enough to point to some commentary I made on the poll over at Red State. However, while I do not disagree with the points I raised overall, I think I fumbled a bit at the …
Yes and no.
The 6 point difference in the margin is probably not outside the margin of error, or is really close to it. So certainly the odds of being off by that much in one survey would be less than 1 in 100– definitely something that could be nothing more than bad luck.
The problem is that it was not just one survey. Consistently, throughout the campaign, their survey was showing what their final survey showed– results 3-8 points more favorable in the gap between the candidates than most other surveys were showing.
That’s not bad luck. That’s being off. And yes, by that much consistently is fairly described, to my eyes, as being “way off”.
Rasmussen suffered the same effect with his polling in 2000, albeit in the other direction. His final numbers were not that off, especially when undecideds were considered. But his polls were consistently showing it, which meant it was not just random chance.
“His final numbers were not that off, especially when undecideds were considered”
When I said this, by “not that off” I meant not all that bad. They were off by more than YouGov’s were, IIRC.
Rasmussen in 2000 had Bush leading Gore by nine (!) points–49-40 in its final poll. http://www.newsmax.com/archives/articles/2000/11/7/143653.shtml
That’s “not all that bad”?!
David–
That is a bit further off than I remember, but also helps make my point rather than contradicts it.
The point was that like YouGov, it was not just his last survey. His last survey had been consistent with his previous ones, just as YouGov’s was consistent with their previous one.
That means it is on thin ice to try to suggest Rasmussen being off 9-10 points (with a MoE of +/-3), or YouGov being off 6 points (with a MoE of +/-2), was attributable to them just having an unlucky final sample.
Both surveys were, simply, off. They were off by a substantial amount. They were off by sufficient amounts to be telling a different story from reality. They were off consistently. That suggests that there was something not quite right with the way they were doing things.
Rasmussen did better in 2004, so perhaps he’s worked out the kinks (although he has written that he is somehow controlling to some degree for party affiliation, which concerns me). Perhaps YouGov will, too. But just like after 2000 Rasmussen had a lot to prove, and just like after, well, most elections save 1996 Zogby has had a lot to prove (or should have– strangely people give him a pass entirely too often), YouGov has a lot to prove now.
Gerry
Bloggers love blogging
I’ve been fascinated with opinion formation and surveying since my junior year of college when I took a few courses on them. Especially Electoral Systems and Processes and my independent study. I recently heard about a study in The Telegraph…
MP or anyone else, if you have a second… 🙂
I understand the logic behind using traditional POTS sampling in order to maximize the probability that each person has an equal chance of being selected. From what I understand, this is why telephone surveys are more reliable than internet surveys.
However, telephone surveys must rely upon:
A) Those that have a landline
B) Those that do not hang up when a surveyor calls, which is less than 30% of the population.
While A is somewhat ubiquitous (except for the MTV demographics of 18-24), B seems to inherently de-randomize the sample. If you are only getting below 30% to respond to a poll, isn’t there a personality trait or such common among these specific people that would add additional bias to the survey, the same way that those who are on the internet frequently tend to be younger and more liberal than the general public? Only getting people who both have a landline (older people) *and* who don’t hang up on an unsolicited telephone call seems to me to significantly de-randomize a telephone sample so that its not much better than an internet survey.
But that’s just me, and I don’t know too much about statistics.
Kizzle:
Topic B is the subject of much discussion in the polling world. It came up recently in this post by MP (also check out the discussion in the comments):
http://www.mysterypollster.com/main/2005/06/ideology_as_a_d_1.html
If you’re interested to learn more, check out John Brehm’s book The Phantom Respondents, which examines what sorts of people do–and don’t–respond to surveys, as well as how to correct for response rate biases statistically.
Also check out these studies by the Pew Research Center, each of which compares the results for two identically-worded surveys, one with a standard response rate and one with a higher response rate:
http://people-press.org/reports/display.php3?PageID=813
http://people-press.org/reports/display.php3?ReportID=94
Very offers Office Equipment
[url=http://colobok.host.sk
]Very offers Office Equipment[/url]