I thought that in light of the interest in the last post on the recent Schiavo polls that it would be good to take a step back from the microanalysis of and write generally about how pollsters write questions about issues and public policy. I was pleasantly surprised to find that our old friend Professor M, a member of the Political Science faculty at a small midwestern college, had posted some comments that accomplished much of this task for me, and said it better than I would have. For those who do not browse the comments section, his comments are more than worthy of promotion to the main page. This is today’s must read:
Mark, I think that your discussion here implicitly endorses a
commonly held error about the best way to interpret polling data about
matters of public interest. (And this error underlies the criticism of
the ABC poll as well.)The error is the incorrect belief that there is a "right" or
"unbiased" way to ask a question about any given public issue. There is
no such thing. Everyone who works within the polling field is well
aware that small changes in wording can affect the ways in which
respondents answer questions. This approach leads us into tortuous
discussions of question wording on which reasonable people can differ.
Further, as you have pointed out many times in the past, random
variation in the construction of the sample or in response rates can
skew the results of any single poll away from the true distribution of
opinions in the population.So how do we look at public opinion on an issue such as the Schiavo
case? The answer is NOT to find a single poll with the "best" wording
and point to its results as the final word on the subject. Instead, we
should look at ALL of the polls conducted on the issue by various
different polling organizations. Each scientifically fielded poll
presents us with useful information. By comparing the different
responses to multiple polls — each with different wording — we end up
with a far more nuanced picture of where public opinion stands on a
particular issue. If we can see through such comparisons that stressing
different arguments or pieces of information produces shifts in
responses, then we have perhaps learned something. Like our own
personal opinions, public opinion is not some sort of simple yes/no set
of answers; it is complex, and it can see both sides of complicated
issues when presented with enough information.If we were to lock pollsters of all partisan persuasions in a room
and force them to pick the "best" question wording on the Schiavo
issue, we might end up with everyone asking the same question, but
overall we would end up with less information about public opinion, not
more. We are better off having the wide variety of different polls,
with questions stressing different points of view on the issues, and
then comparing them all to one another. This is precisely what you do
in your discussion of the ABC poll, but I think you are asking entirely
the wrong question — not "is the ABC wording defensible?" but rather
"what does the ABC poll, when compared to other polls with different
wording, add to our overall understanding of public opinion on this
issue?"Of course, this sort of contextualizing of polling results is
exceedingly rare in the media. Much more common is the front page story
saying "here is our poll, and here is what it found, and it is a true
representation of public opinion" — and by implication, no other poll
matters. Intellectual honesty is trumped by competition. The best we
usually get are vague generalizations of all of the polls lumped
together ("polls have consistently shown disapproval of Congress’
actions"), and even those generalizations almost never appear in the
initial story trumpeting the "exclusive" poll fielded by the
newspaper/network itself.The end result is that even those who pay close attention to the
news media and the chattering classes often have very little real
understanding of how to interpret polls in a thoughtful way — which is
one of the reasons your blog is so valuable.P.S. Polls which attempt to predict election results are a rather
different kettle of fish, for two important reasons: (1) Pollsters have
been experimenting with questions wording for over 50 years and can
keep wording the same regardless of the issues in a race; and (2) There
is an actual real-world "check" on pollsters’ work in the form of the
actual election results. Neither of these characteristics apply to
polling about issues of public interest.Professor M
For those who want to look at all the recent polls on the Schiavo case, the PollingReport provides a great compilation that includes complete wording, sample sizes, interview dates and margins of error.
I’ve got some additional thoughts…but it’s late. More on this topic tomorrow.
Reading a New Blog
Yup. This is why I found the following paragraph from your previous post somewhat wrong:
“ONE LAST THOUGHT: After reflecting on the comments on this post, there is one word I wish I had written differently: “defensible” (as in, “was the language of their question defensible?”). A better word would have been “fair” or as Kaus put it, “reasonably calculated to produce an accurate poll of what people think.”
“Defensible” was the right word originally. Polls are not supposed to be “fair”. They’re supposed to help elicit understanding.
Leave it to others to worry about mau-mau-ing the media to get favorable coverage of favorable poll results. Stick to helping to elicit understanding.
Professor M writes, “I think that you are asking entirely the wrong question — not ‘is the ABC wording defensible?’ but rather ‘what does the ABC poll, when compared to other polls with different wording, add to our overall understanding of public opinion on this issue?'” Far from thinking your question is entirely wrong, I think both questions are important.
If, as critics have argued, ABC’s question actually misled respondents about the facts of the Schiavo case, it would still contribute to our overall understanding, but it would be indefensible to the extent that it was misleading. Or perhaps I should say that reporting the result in the lead would be indefensible. Suppose that ABC asked a question that began with the premise that “Social Security is going bankrupt within 40 years,” and then headlined the result without further comment. That would again contribute to our knowledge of public opinion (although it may well violate AAPOR standards to ask such a question), but the reporting would be beyond the pale.
I agree with Petey that “defensible” was better than “fair.” As Professor M argues in detail, there is no one accurate or right or fair way of posing the question. But precisely because the media do not and will not do a very good job of contextualizing survey findings, I don’t think we can dismiss the defensibility question.
Dr. Lindeman makes a good point. I perhaps overstated the case by implying that decent or “fair” question wording is totally unimportant. However, to suggest that we must accept the inability of media outlets to properly interpret their own data — as you seem to do in the last sentence — is IMHO giving up without a fight.
One of the interesting things about the blog phenomenon is the appearance of a much more robust criticism of the news media than was previously available in wide distribution (and media decision-makers even seems to notice and adjust their behavior occasionally). If through hearing such criticism, news reporters actually manage to find their way to better reporting and greater contextualization in the use of media-based polling, the defensibility question would become less important.
Yeah I said something similar to Professor M, albeit less eloquently:
“More theoretically, any poll result is inherently tied to question wording, question order, response choice, moment in time, etc. so you can then say that asking any question with any set of answers and any order is by definition presenting a ‘biased view.'”
Have there been any polls that simply asked people whether they think the tube should have been removed without giving them *any* background informaton whatever? In a sense, this is the “fairest” solution, since *any* background information can be denounced as selective. (To be sure, such polls would exclude people who had heard nothing at all about the case, but by now you’s have to have been living in a cave for the past two weeks for that to occur.)
David, somebody HAS taken a poll with that question asked, and without a lot of “context”. And the results indicate public opinion is the exact opposite of what one would conclude from the Gallup et al. polls. A John Zogby poll asked the bare question of whether a disabled person like Schaivo “should not have food and water denied”. Only 9% said that the food/water should be cut off. 79% thought it shouldn’t. An article on the poll is at http://www.lifenews.com/bio891.html, and a link to it is in Mickey Kaus’s website kausfiles that our genial host often cites.
Now, I don’t hold Zogby as one of America’s more reliable pollsters (for example, just before the election he guaranteed a Kerry victory). However, the results are there. The margin is so great that even a few points of poll sloppiness couldn’t effect the overall tendency. Makes one wonder why Gallup et al. aren’t asking questions like this.
It goes to reinforce the thesis of Professor “M”–since question phrasing has inherent bias, one should consult a wide range of polls and a wide range of possible phrasings of questions.
Thank you for great web site, i have discovered it recently, lots of good read
CivilWarGuy: “should not have food and water denied” and “should the feeding tube have been removed” would not necessarily get the same result. It seems to me that the second one is more precise.
your blog is cool, never stop blogging.
ok, i know, thanks, share your thoughts for us over and over.