Schiavo: The Return of Professor M

Legacy blog posts Polls in the News

I thought that in light of the interest in the last post on the recent Schiavo polls that it would be good to take a step back from the microanalysis of and write generally about how pollsters write questions about issues and public policy.  I was pleasantly surprised to find that our old friend Professor M, a member of the Political Science faculty at a small midwestern college, had posted some comments that accomplished much of this task for me, and said it better than I would have.   For those who do not browse the comments section, his comments are more than worthy of promotion to the main page.    This is today’s must read:

Mark, I think that your discussion here implicitly endorses a
commonly held error about the best way to interpret polling data about
matters of public interest. (And this error underlies the criticism of
the ABC poll as well.)

The error is the incorrect belief that there is a "right" or
"unbiased" way to ask a question about any given public issue. There is
no such thing. Everyone who works within the polling field is well
aware that small changes in wording can affect the ways in which
respondents answer questions. This approach leads us into tortuous
discussions of question wording on which reasonable people can differ.
Further, as you have pointed out many times in the past, random
variation in the construction of the sample or in response rates can
skew the results of any single poll away from the true distribution of
opinions in the population.

So how do we look at public opinion on an issue such as the Schiavo
case? The answer is NOT to find a single poll with the "best" wording
and point to its results as the final word on the subject. Instead, we
should look at ALL of the polls conducted on the issue by various
different polling organizations. Each scientifically fielded poll
presents us with useful information. By comparing the different
responses to multiple polls — each with different wording — we end up
with a far more nuanced picture of where public opinion stands on a
particular issue. If we can see through such comparisons that stressing
different arguments or pieces of information produces shifts in
responses, then we have perhaps learned something. Like our own
personal opinions, public opinion is not some sort of simple yes/no set
of answers; it is complex, and it can see both sides of complicated
issues when presented with enough information.

If we were to lock pollsters of all partisan persuasions in a room
and force them to pick the "best" question wording on the Schiavo
issue, we might end up with everyone asking the same question, but
overall we would end up with less information about public opinion, not
more. We are better off having the wide variety of different polls,
with questions stressing different points of view on the issues, and
then comparing them all to one another. This is precisely what you do
in your discussion of the ABC poll, but I think you are asking entirely
the wrong question — not "is the ABC wording defensible?" but rather
"what does the ABC poll, when compared to other polls with different
wording, add to our overall understanding of public opinion on this
issue?"

Of course, this sort of contextualizing of polling results is
exceedingly rare in the media. Much more common is the front page story
saying "here is our poll, and here is what it found, and it is a true
representation of public opinion" — and by implication, no other poll
matters. Intellectual honesty is trumped by competition. The best we
usually get are vague generalizations of all of the polls lumped
together ("polls have consistently shown disapproval of Congress’
actions"), and even those generalizations almost never appear in the
initial story trumpeting the "exclusive" poll fielded by the
newspaper/network itself.

The end result is that even those who pay close attention to the
news media and the chattering classes often have very little real
understanding of how to interpret polls in a thoughtful way — which is
one of the reasons your blog is so valuable.

P.S. Polls which attempt to predict election results are a rather
different kettle of fish, for two important reasons: (1) Pollsters have
been experimenting with questions wording for over 50 years and can
keep wording the same regardless of the issues in a race; and (2) There
is an actual real-world "check" on pollsters’ work in the form of the
actual election results. Neither of these characteristics apply to
polling about issues of public interest.

Professor M

For those who want to look at all the recent polls on the Schiavo case,  the PollingReport provides a great compilation that includes complete wording, sample sizes, interview dates and margins of error. 

I’ve got some additional thoughts…but it’s late.  More on this topic tomorrow.

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.