Words Worth Remembering

Legacy blog posts Measurement Issues

One of the great rewards of writing this blog is the incredible diversity of its readers — everyone from ordinary political junkies to the some of the most respected authorities in survey research.  I heard indirectly from one of the latter over the weekend regarding my post on NSA phone records issue on the CBS Public Eye blog, and I would like to share his comments.  They underscore the caution we should all have in placing too much faith in any one survey question about an issue of public policy.

There are few academics more respected on the subject of writing a survey questions than Professor Howard Schuman of the University of Michigan.  In 1981, along with co-author Stanley Presser (now a professor at the University of Maryland), he wrote Questions and Answers in Survey Research, a book that remains required reading for graduate students of survey methodology.  After my article appeared on CBS Public Eye last Friday, someone posted a link to the members-only listserv of the American Association for Public Opinion Research (AAPOR).  Schuman read it and posted some thoughts to the listserv that I will reproduce below.

But first, I thought it a little ironic that I had nearly quoted Schuman at the end of the Public Eye post, cutting the key quotation at the last minute only because my piece was already running long.  It came from the seminal article, "Problems in the Use of Survey Questions to Measure Public Opinion," that Schuman co-authored with Jacqueline Scott for the journal Science in 1987 (vol. 236, pp. 957-959).  The article described several experiments that compared results when similar questions were asked using an open-ended format (where respondents answer in their own words) or a closed-ended format (where respondents choose from a list of alternatives). 

The experiments yielded some very big differences, but also revealed shortcomings with both formats.  They demonstrated that both open and closed-ended questions have the potential to produce highly misleading results.  Schuman and Scott concluded with this recommendation (p. 959):

There is one practical solution to the problems pointed to in this report. The solution requires giving up the hope that a question, or even a set of questions, can be used to assess preferences in an absolute sense or even the absolute ranking of preferences and relies instead on describing changes over time and differences across social categories.  The same applies to all survey questions, including those that seem on their face to provide a picture of public opinion.

Schuman’s reaction to my Public Eye piece (quoted with his permission) show that his philosophy has not changed over the years:

Mark Blumenthal’s relearning of the effects of different formulations of
questions is useful, but might go even further to recognize that the
timing of a poll (and a few other features) can also produce quite
different results.  Given polls on any issue, but especially a new one,
we should all keep in mind the old verse about the Elephant, a copy of
which can be found at:

http://en.wikisource.org/wiki/The_Blindmen_and_the_Elephant

Just substitute "attitude" for "theologic" in the last stanza.

The Blindmen and the Elephant is worth the click.  More importantly, Schuman’s words of advice — new and old — are well worth remembering.

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.