The NYT Reader’s Guide to Polls

Jack Rosenthal, a former senior editor of the New York Times, filled in as the guest Public Editor” this past Sunday and devoted the column to a remarkable “Reader’s Guide to Polls.” The column (which also includes a kind reference to MP’s coverage of the AMA’s online Spring Break study) provides a helpful sampler of the various sources of imprecision in public opinion polls.  It is a worthy general primer, but as with any attempt to condense a complex subject into a very small space, a few items he covered would probably benefit from a more complete explanation. 

One example involves the reference by Andrew Kohut of the Pew Research Center to some findings from their polls on gay marriage:

The order of questions is another source of potential error. That’s illustrated by questions asked by the Pew Research Center. Andrew Kohut, its president, says: “If you first ask people what they think about gay marriage, they are opposed. They vent. And if you then ask what they think about civil unions, a majority support that.”

Those intrigued by that particular finding should definitely download the Pew report from November 2003 that documented the experimental findings.  Here’s the key passage:

Granting some legal rights to gay couples is somewhat more acceptable than gay marriage, though most Americans (51%) oppose that idea. Public views on giving legal rights to gay and lesbian couples depend a good deal on the context in which the question is asked.  On the survey, half of respondents were asked their views on civil unions after being asked about gay marriage, and half were asked the questions in the reverse order.  When respondents have already had the opportunity to express their opposition to gay marriage on the survey, more feel comfortable with allowing some legal rights as an alternative.  But when respondents are asked about legal rights without this context, they draw a firmer line.

This context difference has little effect on core support and opposition to gay marriage itself, which is opposed by nearly two-to-one regardless of how the questions are sequenced.  But opponents of gay marriage are much more willing to accept the idea of some legal rights after they have had the opportunity to express their opposition to gay marriage. The percent favoring legal rights rises to 45% in this context, while just 37% favor the idea alone.  Put in other words, opponents of gay marriage are much more likely to accept allowing some legal rights when they have already had the opportunity to express their opposition to gay marriage itself.

Note also that the Pew surveys have shown a modest increase in support for both gay marriage and civil unions in surveys conducted since 2003.

Another topic, brought to my attention by a very alert reader, concerned this passage from the Rosenthal piece: 

Respondents also want to appear to be good citizens. When the Times/CBS News Poll asks voters if they voted in the 2004 presidential election, 73 percent say yes. Shortly after the election, however, the Census Bureau reported that only 64 percent of the eligible voters actually voted.

Ironically, as the reader points out, the Census Bureau’s estimate of turnout is itself based on a survey, in this case the Current Population Study, which is prone to the same sort of measurement.  Professor Michael McDonald of George Mason University produces his own turnout statistics based on aggregate population and vote statistics.  His estimate of turnout among eligible voters in 2004 was closer to 61%, not 64%.  The Census Bureau explains the imprecision of this particular estimate in a footnote (#2):   

The estimates in this report (which may be shown in text, figures, and tables) are based on responses from a sample of the population and may differ from actual values because of sampling variability or other factors.

The “other factor” in this case is the same phenomenon that results in the over-reporting of voting in the NYT/CBS polling — respondents wanting to be good citizens.  I suppose this example teaches that vetting survey results for publication is not as easy as we might imagine. 

Speaking of which, the Rosenthal piece also included this news:

The Times recently issued a seven-page paper on polling standards for editors and reporters. “Keeping poorly done survey research out of the paper is just as important as getting good survey research into the paper,” the document said.

True enough.  But what standards will the Times now apply?  If that seven page document has been released into the public domain, I haven’t seen it.  ABC News puts its Methodology and Standards guide online.  Why not America’s “newspaper of record?”