NSA Records Polling – Update

I want to continue to discuss the differences in the polls on the NSA records database issue, especially as new surveys are released.  We have no new polls to consider for the moment, but I do want to pass on a bit of overlooked commentary (sent by an alert reader).  In a live chat yesterday morning, the Washington Post‘s Tom Edsall, after receiving several questions on the Post poll released last Friday, passed along this commentary:

Tom Edsall: The very intelligent and gracious Claudia Deane of our polling staff wrote the follow[ing] in reply to an inquiry:

Yes, our Thursday night poll did show a much more positive reaction than the Gallup poll conducted on Friday and Saturday evenings or a Newsweek poll conducted Thursday and Friday nights.

I don’t think it’s a question of the number of respondents — we do get a larger number of respondents for our 4 day projects, but 502 is a perfectly respectable sample size. The margin of sampling error on a sample of 500 is plus or minus 5 percentage points, not much different than the plus or minus 3 percentage points on a 1,000 person sample.

More likely some combination of 1) a learning curve as people come to understand the scope of the program, 2) question wording, and 3) question order.

We’re looking into all these things now to see what else we can learn. The biggest difference between opinions the two surveys seems to be among Democrats, who were substantially more negative in the Gallup poll.

The table below is my compilation of the results by party released by the three organizations (more details in previous posts here, here and here).  The Newsweek and Gallup results are remarkably consistent and – as Deane noted – the differences were larger among Democrats. 

One other note:  As reader Karl pointed out in a comment yesterday, the Gallup survey was done entirely over the weekend, and the Newsweek survey completed on Friday night. Pollsters disagree on the merits of weekend interviewing, but many – including my own firm – recommend that clients confine interviewing to evenings before weekdays (e.g. Sunday through Thursday).   

As Karl noted, I have written about this issue before, as Karl noted, and his comment got me to go back and re-read my previous posts own on this issue.   On example involves a Newsweek poll from early October 2004 conducted immediately after the first debate that showed John Kerry jumping to a two-point lead over George Bush (49% to 47%).  I wrote: 

We have good reason to be cautious [about this result], however, as Newsweek did most of its interviews on Friday night and Saturday afternoon. According to the Newsweek release, it conducted the survey “after the debate ended” with interviewing done Thursday through Saturday.

Because so many adults are away from home on Friday nights and Saturdays, very [few] survey organizations conduct polls on only Fridays and Saturdays. In my experience, weekend-only interviewing yields respondents that are more aware of current events and political figures even after demographic weighting…

To be clear, I am urging caution about these results, not disbelief. We can have much more confidence when we see results less reliant on Friday-Saturday interviewing. I’m assuming many, many new polls will follow in the next 72 hours.

Good advice then and now.  Keep in mind that other surveys confirmed the Kerry trend picked up by the Newsweek poll, but most showed Bush still ahead, albeit by a narrower margin than before the debate.

But also consider this additional bit of data that I learned about in 2005:

MP has his own doubts about the reliability of weekend interviewing but was surprised to see evidence presented by the ABC pollsters at the recent AAPOR conference showing no systematic bias in partisanship for weekend interviews.  The ABC pollsters looked at their pre-election tracking surveys conducted between October 1 and November 1 of 2004, and compared 14,000 interviews conducted on weeknights (Sunday to Thursday) with 6,597 conducted on Fridays and Saturdays.  Party ID was 33% Dem, 30% GOP on weeknights, 32% Dem 30% GOP on weekends, a non-significant difference even with the massive sample size.

So to be honest, I have no theory to offer regarding what impact weekend interviewing might have had on any of these surveys, except that caution is in order, as it should have been for yours truly last Friday.  Ah…physician, heal thyself. 

Again, we will soon have more new surveys to consider soon.  So stay tuned.