As expected, Washington Post and ABC News continued conducting interviews over the weekend and released complete results Tuesday afternoon. The release last Friday morning was based on the first 505 interviews — the complete sample consisted of 1,103 adults. The overall survey has much bad news for President Bush and the Republicans, as reported separately by the Dan Balz and Rich Morin of the Post and Gary Langer of ABC News. However, the summary of results put out by the Washington Post includes new data on the NSA questions we have been puzzling over for the last few days.
One thing is clear: The new results obtained from Friday through Saturday night are virtually identical to the original Thursday night sample. The latest release from the Post (excerpted below) includes two lines of data for each question: The first line, labeled 5/11, shows the now controversial results from the first night of interviewing. The second line, labeled 5/15, shows the data collected from Friday through Monday night. As the tables show, the results differ by at most a percentage point or two:
The Post summary also breaks out results for party, ideology, vote registration and religion so that we can compare the Thursday night interviews to those collected since. There are a few small differences — the Thursday night interviews have slightly more respondents reporting an income over $50,000 annually, for example — but none that can explain why the Post/ABC results on the NSA records questions differ so much from the subsequent poll questions asked by Newsweek and USAToday/Gallup (see previous discussion of these differences here and here).
The bottom line: These new results help rule out two widely floated hypotheses for the discrepancies. One was that “views changed that much in one day” (as Editor and Publisher speculated Saturday). The new results show virtually no change in the Post/ABC results after Thursday. Another theory was a skewed sample resulting from the relatively small number of interviews conducted in a single evening. The new results show that on party, ideology and religion the Thursday night interviews are well within sampling error of those conducted since.
So with respect to the questions on the NSA phone records database, these new data suggest that the differences between the Post/ABC survey were mostly about question wording and question order.
thanks for the earlier response to my comments about non-response bias. Your analysis is thorough and superb.
What are the implications more broadly of these conflicting results? In the initial Washington Post poll post, you suggested that it could possibly help President Bush, but if you look at Newsweek and Gallup, which yield very consistent results, that doesn’t seem to be the case. What do we know about the public’s opinions on this–do they approve or disapprove, ultimately?
I think the most interesting aspect of this entire controversy is how quickly the USA Today story seems to be unraveling, what with BellSouth — and now, Verizon — fairly emphatically saying that they did not and are not turning over customer call data to the NSA.
Seems like AT&T may still be “guilty”, but it’s clearly not the story that it was last week.
Maybe the telcos are simply telling bald-faced lies now to cover up their sinister actions, just like the Bush Administration habitually does. Whoa, did I really just say that in public?
Mark,
Thanks for the superb attention to this.
You are right that we are now down to question order and question wording, but there is a third subtle factor: the whole questionnaire effect, that is, the other questions that are on the questionnaire that may affect responses to later questions.
Before I get to that, though, a personal anecdote. In my own limited experience in designing and carrying out surveys that seek to evaluate the tradeoff between national security and civil liberties, I have noticed the following: in pre-tests that I have listened to, respondents almost always assume that we are asking only about “the bad guys”. So, for example, if you try out a question about wiretapping, no matter how vanilla the formulation, respondents often ask “you mean wiretapping terrorists, right?”.
As I said, this is anecdotal (pre-tests as you know usually retrieve a small number of respondents), but my intuition tells me that it is a pretty fundamental mindset that respondents bring to the issue.
Now to questionnaire effects. I think readers out there should take the ABC/POST and USA/Gallup questionnaires and lay them side by side. I don’t have time to lay it out here, but what I see is:
-ABC/POST has three questions (43,44,45). Question 43 mentions “war against terrorism”; Q44 mentions “terrorist threats”; Q45 (the money question) is chalk full: “to identify terrorist suspects….to investigate terrorism”. Note how this relates to my observation above: Q45 fairly shouts out the assumption that “only bad guys are being pursued here… suspects”.
-Gallup/USA Today: again, 3 questions. Q1 mentions “fight terrorism” {as opposed to “war against terrorism” above}; Q2 mentions “to investigate terrorism”, but also “billions of phone calls dialed by Americans..”; and Q3 is a generic approve/disapprove question harkening back to Q2…very few cues in Q3.
Now, there are some similarities: Gallup/USA does mention terrorism, like ABC/POST, but I think the ABC/Post cues are much stronger…as excerpted above, and the approve/disapprove question (Q3) is fairly neutral.
But I think that results in ABC/Post are affected by the stronger lead-in (Q43..”war against terrorism”)and by the clear suggestion (in Q45) that it is the bad guys that are being “investigated”. Gallup/USA, on the other hand, is sending the message that any old American might be swept up in this (“billions of phone calls dialed by Americans”).
And…to respond to a comment above, the political implications of this are clear…and also clear in the Bush/Rove strategy. To the extent that the debate plays out as a “terrorist surveillance program” –we are only watching bad guys– it will probably not harm the President (although maybe not help him much either). To the extent that the debate –or future disclosures– reveal that “innoncent” Americans may be caught in the dragnet, I suspect it will hurt.
That we only get break numbers for 5/11 and 5/15 still allows for a weekend polling effect. It would be highly useful a breakdown of the numbers and the sample for each day. Maybe the results would be consistent, but it’s possible that the weekend numbers would show a skew against the program that would be consistent with the other polls (though probably not to the same degree, given the final results).