Another Day, A Very Different Result

Divergent Polls Legacy blog posts President Bush

Well, another day, another — very different — set of poll results on the NSA phone records surveillance issue.  On a new survey out today from Newsweek (story, results) 53% of Americans said the “program goes too far in invading people’s privacy” while 41% agreed with the alternative statement, “it is a necessary tool to combat terrorism.”  A similar question on the ABC/Washington Post released yesterday showed 63% endorsing the NSA program as “an acceptable way to fight terrorism,” 35% found it unacceptable.  So what gives?

The Post/ABC poll, conducted entirely on Thursday night, sampled 502 adults nationwide, with a margin of error of  +/- 4.5%.  The Newsweek poll was conducted over two nights — Thursday and Friday — and sampled 1,007 adults, with sampling error reported as +/- 4%.  Clearly, sampling error alone does not explain the difference.

What about the extra night of interviewing?  A story posted by Editor and Publisher today concluded: 

Most likely views changed that much in one day after more negative media reports (including many from conservative commentators such as MSNBC’s Joe Scarborough) surfaced. The Washington Post survey took place before many Americans had heard about, or thought about, the implications.

While attitudes on this issue are undoubtedly evolving as Americans learn more about it, I am skeptical that attitudes shifted quite so dramatically in a single evening.  Given the overlap of interviews on Thursday night, an enormous change would have been necessary to explain the difference between the two polls.  While two-night polls like the one Newsweek does are not designed to track day-to-day changes, a change that big between Thursday and Friday would have been hard to miss.  If the two nights showed different results, the Newsweek story makes no mention of it.

Also, if criticism from prominent conservatives fueled any such shift, as per the Editor and Publisher speculation, I would expect to see the biggest differences between the Newsweek and ABC/Post poll occur among Republicans.  Yet as the table below shows, the difference was mostly constant across the party subgroups.  If anything, the difference was slightly greater among Democrats.

More often than not, the text of the questions is the culprit for this sort of difference.  But compare the two questions below.  They are remarkably similar, at least at first glance:

ABC/Washington Post :  It’s been reported that the National Security Agency has been collecting the phone call records of tens of millions of Americans. It then analyzes calling patterns in an effort to identify possible terrorism suspects, without listening to or recording the conversations. Would you consider this an acceptable or unacceptable way for the federal government to investigate terrorism? Do you feel that way strongly or somewhat?

63% acceptable

35% unacceptable

2% no opinion

Newsweek:  Now on another subject . . . As you may know, there are reports that the NSA, a government intelligence agency, has been collecting the phone call records of Americans.  The agency doesn’t actually listen to the calls but logs in nearly every phone number to create a database of calls made within the United States.  Which of the following comes CLOSER to your own view of this domestic surveillance program:

41% It is a necessary tool to combat terrorism

53% It goes too far in invading people’s privacy

6% no answer

The two questions are not identical, of course.  “Acceptable,” as posed by ABC/Washington Post may have been an easier test to pass than “necessary” as posed by Newsweek.  The Newsweek version also makes the “unacceptable” side of the choice a bit easier (“goes too far in invading people’s privacy”)  and describes the program as involving “domestic surveillance.”  But had I seen both questions in advance I would not have guessed these minor variations in wording would have produced such large differences. 

A better [Another] explanation may be the context in which the questions were asked.  The Post/ABC poll involved just six questions, all of which touched on the issues of terrorism, privacy and the NSA phone records search.  The question above was asked fourth, probably within the first minute or two of the call.  [The three questions on the NSA phone records issue] followed immediately after a forced choice question about the trade-offs between protecting privacy and the investigation of “terrorist threats.”

[Correction:  I overlooked the question numbering on the ABC/Washington Post release.  The three questions specific to the NSA phone records issue were numbered 44 through 47, so the ABC/Post poll was probably of the same length as the Newsweek poll with both sets of questions following roughly the same number of questions.  I obviously do not know for sure, but it appears that the ABC/Post poll went into the field on Thursday night on a poll that remains in the field as of this writing].

The Newsweek poll involved more than 40 items and asked the NSA question at the end of the interview, after eight to ten minutes spent asking about other topics (that’s an educated guess based on the number of items).  These included questions yielding sharply negative job approval ratings for the President on issues like the economy (59% disapprove), Iraq (62%), health care (62%), Social Security (60%), the budget deficit (70%), immigration (61%) and gas prices (76%).  The Newsweek pollsters also asked an unusual question of the 48% who said they thought President Bush’s job performance had “gotten worse” in office.  On Q11, they read a list of ten possible reasons why their opinion may have soured on Bush – including “the warrantless wiretapping that Bush authorized” and the criminal charges against Tom Delay, Jack Abramoff and Scooter Libby – and asked respondents to identify which were important in shaping their opinions. 

So in short, the ABC/Post poll asked its NSA question right out of the box, following three other questions on terrorism and privacy, while the Newsweek question followed a long review of the full gamut of issues, with an extra reminder of the various negative stories about the Bush administration for those already inclined to dislike him.  Call it “priming” or “framing” or whatever you will, but the context in which the NSA questions were asked on these two polls was wildly different.

[Correction:  It appears that both surveys asked the NSA questions after a long review of other  issues, although we do not yet know the other questions asked on the ABC/Post poll.  The context of the key NSA records questions on the poll were still different, but not nearly as different as I had initially assumed.    The ABC/Post question followed two others on privacy and the tradeoff between privacy and the investigation of “terror threats.”   The Newsweek poll’s first question on the NSA phone records issue was the one reproduced above.  Question order effects — or “priming” or “framing” — may have played a role in these differences, although the difference in context may not be quite as stark as I had initially assumed].

Obviously, I am speculating.  And yes, as reader Tano put it in a comment yesterday, questions like these probably tell us about more about “gut-reaction, (or more likely, no informed reaction at all)” than about “fully formed opinion.”  But we need to remember that on complex public policy issues like these, a large portion of the population will remain inattentive and ill informed.  Many will never “fully form” opinions about these sorts of issues.  Yet “gut reactions” to arguments and counterarguments in the midst of political campaigns are the quasi-attitudes that often drive vote choice.   

When public opinion polls probe reactions to complex policy issues, they sometimes get conflicting results because differences in language and context push those without well formed opinions in different directions.  For that reason, the “best question” is rarely just one question or poll.  The most “scientific” approach is to look at many different polls that ask about the same issue in different ways and compare the results (see Professor M’s advice).  In that regard, I would have done better yesterday to heed the comment of reader Nadia Hassan: “I guess we’ll just have to see more polling for a clearer picture.”  Over the next several weeks, we will see many new polls asking about this issue, and this confusing picture should grow clearer with each new survey.

If one of these two polls we have now turns out to be a true outlier, we will know soon enough.  Meanwhile, the safest bet is that reality of public opinion at the moment falls somewhere in between.

UPDATE (5/15)USAToday/Gallup released a new survey this morning that goes into far more depth on this issue and helps explain the conflict between the first two surveys — see my update here

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.