Well, another day, another — very different — set of poll results on the NSA phone records surveillance issue. On a new survey out today from Newsweek (story, results) 53% of Americans said the “program goes too far in invading people’s privacy” while 41% agreed with the alternative statement, “it is a necessary tool to combat terrorism.” A similar question on the ABC/Washington Post released yesterday showed 63% endorsing the NSA program as “an acceptable way to fight terrorism,” 35% found it unacceptable. So what gives?
The Post/ABC poll, conducted entirely on Thursday night, sampled 502 adults nationwide, with a margin of error of +/- 4.5%. The Newsweek poll was conducted over two nights — Thursday and Friday — and sampled 1,007 adults, with sampling error reported as +/- 4%. Clearly, sampling error alone does not explain the difference.
What about the extra night of interviewing? A story posted by Editor and Publisher today concluded:
Most likely views changed that much in one day after more negative media reports (including many from conservative commentators such as MSNBC’s Joe Scarborough) surfaced. The Washington Post survey took place before many Americans had heard about, or thought about, the implications.
While attitudes on this issue are undoubtedly evolving as Americans learn more about it, I am skeptical that attitudes shifted quite so dramatically in a single evening. Given the overlap of interviews on Thursday night, an enormous change would have been necessary to explain the difference between the two polls. While two-night polls like the one Newsweek does are not designed to track day-to-day changes, a change that big between Thursday and Friday would have been hard to miss. If the two nights showed different results, the Newsweek story makes no mention of it.
Also, if criticism from prominent conservatives fueled any such shift, as per the Editor and Publisher speculation, I would expect to see the biggest differences between the Newsweek and ABC/Post poll occur among Republicans. Yet as the table below shows, the difference was mostly constant across the party subgroups. If anything, the difference was slightly greater among Democrats.
More often than not, the text of the questions is the culprit for this sort of difference. But compare the two questions below. They are remarkably similar, at least at first glance:
ABC/Washington Post : It’s been reported that the National Security Agency has been collecting the phone call records of tens of millions of Americans. It then analyzes calling patterns in an effort to identify possible terrorism suspects, without listening to or recording the conversations. Would you consider this an acceptable or unacceptable way for the federal government to investigate terrorism? Do you feel that way strongly or somewhat?
63% acceptable
35% unacceptable
2% no opinion
Newsweek: Now on another subject . . . As you may know, there are reports that the NSA, a government intelligence agency, has been collecting the phone call records of Americans. The agency doesn’t actually listen to the calls but logs in nearly every phone number to create a database of calls made within the United States. Which of the following comes CLOSER to your own view of this domestic surveillance program:
41% It is a necessary tool to combat terrorism
53% It goes too far in invading people’s privacy
6% no answer
The two questions are not identical, of course. “Acceptable,” as posed by ABC/Washington Post may have been an easier test to pass than “necessary” as posed by Newsweek. The Newsweek version also makes the “unacceptable” side of the choice a bit easier (“goes too far in invading people’s privacy”) and describes the program as involving “domestic surveillance.” But had I seen both questions in advance I would not have guessed these minor variations in wording would have produced such large differences.
A better [Another] explanation may be the context in which the questions were asked. The Post/ABC poll involved just six questions, all of which touched on the issues of terrorism, privacy and the NSA phone records search. The question above was asked fourth, probably within the first minute or two of the call. [The three questions on the NSA phone records issue] followed immediately after a forced choice question about the trade-offs between protecting privacy and the investigation of “terrorist threats.”
[Correction: I overlooked the question numbering on the ABC/Washington Post release. The three questions specific to the NSA phone records issue were numbered 44 through 47, so the ABC/Post poll was probably of the same length as the Newsweek poll with both sets of questions following roughly the same number of questions. I obviously do not know for sure, but it appears that the ABC/Post poll went into the field on Thursday night on a poll that remains in the field as of this writing].
The Newsweek poll involved more than 40 items and asked the NSA question at the end of the interview, after eight to ten minutes spent asking about other topics (that’s an educated guess based on the number of items). These included questions yielding sharply negative job approval ratings for the President on issues like the economy (59% disapprove), Iraq (62%), health care (62%), Social Security (60%), the budget deficit (70%), immigration (61%) and gas prices (76%). The Newsweek pollsters also asked an unusual question of the 48% who said they thought President Bush’s job performance had “gotten worse” in office. On Q11, they read a list of ten possible reasons why their opinion may have soured on Bush – including “the warrantless wiretapping that Bush authorized” and the criminal charges against Tom Delay, Jack Abramoff and Scooter Libby – and asked respondents to identify which were important in shaping their opinions.
So in short, the ABC/Post poll asked its NSA question right out of the box, following three other questions on terrorism and privacy, while the Newsweek question followed a long review of the full gamut of issues, with an extra reminder of the various negative stories about the Bush administration for those already inclined to dislike him. Call it “priming” or “framing” or whatever you will, but the context in which the NSA questions were asked on these two polls was wildly different.
[Correction: It appears that both surveys asked the NSA questions after a long review of other issues, although we do not yet know the other questions asked on the ABC/Post poll. The context of the key NSA records questions on the poll were still different, but not nearly as different as I had initially assumed. The ABC/Post question followed two others on privacy and the tradeoff between privacy and the investigation of “terror threats.” The Newsweek poll’s first question on the NSA phone records issue was the one reproduced above. Question order effects — or “priming” or “framing” — may have played a role in these differences, although the difference in context may not be quite as stark as I had initially assumed].
Obviously, I am speculating. And yes, as reader Tano put it in a comment yesterday, questions like these probably tell us about more about “gut-reaction, (or more likely, no informed reaction at all)” than about “fully formed opinion.” But we need to remember that on complex public policy issues like these, a large portion of the population will remain inattentive and ill informed. Many will never “fully form” opinions about these sorts of issues. Yet “gut reactions” to arguments and counterarguments in the midst of political campaigns are the quasi-attitudes that often drive vote choice.
When public opinion polls probe reactions to complex policy issues, they sometimes get conflicting results because differences in language and context push those without well formed opinions in different directions. For that reason, the “best question” is rarely just one question or poll. The most “scientific” approach is to look at many different polls that ask about the same issue in different ways and compare the results (see Professor M’s advice). In that regard, I would have done better yesterday to heed the comment of reader Nadia Hassan: “I guess we’ll just have to see more polling for a clearer picture.” Over the next several weeks, we will see many new polls asking about this issue, and this confusing picture should grow clearer with each new survey.
If one of these two polls we have now turns out to be a true outlier, we will know soon enough. Meanwhile, the safest bet is that reality of public opinion at the moment falls somewhere in between.
UPDATE (5/15): USAToday/Gallup released a new survey this morning that goes into far more depth on this issue and helps explain the conflict between the first two surveys — see my update here.
You are over-thinking it. The key is simple:
ABC/Washington Post talked about tracking ‘terrorists’ and ‘terrorism’ but said nothing about ‘domestic surveillance’ or ‘within the United States’ and then asked if it was ‘acceptable/unacceptable’
Newsweek talked about ‘domestic surveillance’ and ‘within the United States’ but said nothing about ‘terrorists’ or ‘terrorism’ until the actual question of ‘neccessary to combat terrorism/invasion of privacy’.
The questions appear similiar on the surface – but in fact were phrased such that they pushed substantially different buttons – much like questions 25 and 26 of the May 9th CBS/NY Times poll. That showed how people’s opinions of ‘things going better/worse’ changed dramatically depending on whether or not the question used ‘six years ago’ or ‘before George W. Bush became president’ with a 15 percent swing in Republican ‘worse’ numbers and an opposing 19 percent swing in Democrat ‘worse’ numbers.
Read this Post/ABC question again
“…It then analyzes calling patterns in an effort to identify possible terrorism suspects, without listening to or recording the conversations. Would you consider this an acceptable or unacceptable way for the federal government to investigate terrorism?….”
I once worked for a polling company, after college. But let me say that if I was called and asked this question I would have been angry at the premise of the question.
I was being told by the questioner that this was needed to fight terrorism. I was not allowed to think it through myself. I would not have been given the CHOICE that this was not solely to fight terrorism. I would have been channeled to an answer by the premise of the question to think this program was good for fighting terrorism. For many people it would have been too high a hurdle to decide to disapprove of something that was fighting terrorism.
Essentially I think words “terrorism” in the question preordained the response.
I know that word terrorism was used in the answer in the Newsweek poll, but putting the word in the question treated it as a fact rather then something you could have an opinion about. I think the deck was stacked.
quote:
|”… without listening to or recording the conversations..”|
That’s a strongly biased misrepresentation in the poll question.
There is NO public factual evidence that the NSA is ‘not’ listening/recording to domestic phone calls. That is purely a
subjective assumption by the pollster — but presented as fact to those persons sampled… that likely skews their response, after the pollster comforts them with the “information” that absolutely no one is listening to their personal telephone calls.
The President will not confirm or deny even the existence of this domestic NSA program — no one has factually stated that no NSA listening/recording happens… or any other factual details of that program. IMO it is very likely that calls are being illegally intercepted.
The poll is bad & invalid.
The difference between these results illustrates why I always consider interpretation of polls as more art than science. Clearly the folks responding to the the WaPo poll thought they were being asked something that meant: “Should our government do everything it can to protect us from terrorists?” The people responding to Newsweek thought they were being asked something more like: “do you trust the government will stay out of your business when it tracks terrorists?” Different results. I’d stake a campaign on this.
The WP poll question has a critical difference. It poses the question as: “Would you consider this an acceptable or unacceptable way for the federal government to investigate terrorism?”
The question of privacy invasion, one of the options in the Newsweek poll, is not even an option in the WP poll. So the two polls are asking very different questions.
Some further questions which would surely swing the results are:
If the program in question violates US law, would you still back it?
Do you agree or disagree that such surveilance programs should have oversight to prevent abuse, such as spying on political opponents?
I’m going to renew my (hopefully respectful) plea here about how we discuss and criticize polls.
Let us try to avoid dismissing polls as “bad”, “invalid”, “worthless” or worse unless we have evidence of problems in the sample itself (i.e. non-random sample, lack of independence etc.).
Otherwise, what we’re criticizing is not really the “truthfullness” of the poll, but the clarity and accurateness of how the poll is reported in the general media.
Another way to think about this is that a question is only poorly worded so much as it elicits an opinion that is different from the opinion we claim it is eliciting.
It is important to not conflate our criticism of pollsters and the reporters who communicate their results. Otherwise, I believe we are actually contributing to that which we are complaining.
Ok, back to your regularly scheduled poll analysis…! 🙂