I cannot help adding one more footnote to our discussion about "instant reaction" polls conducted after the State of the Union (SOTU) speech last week. One highly valued MP reader emailed to say I had wrongly implied that the networks had been comparing surveys of all Americans conducted before the speech with surveys of the speech audience conducted immediately afterwards. If readers had that impression, that was certainly not my intent. In both their Internet releases and in their brief on-the-air presentations, both CNN and CBS were remarkably careful to say that the instant analysis surveys (including any before and after comparisons) represented only those who watched the speech.
Regarding the network’s effort to accurately characterize the results, here is how Wolf Blitzer announced the results live on the air on Tuesday night (my transcription – emphasis in the original):
We’ve just done, Anderson, a very quick poll, a CNN/USAToday/Gallup poll, and I want to share the results with our viewers. A couple of footnotes before we get into the results though. This is a poll conducted of 464 adult Americans who actually watched the speech. The interviews were conducted during and after the speech. The poll does not — does not –– reflect the broad, the overall sense of the American people. The poll reflects only those who were actually watching the speech.
Take a look at this: Among those who say they watched the speech the overall reaction, very positive 48 percent, somewhat positive 27 percent, negative 23 percent.
Remember because we are only polling people who were watching the speech, and were therefore interested in the speech, it does tend to skew partisan. More people watching the speech tend to like President Bush than didn’t watch the speech.
Now how did these numbers compare overall with some of the other speeches, State of the Union addresses he has given. The 48 percent very positive compares, in 2004 he had 45 percent thought it was very positive. That was the worst number he had in all of these speeches. The best, 74 percent thought he did a very positive speech right after 9/11 in 2002. That was his best very positive number.
Remember these are numbers that reflect those who watched the speech, largely partisan because the audience would tend to skew towards those who like the President himself.
Surprisingly, Blitzer spent more time clarifying exactly who was interviewed and putting those results into the appropriate context than he did presenting the results themselves. [A footnote on Blitzer’s footnote: MP assumes that Blitzer’s misspoke about interviews conducted "during" the speech, probably reading "during" instead of "before." The Gallup survey interviewed respondents a few days before and then again just after the speech].
And here (via Media Matters) is how Bob Schieffer’s presented the results in the CBS coverage:
SCHIEFFER: We want to give you some idea now how the president’s speech went over with people watching at home. CBS News, with the help of a company called Knowledge Networks, chose, at random, 700 adults who told us they planned to watch the address tonight. We gave them WebTV so they could get on to the Internet and answer our questions about the speech.
Now, remember, this does not necessarily reflect the feelings of the country because, traditionally, we found out, in recent years, more Republicans watch when a Republican makes the speech. More Democrats watch when a Democrat makes the speech.
But, here, were the questions: Did our viewers tonight approve of President Bush’s proposals? Seventy-seven percent approved; 23 percent did not. We asked them what affect the president’s proposals would have on their lives: Fifty-nine percent said they would make their lives better; 9 percent said worse; 33 percent said the proposals would have no effect on them. On some issues, our survey shows the impact of the speech tonight. Fifty-two percent say the war in Iraq has been worth the cost. Just last week, just 45 percent of the same group thought so. And finally, here is a disappointment. When asked if President Bush will be able to accomplish his goals — this will be a disappointment to the president, I should say — 32 percent said yes, only — and 68 percent — 8 percent — said no. They did not think he would be able to accomplish his goals.
The point of my original post on this topic was relatively simple. The speech audiences, as measured by instant reaction surveys, typically react very positively, partly due to their skewed partisanship, but – as explained in detail by Gallup’s Jeff Jones — this immediate reaction seldom if ever translates into a lasting "bump" in the president’s approval ratings. The reasons for this phenomenon are unclear. The pollster’s primary obligation in this case is to clearly identify those sampled as those "watched the speech" rather than "all Americans." As far as I can see, CBSNews, CNN and USAToday did so.
The liberal media watchdog organization Media Matters took issue with how conservative commentator Sean Hannity characterized the results on Fox News. Here is Hannity’s comment:
I think the president, very wisely, last night [during the State of the Union address] just stuck to his guns, and what he believes in, and made his case; and that CBS poll shows that 77 percent of people watching liked his agenda.
According to Media Matters, "Hannity touted poll results that CBS’ own Evening News anchor Bob Schieffer warned his audience may not be reliable." To emphasize the point, their headline declared that Hannity, "ignored CBS’s own caveats on poll’s reliability."
MP sees two problems here. The first is that Schieffer never questioned the "reliability" of his poll. He simply noted that Bush’s audiences have typically been more Republican than Democrat and that the poll did not reflect "the feelings of the country," only those who watched the speech.
The second and bigger problem is that in this brief reference, Hannity characterized the poll accurately. He said only that those who "watched" approved of Bush’s agenda, nothing more. Had Hannity said that most Americans liked the speech or that the poll provides evidence of a coming increase in Bush’s job performance rating among all Americans, he would have been quite wrong. But he didn’t do that.
To be sure, Hannity’s comments would have been more complete had he noted that GOP skew in the speech audience, but his reference to the poll was accurate as far as it went. Unfortunately, MP we cannot say the same for the way Media Matters characterized Scheiffer’s remarks.
Media Matters was more on track earlier last week when they called ABC’s Charlie Gibson to task regarding past SOTU audience polls. Gibson said the past surveys indicated that "maybe [the president] will get a pretty good size boost in his polls from this speech." Gibson’s assertion, as noted here last week, has no basis in the data.
SS
