SOTU: On Bounces and Spin

From the polling perspective, if recent history repeats itself, we can assume two things will occur after the President’s State of the Union (SOTU) speech tonight.  First, CNN/USAToday/Gallup and perhaps CBS News will conduct instant reaction polls among speech watchers who will express great enthusiasm for the President and his address.  Second, the traditional poll of all Americans conducted in the weeks following the speech will show little or no “bump” in the President’s job approval ratings. 

MP did a bit of research on this question over the weekend and was somewhat surprised to see a bit of a spin war develop over it.  According to The Hotline, the Republican National Committee distributed an email yesterday from Bush strategist and senior RNC advisor Matthew Dowd that attempts to lower expectations of a Bush bump:

In looking at poll movement before and after State of the Union addresses, the average over the last fifty years is actually a slight drop (-0.2%).  President Bush’s average change is also a drop (-0.4%).  Only one of his SOTU addresses showed positive movement (2005), which is likely attributed to the intervening events of the 2005 Inaugural and January 2005 Iraqi elections.

In reaction, the Democratic National Committee (DNC) put out a counter release claiming that yes-there-is-so a SOTU bounce: 

[Dowd’s] assertions are hardly credible given the press accounts following the last four Bush State of the Union addresses…

President “Always” Gets A Bump From State Of The Union. During an appearance on CNN, Rick Dunham, White House correspondent for Business Week Magazine, noted that, “And if you look at the CNN- Gallup polls, (the) president always gets a big bump out of State of the Union. There hasn’t been a bad State of the Union speech, according to instant reaction, for — I mean, for all these years.” (CNN, Reliable Sources, 2/3/02)

2005: “Poll: Bush wins converts among speech-watchers” (CNN.com, 2/2/05)

2004: “Speech Watchers React Positively to Bush’s Message” (Gallup Poll News Service, 1/21/04)

2003: “Poll: Bush Gets Boost From Speech” (CBS, 1/29/03)

2002: “Bush, sustaining stratospheric poll ratings after Tuesday’s State of the Union address?” (Washington Post, 2/2/02)

So who’s right?  Well, in a sense, both.  Here’s the story: 

Dowd’s numbers come from a very helpful analysis released just before last year’s speech by Gallup analyst Jeff Jones.  Gallup today released an updated version (which should be free to non-subscribers for the next few days) that shows that Presidents historically get little or no “bump” in their approval ratings following a State of the Union address:

Historical Gallup findings dating back to President Jimmy Carter’s administration indicate that presidents rarely are able to increase their popularity following a State of the Union address. George W. Bush may have done so temporarily with his speech last year, but his public standing was largely unchanged following his three prior State of the Union addresses.

Here is a slightly compressed version of the numbers in the Gallup analysis.  Changes of four or fewer points would not be statistically significant:

MP has one minor quibble:  The apparent six point gain in Bush’s job rating last year looked more like a statistical outlier than either a “temporary” bump, or as Dowd put it something “attributed to the intervening events.”  Check out the Prof. Pollkatz graphic summary of all Bush job ratings (you will need to click to enlarge and see the detail, click here for the Prof. Pollkatz original).  I circled the period just before and after the 2005 SOTU.  The little purple diamond that stands out above all the other points is the Gallup 2/4-6/2005 survey that showed Bush’s approval rating at 57%.  Put simply, no other poll conducted at the same time, including another survey that Gallup conducted immediately afterwards showed Bush’s rating to be quite that high (the 2/7-10/2005 Gallup poll had Bush’s approval rating at 49%).

The one immediate and sustained bump in the Gallup data was for Bill Clinton’s 1998 SOTU address.  Check out the Clinton ’98 bump in another Pollkatz graphic just below (original here), but keep in mind why that was a very unique event.  The Monica Lewinsky story had broken just a few days before.  The day before that speech, Bill Clinton faced the cameras and delivered his infamous “I never had sex with that woman” quote.  MP cannot find the ratings for that speech, but interest in the speech was certainly high.  Ironically, the reaction to Clinton’s performance – seemingly unfazed by the scandal erupting around him – help boost his numbers in a way that persisted until the impeachment trial ended with an acquittal. 

So what generated all those headlines speaking of a bump in the DNC release?  Those came mostly from the instant reaction polls conducted by Gallup and CBS News.  Consider how these are done.  Gallup included two questions on their recent surveys of adults that presumably looks something like these from 2003:

As you may know, President Bush will present his State of the Union address in a speech to Congress next Tuesday night, January 28th. The speech is scheduled to begin at (time) in your area and will be shown on all the major television networks. Do you think you will watch that speech, or not?

[IF YES, ASK:] Gallup will be conducting a very short survey next Tuesday night, immediately after President Bush’s speech. If we wanted to include you in the survey after the speech, would it be all right to call you back at that time?

Then after the speech tonight they will call back all those who answered “yes” on both questions and interview those who watched.  In recent years, CBS news has used essentially the same approach, except they interview debate watchers before and after with the “projectable” Knowledge Networks panel (which we have discussed previously in the context of post debate polls and the last fall’s elections in California).

The results on the Gallup survey have been very consistent over the years.  As the table below shows, virtually all debate viewers have a positive reaction to the speech:

Also, agreement that the president’s policies are “moving the country in the right direction” always increases by at least ten percentage points (more complete data can be found in the report on last year’s survey by Gallup’s David Moore).

So why a “bump” in the instant reaction surveys but not in more rigorous polls of all Americans?  A big clue comes from the self-reported partisanship of those surveyed immediately after the speech (as reported in Jones’ analysis).  It always seems to attract a disproportionate number of partisans from the president’s party.  The 2005 speech had the most Republican leaning audience yet.  MP can only speculate, but my guess is that this change had mostly to do with the ever shrinking network audience.  As interest in the SOTU falls, the President’s fans are increasingly the only ones who make the effort to tune in. 

The instant polls that will be done tonight will no doubt provide interesting results.  As Gallup and CBS can compare the results to similar polls done with comparable methodologies in past years, there may be findings of interest.  However, before jumping to conclusions about the real impact on public opinion, consider this advice from ABC polling director Gary Langer quoted in today’s The Note:

1. “Partisans watch these things; rather than torturing themselves, people who don’t like the guy can just turn to another of their 100 channels. When we polled on the SOTU in 2003, we found that the president’s approval rating among speech watchers was 70 percent, versus 47 percent among those who didn’t watch. As we put it at the time: ‘Simply put, people who don’t like a particular president are considerably less apt to tune him in.'”

2. “These speeches tend to be composed of poll-tested applause lines, so the people who watch are already predisposed to like what they hear.”

“We haven’t done immediate post-SOTU reax polls in years (pre-war 2003 was an exception) because, given 1 and 2 above, they are so dreadfully predictable.”

So here is a bit of advice to those who spin:  The instant analysis polls may show a positive reaction, but they always do.  Such reaction is not evidence of a “bump.”  To know if this year’s speech breaks precedent and moves public opinion in a significant and lasting way, we will just have to be patient and wait.

UPDATE:   See my subsequent posts on this year’s results, the way the networks characterized those results and what the Nielsen ratings had to say about the audience size.