‘Twas the Survey Before Christmas…

As things wind down a bit before the holidays, I want to put a little perspective on our recent speculation about the trends in attitudes about President Bush.  It may be helpful to think about two different questions: (1) Have attitudes toward Bush improved since early November?  (2) Have attitudes improved significantly in the week since the election in Iraq?  Surveys have been fairly consistent on the first question (Bush’s numbers have improved), but much hazier on the second.  That haziness stems from the usual imprecision of sampling error and timing but also from something few are talking about:  The challenge of conducting a telephone surveys just before Christmas. 

Have attitudes toward the President improved since early November?  Yes.  Every national poll conducted in recent weeks shows at least some increase in the overall Bush job rating since late October.   Although the gain in approval averages to about four percentage points, it ranges from a low of one or two percentage points (NBC/Wall Street Journal and Pew Research Center respectively) to highs of 8 and 11 points (ABC/Washington Post and Diageo/Hotline).  This change and the wide range of results is easier to see in Prof. Franklin’s chart, copied below

But have views of the President improved significantly over the last week?  That is a much tougher question.  Only two organizations conducted conventional surveys just before and just after the Iraq election.  Gallup showed a statistically insignificant one point drop in the Bush rating (from 42% to 41%), while the just released Zogby survey showed a six-point gain in the Bush job rating (from 38% to 44%). 

Yet as noted here earlier in the week, two surveys – from ABC/Washington Post and Diageo/Hotline – showed much higher approval percentages (47% and 50% respectively) but neither organization has conducted surveys in early December.  Just this morning, NPR released yet another survey, conducted by Democrat Stan Greenberg and Republican Glen Bolger last Thursday, Sunday and Monday, that has the Bush approval rating at 44%. 

The problem is none of these organizations has polled since at least early November (the last NPR survey was in July), so trying to use these results to judge the trends of the last week requires direct comparisons among results from different firms.  That is always a dicey proposition because of the “house effects” that can cause small but consistent differences across polling organizations.

The Rasmussen automated tracking survey complicates the picture further.  During the month of November, the Rasmussen survey crept up from 43% approval to an average of 45% between December 2-13.    Then during December 14-19 — the period when both the ABC/Post and Gallup surveys were in the field — Rasmussen showed approval just a point higher at 46%.  The most recent three day average covering December 20-22 now has Bush up to 50%. 

Professor Franklin has much number crunching and discussion on this subject as well as some new thoughts about the number of days between polls from any one organization can confuse our interpretation of their results.  Over at “The Fix” at WashingtonPost.com, Chris Cillizza has some thoughts as well.   However, MP would argue that given the small number of before and after polls that allow for true apples-to-apples comparisons, it is hard to conclude much except that Bush’s overall numbers have probably increased again slightly since last week’s elections in Iraq.

But again, we should try to avoid getting so fixated on the day-to-day or week-to-week numbers that we lose sight of the larger picture of public opinion.  First, the recent improvement in the overall Bush ratings are small compared to the year long decline in evidence in the chart above.  Second, on the subject of Iraq, the polls that have tracked attitudes over the course of the year show that attitudes on the whether the war was a mistake or was worth the cost have not varied significantly over the course of the year. 

The data and analysis on Iraq is rich and not easily distilled in a quick blog post, so I would urge interested readers to review the reports from the Pew Research Center and ABC News which go into great depth on attitudes in Iraq.  And if you have time for a third, try the analysis from Gallup (if you have a subscription) or scan their full results (as posted in USAToday). 

Pre-Christmas Effect?

Another important issue to consider is that the challenges facing telephone surveys may be greater at this time of year.  How much greater is a source of some debate among pollsters.  The firms I have worked with have typically advised our clients against fielding polls in the latter half of December (so I have little personal experience with surveys conducted in this period).   Almost every survey organization I know of shuts down altogether between Christmas and New Years.  The reason is that we assume that as the holidays approach, Americans are more likely to be away from home doing shopping, socializing, vacationing or visiting their families.

But in thinking about the recent polls, I realized I cannot remember seeing specific evidence to back this conventional wisdom.  So I went in search of what academic data exists on this subject, and with the help of colleagues on the listserv of the American Association for Public Opinion Research (AAPOR) here is what I can report. 

There are really only two significant studies on this topic.  The first involved research by Gideon Vigderhous published in Public Opinion Quarterly in 1982 (45:2, pp. 250-259).  Vigderhous worked with Bell Canada and he conducted an analysis of year-round customers satisfaction surveys fielded by his company during 1978.  He found that December had the lowest response rates of any month of the year at 40.8%, compared to an average of roughly 57% during the rest of the year.

Twenty years later, five health care researchers who worked on a survey of health issues for the state of Iowa (the Iowa Behavioral Risk Factor Surveillance Survey or BRFSS) set out to gather additional data on this question.  Their study was published in Public Opinion Quarterly in 2002 (Losch, et. al. 66:2, pp. 594-607). They looked at the Iowa BRFSS survey as well as similar health surveys in 29 other states in 1998 and 1999.  They found that the contact and cooperation rates for December were no different than those obtained in other months.  They concluded (p. 606):

These findings cast doubt on the universal recommendation that summer and/or December data collection should be avoided if at all possible.

Unfortunately, this study was limited in one very important way.  All of the December surveys were “completed by the end of the second week of the month” as per the standard practice for most survey researchers.  So the data they examined tells us little about the last ten days or so before Christmas.  “Other studies will need to be conducted,” they noted (p. 604), “to fully explore the impact of any ‘December’ effect.”   Sadly, if any such analysis or research exists, MP is not aware of it (at least not as of this posting). 

Armed with what research we do have, let us consider that the two of the most scrutinized surveys — ABC/Washington Post and Gallup/CNN/USAToday — were conducted not only at the end of the third week of December but also largely over the weekend.   The Gallup poll was conducted on Friday, Saturday and Sunday.  Calls for the ABC/Post survey were done from Thursday through Sunday.  Did the weekend interviewering make it more difficult to conduct the surveys? 

Unfortunately, the academic are cautionary but not conclusive on this topic.  The most recent looked only at the first two weeks in December.  The Canadian study is more alarming but it is now over 25 years old.  Neither study looked specifically at weekend interviewing in late December. 

I posed the question of polling close to the holidays to the Washington Post’s Richard Morin as part of his Tuesday “live chat” on WashingtonPost.com (I was the lucky reader from “Washington DC”).  His answer:

I cannot tell you what the cooperation rate for our latest poll because I have not yet received the data. I can’t wait to see what it was. My guess is that we found it harder to reach people (lots of no answers) and that cooperation was down a bit. Call it the “Shop-Till-I-Dropped” effect. We did pause before going over [it] this weekend, which is when America–including at least one pollster–did much of its holiday shopping. But the value of going into the field after the Iraq election outweighed these concerns.

Asked to explain the differences between the ABC/Post and Gallup surveys, he added:

When I, just for fun, dropped our Thursday interviews and just looked at the results of our Friday through Sunday subsample, I found that Bush’s approval rating was 44 percent. That’s still different but far less startling than Gallup’s 41. In our poll, Bush had good nights on Thursday and Sunday; his worst nights were Friday and Saturday. A weekend-before-Christmas effect? Perhaps, but no one can say for sure.

I raise this issue not to be critical of Morin or the pollsters at Gallup.  They saw the value of tracking attitudes on an important and developing story and decided that those needs “outweighed” whatever methodological challenges may exist.  As we have seen before, pollsters will sometimes compromise a bit in the face of breaking news.  I also have no particular theory as to whether “weekend-before-Christmas effect” somehow improves or worsens the President’s numbers.  However, if the results over the last ten days or so have been a bit more erratic than usual, perhaps some pre-holiday effect is the reason.  Mainly, I wan to issue this caution:  We should assume, until someone produces data to prove otherwise that conditions for telephone polling are more challenging than usual in late December.  Until the next round of surveys appears in early January, we should take the data with a larger than usual grain of salt.   

UPDATE –  Gallup’s Jeff Jones emails about their experience with contact and response rates in December:

The last two years we have seen no appreciable change in any of the rates in Dec. compared with earlier months.  We did see some evidence of a slightly lower contact rate in our mid-Dec polls in 2001-2003, however.