Katrina Polling – More from SurveyUSA

[Support Katrina relief!]

SurveyUSA’s Jay Leve emailed with some strong objections to my characterizations earlier today of their ongoing tracking of public reaction to the Katrina Hurricane and its aftermath.  I have posted those below, and they are worth reading in full. 

Before getting to the substance of Leve’s objections, some new data and a mea culpa.  The mea culpa is that in a sleep deprived haze last night, MP somehow missed the most important links (and information) on the SurveyUSA site.  Instead of a PDF file, they now provide — free of charge — a remarkable interactive page for each “breaking news” survey that shows an analysis, methodological information, complete question text, full results, links to graphs of each question and to crosstabulations of every question by demographic subgroups.   They also included a methdological statement addressing some of the issues I raised earlier:

Like the situation on the ground, which is volatile, public opinion is fluid,  consensus just beginning to form. SurveyUSA attempts here to capture snapshots of a unique moment in time in American history. Some portions of the Southeast were without electricity and telephone service during the field period for this survey; areas without phone service would be under-represented in this telephone survey.

Here are the links I missed for complete results for each of the nightly national tracks on Hurricane Katrina (dates indicate when the survey fielded):

• Wednesday 8/31
• Thursday 9/01 – First Poll & Second Poll 
• Friday 9/02
• Saturday 9/03
• Sunday 9/04
• 
  Monday 9/05
• UPDATE: Tuesday 9/06 – First Poll & Second Poll

Now, most important, here is as summary of trend data for two key questions I discussed in the last post, updated to include results from Sunday and Monday.  The trend in the President’s job rating on responding to Katrina has been a steady decline over the last week, a story not told by any other poll.  Most important (at least to MP) is that interviewers conducted last night tend to confirm the drop in Bush’s rating seen in the polls done in the heart of the weekend.

UPDATE:  The results for Tuesday indicate a seven point improvement in the percentage who believe the federal government is now “doing enough” to help the victims of Katrina.  Also, Bush’s job rating increased five points, from 38% to 43%.   The tables below have been updated to include last night’s data.

In his response below, Jay Leve takes strong exception to my description of the various one-night surveys conducted last week as “less than optimal.”  I’ll stand by that label, but concede his point about the potential merits of one-night surveys in the context of a huge breaking story like the Katrina Hurricane.  All things being equal, MP maintains that a survey that persists in re-contacting initially unavailable respondents over three or more nights is preferable to a survey that completes all calls in a single evening. 

But Leve’s point is essentially that all things were not equal this past week.  Public opinion seemed “fluid” enough to justify the compromises of a one-night methodology.  Presumably, the directors of the Washington Post/ABC poll would agree, as they also fielded a one-night poll last week.  The trend line in the SurveyUSA data certainly suggests that attitudes on the Katrina response have been changing and evolving in days, not weeks.

One thing I do want to make clear:  In using the phrase “less than optimal,” I intended no slight on the professionalism of SurveyUSA.  To the contrary, I have been impressed by their professionalism and dedication to the science of public opinion. Yes, Leve and the pollsters at SurveyUSA “push the envelope” of pollster conventional wisdom, but sometimes that wisdom deserves some skeptical questioning.

News junkies may not realize it, but the survey research establishment looks on SurveyUSA and other automated pollsters with great skepticism.  Yet while most of the well-known national pollsters were out of the field over the last week, Leve and SurveyUSA used their surveys to try to demonstrate that their automated methodology provides a better way to track reactions to truly “breaking news,” mostly because they can complete more interviews more quickly for the same or less money.  If the conventional surveys coming soon from CBS and ABC/Washington Post confirm a public souring toward Bush’s handling of Hurricane Katrina over the Labor Day weekend, we will all need to sit up and take notice. 

See Leve’s complete comments on the jump.

Mark,

Your 9/6/05 post reflects an uncharacteristic lack of imagination and rigidity in thinking that in many ways mirrors the government’s response to Katrina. As soon as New Orleans filled with water, SurveyUSA began polling the nation on volatile, fluid and historic events. We wrote questions we could re-ask, day after day, so that answers could be tracked, and so that after a number of days, SurveyUSA would have compiled as good an EKG of America’s reaction to these events as was possible to create. We interviewed 1,200 Americans Wednesday 8/31 in a deliberately compressed field period, so that everyone who took the survey was reacting to the same set of  images (as opposed to some being interviewed before President Bush spoke to the nation and some after).

We interviewed another 1,200 nationwide on Thursday 9/1, another 1,200 on Friday 9/2, another 1,200 on Saturday 9/3, another 1,200 on Sunday 9/4 and another 1,200 yesterday Monday 9/5. Our TV stations across the country are airing these results now. Today, Tuesday 9/6, we will ask the same questions to another 1,200. These opinion polls capture the nation’s reaction to this storm with breathtaking clarity. I cannot imagine how any pollster’s response, using any methodology, could have been more professional, nimble or exhaustive, given the fast moving events. Further, the results that SurveyUSA released to its clients, which are publicly available, were unprecedented in their scope, breadth and depth.  Interactive Tracking Graphs allowed journalists and scholars to study how whites and blacks … young and old … male and female … reacted to the images from the Gulf. Wholly separately, SurveyUSA released a 2nd tracking poll on Thursday 9/1 that touched on looting and price gouging, issues not addressed in our day-by-day Katrina tracking.

Today 9/6 we are in the field with still another (3rd) nationwide survey: this one will rank the job performance of the 7 key players in the Hurricane story: Nagin, Blanco, Barbour, Chertoff, Brown, Honore and Bush. Our clients will have those results tonight. (Wholly irrelevant to this post, but importantly,

SurveyUSA had within 90 minutes of Justice Rehnquist’s death Saturday night written a questionnaire on the chief justice, and that nationwide poll ran Sunday 9/4 — the only poll that was entirely fielded (to my knowledge) by any polling firm after the Chief’s passing but before Bush announced Roberts as the replacement 8 am ET Monday morning. Only SurveyUSA will be able to tell historians who America thought Bush would pick to replace Rehnquist. We had 36 hours to get that poll written and fielded, and we did it.) With each Katrina poll SurveyUSA has released, I personally included a “pollster caveat” in the analysis, clearly stating the limitations of the research.

Polls should be conducted over an extended field period when they attempt to measure something stable, like the incidence of liver cancer. Polls conducted over a long field period, because the researcher has been brainwashed that this is the only way to gather opinion, or polls that are spread out over several days because the pollster does not have the resources to interview more individuals in a shorter time-frame, in response to rapidly changing news events, do nothing more than blur what is happening. They obfuscate what Americans are thinking, rather than capture same in freeze-frame.

“Less than optimal,” you say?

“Nothing short of remarkable,” I say.

Jay H. Leve

Editor

SurveyUSA