Has George Bush’s job approval among African Americans really fallen to just 2%? While MP was off atoning for his sins, several national pollsters released new surveys. More on the bigger picture later, but for now let’s focus on that 2% number that comes from the latest Wall Street Journal/NBC poll. While this particular result appeared nowhere in either of the poll summary stories that appeared in the Journal or on MSNBC.com, Tim Russert hyped it with great enthusiasm to a considerably larger audience on the NBC Nightly News with Brian Williams on Wednesday night:
Brian, listen to this. Only 2 percent — 2 percent! — of African-Americans approve of George Bush’s handling of the presidency — the lowest we have ever seen in that particular measure.
WashingtonPost.com’s Dan Froomkin (who also transcribed the above) contacted Democratic pollster Peter Hart, who regularly conducts the NBC/WSJ poll with Republican Bill McInturff . That support, according to Froomkin, registered at 19% “as recently as six months ago.” Hart, Froomkin reports, had “never seen such a dramatic drop in presidential approval ratings, within any subgroup.”
Now, the full results posted online by the Wall Street Journal do not include either a cross-tabulation of the job rating by race or the racial composition of the poll sample. However, Froomkin reports the following:
This latest poll included 807 people nationwide, and only 89 blacks. As a result, there is a considerable margin or error — and the findings should not be considered definitive until or unless they are validated by other polls.
How big is that margin of error? Assuming simple random sampling, the standard 95% confidence level used in most public opinion surveys, the maximum margin of error on 89 interviews is 10%. However, those who took statistics may remember that the maximum error range applies for percentages near 50%, it gets smaller as percentages approach 0% or 100%. If one plugs 2% and n=89 into the standard formula for the margin of error, and I get +/- 3%. However, statisticians have long debated whether we should ignore the sharp reductions in the margin of error at very small percentages. Regardless, media pollsters are typically cautious about reporting on results from sample sizes of less than 100 interviews.
That is one reason why national media pollsters typically conduct special “oversamples” of African Americans when they want to investigate racial differences in attitudes. They weight those additional interviews back to match census estimates of the racial composition of American adults for their overall tabulations. However, the extra interviews give them more confidence in crosstabulations by race. Gallup, CBS/New York Times and ABC/Washington Post did just that with their post-Katrina polls last month.
MP also wonders about the 89 interviews of African Americans that Froomkin reports. Is that a weighted or unweighted total (and thanks to a loyal but anonymous MP reader that emailed with that question)? MP suspects it is weighted and that the actual number of African American interviews was a smaller number than 89.
Here’s why: 89 interviews amounts to 11% of a sample of 807 adults. Eleven percent happens to be the percentage of African-Americans among US adults (as estimated by the 2000 Census). Most national pollsters weight their samples to match that estimate (see the percentage black in the last Washington Post poll). Response rates tend to be lower in urban areas, and as a result, unweighted national samples typically under-represent African Americans. As such, most national pollsters weight African-Americans up slightly to match the estimates provided by the US Census. Thus, the odds are good that the WSJ/NBC poll did not come in at 11% black unweighted.
But MP is burying the lede. Rather than speculate, we can simply check this result against other national polls conducted in the last week to see if the NBC/WSJ poll was a statistical outlier. Dan Froomkin found one:
Late Update: The Pew Research Center is just out with its latest poll, which has a larger sample, and it finds Bush’s approval rating among blacks at 12 percent, down only slightly from 14 in July. Here are those results.
Note: Pew’s sample included 135 African Americans and was fielded October 6 through 10. It should not be a difficult matter to obtain similar tabulations from some of the other public polls released this week. So stay tuned.
Meanwhile, a note to powerful pundits who present polling data on national television: A sample of less than 100 interviews can be a dangerous thing. Handle with caution, not exuberance.
PS/Irony Watch Update: Few have been more consistently derisive of political polling than Arianna Huffington. In 1998, in a post titled “Margin of Error or Credibility Gap,” she wrote:
Isn’t it time we took pollsters at their word and started treating polling results as estimates? Wouldn’t it be nice to see these “snapshots” posted in the back of the newspaper, next to the astrology charts?
Looks like the Huffington Post isn’t exactly following that advice. As of this writing, the 2% among African Americans result blares from its front page. The HuffPo “Newswire” says [correction: quotes from Dan Froomkin’s lead]: The result “may turn out to be one of the biggest free-falls in the history of presidential polling.”
Uh huh. Perhaps not as “amazing” as the first item on the HuffPo front page (see the screenshot) but still quite amazing.
UPDATE: I’ve posted some additional data and thoughts in a new post – continue reading here.
Bush approval dips again
The latest NBC-Wall Street Journal poll shows President Bush’s approval rating down to 40 percent, a
Ahem. [COMMENT DELETED]
I mean, really! I live in Baltimore, MD and work and live with blacks, and I find it AMAZING that you could be so out of touch with how the failure to save black residents of New Orleans has affected how blacks view the federal government under Bush’s “leadership” (such as it is). To wonder in the abstract about how such a huge drop in support for Bush can happen when there is VERY CONCRETE reasons for such a drop –reasons that are openly stated in the black community– just goes to show that you people live in same pathetic vile bubble that the White House confines itself to. What a bunch of sad sacks!
Dan in Baltimore
“What a strange person.”
Uh… can I agree with Dan Cobb, except, you know, *without* the craziness and personal attacks?
Those poor black people lived in New Orleans when Bill Clinton was president too. You wacko lib..
Here’s a question:
You state: “Response rates tend to be lower in urban areas, and as a result, unweighted national samples typically under-represent African Americans.”
I’ll operate with the assumption that African-Americans in urban areas tend to be less favorable toward this President and republicans as compared to African-Americans in more affluent suburban regions. If we were to break up the African-American sampling into urban vs. suburban response rates, what would the result be?
My guess is that the suburban African-American population in general is being oversampled with respect to the urban population. Thus, the overall African-American response rate (in most polls) may over-rely on a more non-urban and likely more affluent population. Therefore, the base rate of 12-19% that you are comparing the current data to may not be particularly well sampled either. However, I haven’t been able to find this distribution data. Glancing at the Exit Poll data, suburbanites voted 62-48 in favor of Bush, whereas urbanites were 54-46 Kerry. Again, without the cross-tabulation of the data by race (which I’m sure is somewhere in the overall data), it is difficult to say with confidence.
Please note that I deleted the first line of Dan Cobb’s post above, as it managed to be both morally offensive and utterly incomprehensible.
To the extent that I have a comment policy, it is explained in this post:
http://www.mysterypollster.com/main/2005/06/uscv_vs_uscv.html
I agree with Dan Cobb (as amended). The margin of error may be big but hardly any black people like bush.
Mark,
I made the same point regarding the statistics of the MoE when far from 50% in the comments over at Ed Morrissey’s blog. Good to know I am still picking up on some of the same things you are.
But here’s the thing about coming up with +/- 3%. Shouldn’t the number that is plugged in to the formula not be the measured number, according to sampling theory, but the number we do not really know– namely, the percentage in the population that holds the opinion?
Gerry – I think the answer to your question depends on the formula involved. However, to compute the confidence interval around a survey statistic, you only have the survey statistic.
Joe – I agree that “hardly any black people like Bush.” Look at the Pew survey, which says that Bush’s job rating among African Americans is only 12%. That means that the overwhelming majority (e.g. 88%) of African Americans don’t approve of Bush.
My point is that the approval number is more likely closer to 10% than 2%.
Mark,
I mentioned the likely high MOE on my site and it’s fairly prominently featured in a cross-post over at Arianna’s, so the full picture should get displayed.
I’m not sure you’re using all the available information. We have a long history of results from lots of polls all of which indicates that African Americans don’t like Bush much at all. We also have information on how much opinions change in the wake of dramatic events (e.g., 9/11, Carter’s botched Iran rescue). So the real quesiton is: In the context of this climate of community opinion, how likely is a drop of the magnitude the WSJ/NBS poll purported to find in the context of Katrina.
I don’t have the numbers to test this, but the finding seems plausible to me.
Mark and Gerry,
Actually, in this case, we can calculate an exact confidence interval based on the binomial distribution. Assuming that 2% resulted from 2 succeses out of 89 trials, I calculated a confidence interval of (0.002733, 0.078828), compared to the normal approximation of (0.000000, 0.053264).
The fact that both the Pew and the CBS poll that were done right around the same time and which had larger samples of African Americans found that 12% of A-A approve of the job that Bush is doing leads me to believe the number is closer to that than the 2% reported. It is not like 12% is high level of support – that means that for every person in that group who think he is doing a good job there are more than 8 who disagree. Especially compared to the general public.
Beside it is important to realize just how rare numbers under 10% are in questions with yes/no or approve/disapprove responses. I have been doing polling/surveys on all sorts of topics for 13 years and I can’t think of more than a handful of cases where we had splits that extreme.
My previous not should read – It is not like 12% is high level of support – that means that for every person in that group who think he is doing a good job there are more than _7_ who disagree.
Sheesh I must learn to proof read.
Regarding our good friend Ms Huffington I always find it interesting that those who scoff at polling suddenly find faith when the polls turn in their favor.
the fact that Arianna advocates that politicians pay less attention to the polls and the posts a news items about a very news worthy poll isn’t inconsistent. If you can’t see that, you’re simply not a very strong thinker. The items in the “news” section of her blog are news items. That poll–a poll that, as you demonstrate by reporting a conversation about it between Russert and Williams–is news, no matter what Arianna believes the role of polls should be in our politics. There are lots of items on the “news” section of her blog that she probably doesn’t agree with. that’s how news works. understanding that is how strong thinking works. You should try it.
Someone said –
“I agree with Dan Cobb (as amended). The margin of error may be big but hardly any black people like bush.”
Black folks don’t like Bush?! How ’bout you quit cherry picking and put up the poll numbers showing what blacks think of every other individual politician. You might find that Bush getting 12 or 14 percent leans toward the high end of the appreciation scale as far as Blacks and politicians are concerned. Please.
And Dan Cobb.. I don’t know what turnip truck you fell off of.. But when blacks see other backs in situations like those witnessed in New Orleans. That first reaction isn’t “damn those racist politicians…” When you see white folks stuck up on a roof during a flood do you blame their predicament on the intolerance of someone else? No. You wonder why the hell the poor fool didn’t get out when everyone else did. What about all his neighbors heading for the hills didn’t he understand? Why would you think black people would look at it any different? I’ll tell ya why – When you see someone white stuck on a roof – you see a person. When you see someone black stuck on a roof – you see a victim. You following this my man- I hope so. Before you start talking smack about anyone else’s hang-ups – you might want to get real with your own first. People don’t see others as victims because they respect them – they see em as victims because they pity them.
No one wants your pity – black or otherwise.
M
This situation illustrates why it’s important to generate samples of acceptable size prior to reporting results. For example, asking one AA person whether they like Bush and reporting “0% of AA’s like Bush” when she answers “no”, is not a representative sample in (just about) anyone’s mind.
But to a statistician, reporting that Bush’s support is 2% from a sample of less than 100 is just as bad. There is simply too much error around that number to report the result.
For example, let’s say the true sample size was 50 and one person liked Bush (2%). What if you asked five more people and they all liked Bush? You’re now at 6/55 or 11%. The 2% estimate is simply not stable enough to report at that sample size.
However, if the sample instead was 10/500, then it’s a different story. Five more “yes” answers change the tally to 15/505 or 3% – a much more stable answer.
That’s why I teach people they need at least 5 responses (actual, not weighted) per category to begin throwing out estimates (with confidence intervals or not) or making sample to sample comparisions.
Remember, a confidence interval won’t save you from a nonrepresentative sample, and the possibility of getting a biased sample from a small phone survey is quite good. You may be starting with a sampling scheme to generate a representative sample, but if you don’t get enough respondents from that sample plan your results are still likely to be biased in one way or another.
That means very small response rates in one category (yes for Bush) need to be backed up by a larger sample size than those closer to 50%, which are much more stable at smaller sample sizes.
Likewise, any dramatic change in sample results (like from 14% to 2%) would require additional samples to confirm the result. If Katrina hadn’t happened, would the pollsters and media have rejected such a drop, or worked much harder to confirm it? Of course. But given a plausable excuse, it seems they rushed to report what they “knew” was true.
Somebody dropped the ball on this one.
This is baby math. There’s no need to even state more. If one were to objectively view political matters, and religious matters, their opinions on bigotry and “propoganda” would change.
Mike,
Ask and you shall receive. So you think Bush leans toward the high end? Evidence from 2004, when Bush was presumably doing much better among Blacks than now, contradicts your claim. Like it or not, Bush ranks lower than any current or even past politician among African American respondents.
Here are figures from 2004 (accessed at http://sda.berkeley.edu). These data are “feeling thermometers”, 0-100 rankings of candidates.
Average Black Ranking of: