New polls are out this morning from both CBS (story, results) and CNN/USAToday/Gallup (CNN story & results, USAToday story) that show continuing low marks for President Bush and eroding views of the Iraq War. These two new releases provide a good opportunity to review two issues that seem to come up again and again, particularly among partisans who seek to discredit the results.
1) Despite what you may have read elsewhere, CBS News does not weight its polls by party identification. Neither do Gallup, the Pew Research Center, Harris Interactive, Time, Newsweek, AP-IPSOS, Fox News, LA Times/Bloomberg or ABC News/Washington Post among others.** These organizations do typically weight or statistically adjust their samples of adults so to match US census estimates of demographic characteristics like gender, age and race. The procedure eliminates minor errors in the demographic representation of these samples due to either random sampling variation or response bias. CBS explained the process here, MP explained it here (at the beginning of a long series on party ID weighting).
What seems to confuse nearly everyone is a table that CBS regularly provides with its releases that shows the number of weighted and unweighted interviews for the three party subgroups (Democrats, Republicans and all others) by which they typically tabulate their data. Yes, weighting by demographics will typically change the party percentages slightly. Typically, because telephone survey response rates are slightly lower in urban areas, the weighting typically corrects a slight "response bias" that leads to an undersampling of urban Americans. These Americans tend to be slightly more Democratic in their politics, so the weighted numbers are typically a point or two less Republican than the less representative unweighted results.
In releasing party cross-tabulations for every question, CBS provides us with an incredibly valuable tool not routinely available from other public pollsters. In providing the counts for the weighted and unweighted subgroups, they also adhere scrupulously to both the letter and spirit of standards of disclosure set by National Council on Public Polls (NCPP). NCPP requires the "size and description of the sub-sample, if the survey report relies primarily on less than the total sample." CBS News goes a bit farther and emulates the practice of academic journals of providing both weighted and unwieighted sample sizes because calculations of sampling error derive from unweighted counts.
2) Yes, CBS interviews samples of U.S. adults. So does Gallup. So do the Pew Research Center, Harris Interactive, Time, Newsweek, AP-IPSOS, LA Times/Bloomberg, ABC News/Washington Post and NBC/Wall Street Journal, among others.
A number of blog sites now routinely seek to discredit results from these national surveys by comparing their party and demographic results to those obtained by the 2004 exit polls (see examples here, here, here and here). They are quite proud of their "debunking," but the process is akin to carefully cataloging the differences between and apple and an orange and concluding that because an apple is not an orange it is therefore inedible "garbage." No, surveys of all adults do not represent the demographics of voters, but no one claims they do. Also, unlike my imperfect apples-to-oranges metaphor, samples of adults include all registered and likely voters.
[For the record, I was in error linking to the post by the blogger Seixon above, as he did not compare the CBS party numbers to exit polls but rather to results from other recent public polls. My apologies to Seixon and others for the error and also for having been inattentive to the comments for the last 48 hours].
Yes, most public pollsters will report results among "likely voters" on surveys conducted closer to an election, and most typically report vote preference questions among either registered voters at other times (see for example, questions 2 and 5 on the current Gallup survey). Yes, samples or registered or likely voters are typically a point or two less favorable to Democrats than the somewhat broader samples of adults, but rarely more than that (see also those same Gallup questions). And yes, campaign pollsters (including MP) whose surveys guide internal campaign strategy typically screen to include only some definition of "likely" voters.
However, the notion that reporting results of adults is some sort of plot to "distort the political landscape to the left’s advantage" or that "what ‘Americans’ or ‘adults nationwide’ think, doesn’t matter one iota in politics, or the polling world" is ludicrous. The Gallup organization — not exactly a liberal darling — has been reporting results of political surveys for "adults nationwide" since the 1930s. The National Election Studies conducted by the University of Michigan have reported on all-adult samples since 1948. This practice as emulated by most of the public pollsters follows the quaint notion that "public opinion" on politics and government encompasses all of the public, not just voters. How shocking.
And as long as we are on the topic, keep in mind that pollsters disagree with each other (and sometimes even with themselves) on how to select or model "likely voters." The campaign pollsters who use likely voter screens year round typically use a broader definition than the classic Gallup model, largely to maintain consistency of the sampled population from survey to survey. At this point in the election cycle, public pollsters avoid such screens and campaign pollsters keep them loose because few of us believe we can predict the ultimate electorate this far out with great precision. "The polls," Michael Barone noted just yesterday, "whatever their bad news for Republicans, offer few clues about who’s actually coming out to vote."
And even if we could precisely model "likely voters," which particular electorate should public pollsters model? Keep in mind that we use public polls to make historical comparisons, not just those from survey to survey. So should pollsters survey only self-described registered voters (typically 80% or so of adults)? The 60% of adults that voted in 2004? The 55% that voted in 2000? Or the 40% that voted in 2002 or in off-year elections generally (who tend to be a bit older and more Democratic than those who vote in presidential years)? These choices are obviously not self-evident.
Those who warn against treating the current round of polls as infallible predictors of an election that is still eight months away are on solid ground. However, the efforts of rabid partisans to discredit the current round of public polls as "dirty" or "garbage" are unfair, unfounded and just plain wrong.
**The ABC News/Washington Post survey does sometimes weight by party in tabulating results among likely voters in the daily tracking surveys it typically conducts in October of presidential election years.
That was quite dishonest Mark, as I did most certainly not compare the CBS numbers to 2004 election results. I also have never compared polls of adults to those of voters.
The numbers show that CBS always has the most amount of Democrats in their samples (before and after weighting, which I also did not claim was anything other than due to demographics), compared to any other poll of its type.
Being dishonest and claiming I am doing something I am not will not change this.
I also did not allege that CBS must have purposefully gotten bad samples, but it is quite clear that CBS has a problem with getting too many Democrats. They don’t seem very intent on alleviating this problem, as it continues to happen.
All the other polls are more or less consistent – CBS is the outlier.
Would like you like to correct your post?
My post that you linked to says: “Now, CBS weights their sample according to Census data, such as age, sex, race, and so on.”
Also, it never mentions the 2004 exit polls once. Yet this didn’t prevent you from dishonestly claiming: “A number of blog sites now routinely seek to discredit results from these national surveys by comparing their party and demographic results to those obtained by the 2004 exit polls”
We going to see a correction soon? Or would that unfortunately involve you admitting that I did, in fact, debunk CBS and you in your role as CBS-apologist?
Seixon-
First, I think Mark was clearly wrong to imply that you criticized the CBS poll by comparing the party ID numbers to 2004 exit polls. You didn’t actually say that in your post, and I expect Mark will eventually address this (right Mark?).
He might still take issue with your general argument that CBS’s party ID numbers don’t match up with that of other organizations. The basic problem is that there is really no way to know for sure which poll is measuring party ID accurately. In any case, I find it interesting that despite CBS’s consistently different party ID numbers, they’re estimated approval rating differs rather little from the average (about -3 points). In a world of +/-3 margin of error, that’s a pretty small difference.
It is our monomaniacal obsession with approval polls that makes this difference seem huge, when in fact it is rather small, by any statistical measure. We obsess over polls so much that even 2-3 point differences seem important, when really we probably can’t measure differences that precisely.
However, your position that the CBS poll is somehow “wrong” and that their sampling methods need to be “fixed” is incorrect. So while Mark’s specific critique of your post is inaccurate, your critique of the CBS poll is also wrong (in my opinion).
Every polling organization has a bit of a “house effect”, relative to the mean (i.e. averaging all polls together). Some above, some below. CBS happens to be pretty consistently 3 points lower (on average) that other polls. Some organizations tend to overestimate Bush’s approval ratings.
But it does not follow that CBS’s numbers are wrong. They’re just different. And only marginally, at that. There were several other polls recently placing Bush’s approval in the 36-38% range. I have seen no (credible) statistical analysis that would even come close to classifying CBS polls as an “outlier”. I would be genuinely curious to see if you would be prepared to define precisely what constitutes an outlier in this context (a mathematical definition, not a political one) and show how the CBS polls fit that definition. Especially since the +/-3 margin of error leads the CI for CBS’s polls to overlap (significantly) with at least 2 others.
I think the correct viewpoint is that CBS is capturing an aspect of public opinion that isn’t shared by other pollsters, and that the correct way to incorporate this information is to average a variety of polls over similar time periods. In that sense, no single poll is “right” or “wrong”. They all capture a fraction of the truth, to some degree.
I think what partisans _ought_ to be criticising is the non-aggregate presentation of polls by the media organizations that sponsor them. It is just as silly for Pew/Gallup/Fox etc. to claim that their single poll represents Bush’s _true_ approval as it does for CBS to make that claim.
In the end, I think your beef is with the sensationalism surrounding polls in general, not CBS’s polls in particular.
And again, in an attempt to keep this discussion civil, I will reiterate that Mark does indeed owe you a correction (even though I think he will still have a pretty good case against your post).
Cheers!
joran,
CBS’s numbers are (often) outside the margin of error of the other polls. In statistical terms, that is significant. Also, while the other polls are generally similar, CBS is the one who is usually different in both its final approval rating, party ID, and support from people of each party.
Is it just a fluke that CBS has had all the records for lowest approval rating? First and only at 35%, now first and only at 34%.
Let’s just look at the latest CBS and Gallup polls:
CBS: 28.9% Republicans; 34% Democrats; 37% Independent
Gallup: 32% Republicans; 34% Democrats; 33% Independent
Again, CBS either has more Democrats, or less Republicans. Consistently.
A whole 4% difference is a lot, considering the margin of error is around 3-5%. The fact that CBS consistently gets these kinds of results is also alarming. I mean, if it were really just the result of some fluke, then it wouldn’t happen consistently.
Internals, support by party:
CBS: 74% Republicans; 6% Democrats; 28% Independent
Gallup: 75% Republicans; 13% Democrats; 23% Independent
A full 7% difference on Democrat approval, and 5% difference on Independents. That seems quite erratic to me, and makes a big difference when it comes to the final approval rating.
Now, we can’t out of hand say that CBS is wrong, and Gallup is right in this instance. However, as we have seen time and time again, the majority of polls are similar, while CBS is notably different than the rest. You can see this on the last chart on the post that Mark linked to.
I mean, there’s no doubt that Bush is in trouble right now, but CBS has proven to be the black sheep in the poll department for quite some time. Yet as you say, the wisest thing to do would be to average the polling results, which I think Real Clear Politics does. However, from CBS’s polling history, I don’t feel that they have any business even being incorporated into such a measurement.
Whether CBS’s polling bias comes from CBS itself, or from the people who happen to answer them is something only CBS knows.
MP–Joran is right. Seixon never did mention the 2004 exit polls.
Seixon–Joran is right. A 3% difference in approval rating is neither here nor there.
Seixon–in your post that MP links to you mix a national sample with a voters sample when you seek to demonstrate that CBS is out of line. You yourself noted that two polls you compared CBS to (WNBC and Diageo) were of registered voters, so as MP pointed out, were inappropriate as in an “imperfect apples-to-oranges metaphor.”
Andrew,
A 3% difference is significant when the margin of error is exactly that.
I did lump WNBC and Diageo in there, yes, but I also compared it to Cook/RT which was of national adults, and not voters. That was only for support from each party.
When it came to party ID, I only compared the CBS poll to polls like it, measuring national adults.
In other words, the “apples-to-oranges” bit doesn’t fly. I compared apples to apples, as well as apples to oranges.
The recent data I just posted from Gallup also shows that there are significant differences between it and CBS – which means they are not consistent statistically.
Seixon,
You simply can’t meaningfully compare different polls like that, at least, not in the way you’re trying to do it. Read up on “house effects”, for starters, either here, or over at Political Arithmetick, or both. And after all that, if you still think CBS’ results are suspicious, then you must be going nuts over Rasmussen’s results…
Seixon-
I think you might be making a common mistake in evaluating when two poll results are “different”. You must remember that the +/-3 margin of error holds for _both_ polls you are comparing.
This means that for two poll estimates to be (strongly) different, they would have to differ by at least 6 points. In which case the only recent polls that are really _different_ than CBS (from RealClearPolitics) are the last Rasmussen (42%) and ABC/WashPost (41%). And even the second is only barely different.
I find it curious that you latch onto CBS for its routinely -3 difference. Why shouldn’t we reject all Rasmussen polls, since they are routinely 3 points higher than average? By your measure, this makes them a serious outlier, and we should exclude them.
The fact that CBS routinely has the lowest estimates is not evidence that their’s are outliers. Specifically, it isn’t enough to be the lowest; you’d have to be _much_ lower. For example, if anyone came out with a poll tomorrow pegging Bush’s approval at 27% I’d be mighty suspicious. But even then, I wouldn’t dump it until I had others in the same time frame to compare it to.
As for your comparisons with Gallup: Neither of your comparisons are (statistical) evidence that the two samples collected by CBS and Gallup are so wildly different that one must be wrong. Even if the differences you note showed up consistently over time between CBS and Gallup, that would not be evidence that one is “better” than the other. It would just establish the (unsurprising) fact that when different polling organizations conduct polls, they get different answers, that are often different in the same way. i.e. a house effect.
Again, most of the differences you note are not substantial (the difference in D support, 6 vs. 13, is only weakly different). All the rest are within a 3 point margin of error.
CBS polls have a negligible (if any) effect on cumulative averages of polls. Check our the post from March 7th on Political Arithmetik. Removing CBS polls has essentially no effect on his loess (I assume, splining, or something like that) line. Additionally, of the 16 CBS polls there, 10 of them are hitting the average approval rather well. Indeed, the 6 that are clearly low all have a fair bit of company down there; see all the other points around them?
(Here would be a good place to note that your claim that “we have seen time and time again” that CBS results are notably different simply isn’t true. It has happened on occasion, but a solid majority of their (recent) polls have been pretty spot on. Additionally, the table you refer to is misleading. If you continue in time past the three polls you note, you get three successive polls that are all within the margin of error of CBS’s 34%. So you have identified _one_ instance in which CBS either was somewhat low, or CBS was right, and the other polls failed to pick up the trend.)
In short, since CBS polls are not really very different from the rest of polls, as a group, and what differences exist have a negligible effect on average approval ratings, there is really no reason not to include them.
The general rule is “More data = Better”. Which is why I would strongly oppose attacks on Rasmussen as well for having estimates that are too high. They are high, but not outrageously so; they provide a slice of reality that perhaps the rest of the polls are missing. Same with CBS.
But to end on a positive note, after all this disagreeing, we agree that averaging polls is best (oh happy day! We agree!).
My point is that the CBS “house effect” is what their problem is. The Rasmussen polls are not conducted in the same way as the other polls, thus obviously we can’t compare them.
A 3% margin of error means that it is 95% probable that the real result lies within the range, and this gets more and more unlikely out from the central figure they find. In other words, it is very unlikely, according to the Gallup poll, that the true approval rate is 33% (3% lower than their latest finding). Thus, CBS at 34% is barely within that, and likewise for Gallup in regards to Gallup.
However, more important is the internals of the polls. A full 7% difference on support by Democrats is completely out of bounds statistically. The Gallup poll will have Democrat support at 10-16% at 95% confidence. The CBS has this as 3-9%. Thus, the outer edges of the confidence intervals don’t even overlap at all. When you take into consideration that the probability distribution says that at the lower or higher end, that the probability quickly approaches 0%, then you see why this is a problem.
Back with the comparison of Cook/RT and CBS from a few weeks ago, CBS had 72% Republican support, while Cook/RT had 82%. A full 10% difference, well outside all margins of error. This while Cook/RT’s party ID numbers were virtually identical to Gallup’s.
That shows that CBS is consistently in a class of its own. Bringing up Rasmussen, a poll that is conducted and tabulated completely differently (a “pear”, if you will) won’t deflect from that fact.
Seixon-
Sigh. Well, I tried. You’re fundementally misunderstanding the statistics at work here. You wrote,
“A 3% margin of error means that it is 95% probable that the real result lies within the range, and this gets more and more unlikely out from the central figure they find.”
This is simply false, and any stats professor you can dredge up will tell you so. We cannot assign probabilities to the true proportion being anywhere (unless there are Bayesian pollsters out there somewhere…which I highly doubt).
A 95% confidence interval means that if we repeated our poll 100 times (over the same time period) then we’d expect 95 of the confidence intervals to contain the true proportion. But this tells us _nothing_ about _where_ in the CIs the true proportion lies.
There is, most certainly, no “dimishing probability” of finding the true proportion upon moving away from a poll’s estimate. All we can say is that we are confident that the true proportion is somewhere in the CI, and that based on our actual sample, our estimate is our best guess.
Also;
“When you take into consideration that the probability distribution says that at the lower or higher end, that the probability quickly approaches 0%, then you see why this is a problem.”
This is gibberish. Again, unless there is some Bayesian pollster out there, there is no probability distribution associated with our estimate of the proportion.
Finally, we are still only talking about one CBS poll. As I pointed out before, CBS’s estimates are often very much in agreement with other polls.
Ok, and now I have to go explain the exact same things to my actual statistics students…in any case, it was nice chatting. (And I’m still waiting for that correction, MP!)
Seixon — the 3% margin of error would apply to the entire sample. Are you sure that 3% also applies for the subsample of Democrats at the 95% level? Might that subsample not be small enough to increase the margin? regards — andrew
Seixon –
You write:
A 3% margin of error means that it is 95% probable that the real result lies within the range, and this gets more and more unlikely out from the central figure they find. In other words, it is very unlikely, according to the Gallup poll, that the true approval rate is 33% (3% lower than their latest finding). Thus, CBS at 34% is barely within that, and likewise for Gallup in regards to Gallup.
A full 7% difference on support by Democrats is completely out of bounds statistically. The Gallup poll will have Democrat support at 10-16% at 95% confidence. The CBS has this as 3-9%. Thus, the outer edges of the confidence intervals don’t even overlap at all. When you take into consideration that the probability distribution says that at the lower or higher end, that the probability quickly approaches 0%, then you see why this is a problem.
These statements are not true for several reasons.
First – the margin of error when comparing two samples is not the same as the margins of error for each sample.
Second – the margin of error will be greater for any sub-samples.
Third – you cannot assume that Democrats in one survey are the same as Democrats in a survey from a different polling organization.
And fourth – your assumption that the proportion obtained from one sample somehow represents the central figure of the distribution of all possible samples is incorrect.
I realize this is the Internets, but your analysis is flawed because of your misuse of elementary sampling theory.
The CBS poll does not *currently* seem to be an outlier. It is true that it has Bush approval slightly lower than the CNN-USAToday-Gallup or AP-Ipsos polls. But it also has his *disapproval* slightly lower. So the gap between Bush approval and disapproval is 23 in CBS, 23 in AP-Ipsos , 24 in CNN/USA Today/Gallup. (Yes, Fox/Opinion Dynamics and ABC have the gap somewhat smaller but they were taken earlier.) http://www.pollingreport.com/BushJob.htm
So it’s not even a question of Bush’s worse performance in the CBS poll being statistically insignificant–it’s not even existent!
Seixon–
Before you go running off at the mouth about things you obviously don’t really understand, you might want to get a fuller picture than just the last few polls. Here’s one trivial analysis to look at:
http://anonymous.coward.free.fr/polls/pollbias-details.html
It’s certainly not obvious from this that CBS has a pronounced anti-Bush house effect. [Feel free to learn some basic statistics and then in a year you can try this for yourself.]
Or, to put it another way, even with a fair coin I expect to occasionally get several “heads” in a row.
joran,
If the 34% of CBS’s figure isn’t the most likely in the probability distribution, why is it the one chosen? In other words, the confidence interval, 95%, is from 31%-37%. So, if you are saying that 34% is not the most likely in the range, why is it the one chosen?
Well, because it is.
Others point out that the subsamples of party ID may have higher margins of error, and I suppose that is true, but they would not be more than 5%.
To summarize, the true number is likely to be within the range of each poll conducted, unless, of course, there are severe problems with the sample.
So when CBS says that 72% of Republicans approve of Bush, while Cook/RT says 82%, even though they are sampling the same population, that is a spread of 10%. Even taking a 5% margin of error into consideration, that means that the true figure is 77%… but since that is actually outside the 95% confidence intervals of both, that is higly unlikely.
If CBS didn’t consistently pull up the least amount of Republicans, most amount of Democrats, and the lowest support among Republicans of all pollsters consistently, I wouldn’t have a problem with it.
I would love to have you explain why the center of the 95% confidence interval isn’t actually the most likely. People who defend the Lancet study on mortality in Iraq made quite sure to tear anyone who said otherwise a new one – quite correctly might I add.
Seixon –
“I would love to have you explain why the center of the 95% confidence interval isn’t actually the most likely.”
Gladly.
It’s a subtle point. First, repeat after me: there is no probability distribution over the CI! There is no probability distribution over the CI!
Think of the CI as the collection of values are are all plausible, given the data. The interval just _is_. So there is absolutely _no_ way to assign probabilities to the values in the CI (again, unless you adopt some Bayesian framework).
The reason we report the center value is that it minimizes our risk of reporting the wrong value within the CI. This is just common sense: if I’m trying to guess a number between 0 and 1, each number is equally likely and I want to minimize my error, I’ll guess .5. This is not the same as it being the “most likely”. It’s just the most “risk averse” guess.
Another way to think about it is that our estimate is our best guess _prior_ to allowing for possible errors in our estimate. We acknowledge potential errors in our estimate by saying that we cannot (safely) distinguish between any of the values within +/-3 (i.e. in the CI). Since all the values are in this sense equally plausible, our safest course of action is to go with the center.
To be perfectly honest, really undertanding CIs is not easy. They are very subtle, and it is unreasonable expect most people to pick up a deep understanding of them by simply reading about them casually. Indeed, many people who have taken several statistics courses at an undergraduate (or graduate!) level often fail to interpret them properly.
Also:
“Others point out that the subsamples of party ID may have higher margins of error, and I suppose that is true, but they would not be more than 5%.”
Again, increasing the margin of error to 5 points would cause your example of support by party (6 vs. 13) to overlap considerably. (i.e. (1,11) vs. (7,18)).
On top of that, CI width does not decrease linearly with sample size. It only grows with sqrt(sample size). (Ever wonder why national polls always take around 1000 samples? That’s why. If they wanted to get all the way to +/-2, they’d have to sample more like 2000 people. Getting to +/-1 bumps you all the way up to like 10000 people I think. Which takes too much time/money).
This means that, reducing the sample size by roughly a third could very well balloon the MOE past 5. Indeed, my rough mental calculations suggest it wouuld be around 5.5 or 6.
Now, I’m not saying that there will _never_ be an instance where two polls cross-tab estimates are distinguishable. Indeed, your 72 vs 82 is getting pretty close, even though I’m betting the true MOE near 6 will save it in this instance.
The larger point is that equipped with a proper understanding of CIs and the increased sampling variation in subsamples, the amount that CBS’s polls differ from most others is _drastically_ reduced.
I hope most of that made sense…stats is difficult to explain in little snippets, let alone over the internet.
Seixon –
You’ve got it backwards.
The central limit theorem states that the sampling distribution of the proportions tend to be normally distributed.
A single sample provides just one proportion and there is no way to tell from the single sample where that proportion falls within the normal distribution of all samples.
Simply put, the sampling distributions are not normally distributed around the first proportion as you claim.
Hope this helps.
Matt wrote:
> Here’s one trivial analysis to look at
Ouch.
Seixon wrote:
> Is it just a fluke that CBS has had all
> the records for lowest approval rating?
> First and only at 35%, now first and only
> at 34%.
and
> That shows that CBS is consistently in a
> class of its own.
Evidently not. Pew just released at 33%.
http://people-press.org/reports/display.php3?PageID=1039
> >Here’s one trivial analysis to look at
> Ouch.
Sorry. I meant trivial in the mathematical sense, not in terms of its interest or worth.
joran and frosty,
So the Lancet study that claimed 98,000 excess deaths from the war in Iraq, with a 95% CI of 8,000 – 194,000… you’re telling me that every number within that range is equally likely?
Or is there some reason why that poll is different from the CBS poll?
matt,
That was a good link, but it shows that CBS has been consistently under the trend since the end of 2004.
I’m not sure a poll of a thousand people carried out several times can be directly compared with the tossing of a coin several times, but hey, I guess I’m just an innumerate… 😉
R. Chung,
Already duly noted in my latest post: http://www.seixon.com/blog/archives/2006/03/pew_outdoes_cbs.html
seixon — Doesn’t the link Matt pointed to at Anonymous Crowd show that CBS was consistently over, not under, the trend before the end of 2004? Taking the whole five-year period of the Bush Presidency, according to that Anonymous Crowd study, CBS was the sixth closest to the mean of the 16 polls compared, with its average difference from the mean falling on the favorable (pro-Bush) side. This is not prime facie evidence of an anti-Bush house effect at CBS. Regards — andrew
Matt wrote:
> Sorry. I meant trivial in the mathematical
> sense
Uh, thanks, I think.
Andrew wrote:
> Anonymous Crowd
Here’s a conundrum: when one is an anonymous coward, should one correct misattributions?
Seixon-
“So the Lancet study that claimed 98,000 excess deaths from the war in Iraq, with a 95% CI of 8,000 – 194,000… you’re telling me that every number within that range is equally likely?”
Well, yes. I know nothing about how they calculated that CI, but we’ll assume for the moment that it is indeed a correct 95% CI.
My response to that is that this Lancet study you cite is precisely why point estimates are not enough. I would interpret that CI, (8000, 194000), as follows:
Our best estimate for excess deaths is 98,000. But given the enormous variability in in our CI, this measure is _very_ uncertain.
Excess deaths in Iraq is (not surprisingly) a _much_ more difficult quantity to estimate. There are many more sources for error, and it is much more difficult to protect against them. Hence, the CI reflects this added uncertainty by being _very_ wide.
But the meaning is the same. 98000 is our best guess; but due to the (large) amount of uncertainty in this measure, we can’t conclusively rule out any value in (8000,194000).
So yes, they are all “equally likely”, although again that’s somewhat sloppy language. It’s better to say that we can’t really distinguish between them based on the data.
Like I said, this is why CIs are so important. By itself, 98000 would be a horrible estimate, because you haven’t indicated your level of certainty regarding the precision. But once you provide that very wide CI, all I’d be prepared to conclude is that we’re reasonably certain the number of excess deaths is at least in the thousands (but that it could be much higher; we don’t know).
Without knowing precisely how Lancet calculated that CI, I can’t explain how it ended up being so wide. However, the basic idea is that CIs are calculated differently in different situations, and not only was Lancet not estimating a proportion, I’d be willing to bet they weren’t polling! 😉
So they would have used a different (though hopefully appropriate) method to get that CI that reflects the uncertainty in their estimate. I’m not sure I can explain any better why the CI widths are so different without wading pretty deep in the actual math; and to do that I’d need to know precisely how Lancet found their CI.
Andrew,
Think back to September 2004. What did CBS do at that point? Now, try to think of reasons why CBS polls might have suddenly changed from generally average to the trend, to consistently below. Can you figure out the mystery?
joran,
You know, I would love for you to go to timlambert.org and make that same argument. They would seriously tear you a new one and call you an innumerate. The reason why their CI was so large is because well… they pretty much screwed their own sample to the point of being completely unrepresentative of the population being sampled.
However, the argument was made that the 98,000 was the most probable number out of the entire range, and I always assumed this to be the case, although my problem with it is other things due to how they collected and chose their sample.
Seixon–
OK, now you have stumbled into an area I am actually an expert in — network television news — as opposed to my visits here at Mystery Pollster, which while being fun, are those of an enthusiastic amateur.
September 2004 was the date of the fateful 60 Minutes II story on George Bush’s role (or lack of it) in the Texas National Guard during the Vietnam War.
Since that story aired, CBS News has:
1) Commissioned and published a report harshly critical of the journalism in the Bush story.
2)Replaced the anchor who was correspondent for that story with another, Bob Schieffer, universally recognized as being ideologically evenhanded. Schieffer moderated one of the Bush-Kerry debates to the acclaim from partisans on both sides.
3)Launched Public Eye, an ombudsman-style Website monitoring its own journalism, conducted by Vaughn Ververs, a self-styled libertarian conservative.
Now you argue that September 2004 is the turning point for CBS News’ polling operation to change its house effect — deliberately or unintentionally — to be, compared with the average, on the unfavorable side to President Bush having been on the favorable side.
Under what logic do you suppose that the editorial arm of CBS News would take highly-publicized steps to distance itself from insinuations of pre-September-2004 anti-Bush animus while its polling arm would take steps to repudiate its own management?
Regards–Andrew
PS. My bad, R.Chung. I cannot tell a crowd from a coward.
Andrew,
It doesn’t matter how CBS tried to recover its image – it is forever stained. The fact that they didn’t even fire Dan Rather has lots of people pissed still. I would suspect that the “in house effect” at CBS after this incident is closely related to its reputation.
NBC/Wall Street Journal just came out with one at 37% – a full 3-4% above Pew and CBS. I guess the truth lies in the middle somewhere.
Seixon-
“NBC/Wall Street Journal just came out with one at 37% – a full 3-4% above Pew and CBS. I guess the truth lies in the middle somewhere.”
A statistician is born! 😉
Once again, let’s look at the approval *minus disapproval* in the polls conducted in the last ten days (i.e., since March 6):
NBC/Wall Street Journal -21
CNN/USA Today/Gallup -24
CBS -23
Pew -24
AP-Ipsos -23
Far from being an “outlier,” CBS is right in the middle here. If it even makes sense to talk about a middle when the difference between the highest and lowest figure is only three points…
Bush at 33: The ‘I’ word
‘Honesty had been the single trait most closely associated with Bush, but in the current survey “incompetent” is the descriptor used most frequently …’
ouch…
So no correction from Mark then eh?
Seixon:
It looks like Mark has made a correction.
Not so BTW, you wrote: “Is it just a fluke that CBS has had all the records for lowest approval rating? First and only at 35%, now first and only at 34%.” Harris was first at 34%, in November, 2005; and Pew is now at 33%. Time for a correction?
First, my sincere apologies to Seixon for a terribly sloppy mistake. His post obviously did not reference the 2004 exit polls and the link should not have been included among the other three. I corrected the posting above.
Second, bigger apologies to Seixon and everyone else for my inattention to the comments for the last few days. With the demands of a “day job” and two young children at home it does happen, but that still does not excuse the fact that I let the error go uncorrected for two days. Again, apologies to all and thanks to Joran for alerting me to the need for a correction via email.
As to the substance of this argument,Joran, Frosty and others have the statistics right, but there are two even more fundamental issues.
One mistake that Seixson makes is to assume that except for the target populations, all polls are created equal. They are not. CBS and Gallup ask a different party ID question, and have been getting consistently different results for the “don’t know percentage to the job approval rating. The consistency of these differences suggests a different type of error — measurement error — is probably at work.
Moreover a “house effect” is no more evidence of a cooked poll than a sloppy oversight in a blog post is evidence of “dishonesty.” Though Seixon, you sure did go there in a hurry, didn’t you?
Your theory seems to be that since September 2004, someone at CBS has intentionally cooked thir polling data to favor the Democrats. If so, how do you explain Robert Chung’s analysis of the “head to head” Bush-Kerry results from 2004? If I’m reading it correctly, it shows that CBS had Bush ahead by a *larger* margin than other pollsters in September 2004, and was then right in line with other pollsters during October.
Check the summary of 2004 poll results at RealClearPolitics (link below). CBS had Bush ahead by margins of 9, 9 and 7 points during September. Their final poll had Bush ahead by two (49% to 47%) and he won by 2.5%.
http://www.realclearpolitics.com/bush_vs_kerry.html
By the way, it’s also flat wrong to say that CBS “has had all the records for lowest approval rating.” Take a look at Charles Franklin’s analysis of the most extreme outliers for the Bush job rating. The low records belong to Zogby and IBD/CSM/TIPP:
http://politicalarithmetik.blogspot.com/2005/12/how-out-of-line-is-new-hotline-poll.html
One clarification about the last point above: Franklin’s outlier analysis pertains to lows *relative to other polls.*
The historical pages at RCP are not available, but in addition to the first lows that RChung sites above, the first pollster to hit 36% approval was American Research Group in August:
http://www.realclearpolitics.com/Presidential_04/bush_ja.html
R. Chung,
Harris Interactive polls are definitely an orange to the apples we are comparing. No need to even bring that poll up at all.
Mark,
If it was just a mistake on your part, then I apologize for saying you were dishonest. Yet you were very hasty to fling me in with all the others who were running a faux argument.
Further, my position isn’t that someone at CBS is intentionally cooking their polls (although it certainly is possible), but that since the Rathergate fiasco, their samples might be getting biased due to the responses of the people they call. For instance, Republicans might hang up on them right away if they identify themselves as with CBS. This might also explain the vastly different type of Republican that CBS has been getting compared with other polls.
The possible effects of this don’t seem to have come around until the end of 2004, after which CBS is entirely under the trend line.
It was CBS first at 35%, and first at 34%. Now Pew has outdone them at 33%.
The largest point to make is that none of these polls are satisfactory on their own, and it’s dishonest for partisans of all colors to cherry-pick the poll they want. Average them all, and we get a more honest look at the numbers.
I’m an Independent that disapproves of Bush at the current time, but you’re not going to see me drooling all over the Pew or CBS polls, nor the other end of the stick. I just wish other people were as principled.
Seixon —
Thanks for the clarification.
If I understand you correctly, your argument is not that CBS has an ax to grind with regard to Bush…
…it is that many Bush supporters have an ax to grind with CBS and so are failing to respond to their surveys (and reciprocally many Bush opponents are so enamored of CBS News that their normal inclination to refuse to cooperate has been overcome).
Your argument is not that CBS pollsters are bad actors — instead they are the victims of the skewed public relations image of their news division.
In this light you might want to reconsider your allegation that “CBS has proven to be the black sheep in the poll department for quite some time.”
A black sheep is usually one that misbehaves. In your scenario, they are more like Dan Rather’s scapegoats.
Anyway, to test your hypothesis.
Do we know how CBS interviewers identify themselves during the screening process?
Have CBS’ overall refusal rates changed since September 2004?
Has the demographic/socioeconomic composition of the refusers changed since September 2004?
Do data from Pew (or somesuch source) demonstrate that CBS has a disproportionately negative image among Bush supporters compared with other news organizations that conduct polling (ABC, NBC, CNN, USA Today, Washington Post and so on)?
If so, was that negative image the consequence of a massive change in CBS’ reputation during the fall of 2004?
Regards — Andrew