Monday’s edition of National Journal’s Hotline listed seven headlines from the last few days that “say it all,” as they put it, about recent political polling:
· “Divergent Opinion Polls Reflect New Challenges to Tracking Vote” (Wall Street Journal, 9/20).
· “Seesaw of Polls Leaves Lots for Debate” (Newark Star Ledger, 9/20).
· “Varying Polls Reflect Volatility, Experts Say” (New York Times, 9/18).
· “Wide Gap Among Poll Results Mystifies Campaigns, Pundits” (Washington Times, 9/20).
· “Conflicting Polls Confuse Voters, Pros” (Chicago Tribune, 9/18).
· “Why Voter Surveys Don’t Agree” (Christian Science Monitor, 9/20).
· “Despite Disparity, Pollsters Back Surveys” (Detroit News, 9/19).
These are all good reviews –Harwood’s WSJ piece is arguably a must-read – yet all seem to miss the most basic source of much of the “divergence:” statistical sampling error, the random variation associated with looking at a sample rather than the entire population.
Yes, the conflict between polls seems a bit greater this year, and issues like response rates, likely-voter screens, party ID weighting and the like are worthy topics for discussion, but my sense is that much of the recent confusion comes from a basic mistake. Most observers wrongly assume that the “margin of error” applies to the spread between the candidates. EvenOne notable exception is the WSJ’s Al Hunt, who in a column on polling last Friday wrote, “Poll watchers must remember that the best survey has a three or four-point margin of error; that means if it shows the race even, one or the other candidate actually could be up by a half-dozen.”
Well, no. Actually, one of those candidates could be up by more than that.That’s right, since the margin of error applies separately to each candidate’s support, the margin of error effectively doubles when applied to the margin between candidates.
Consider another example: If a single survey with a sampling error of 3% (based on a 95% confidence level) shows Bush at 49%, we know with 95% confidence that if every voter in the country were interviewed for that survey, Bush’ support would lie somewhere between 46% and 52%. If the same survey has Kerry at 42%43%, his support could range from 39%40% to 46%. Thus, the 49% to 42% survey tells us with 95% confidence that the race could be anywhere from a dead heat to a 1412-point Bush lead.
Of course, I’m oversimplifying a bit. Sampling error is really a grey concept, not black or white. When it comes to measuring the spread between candidates, the odds are still good that most polls will fall within a narrower range, but the bigger point is inescapable: Most poll readers overestimate the precision of polls of random sample surveys, and those of use who conduct and report on polls are not doing enough to enlighten them.
As Ana Marie Cox wrote earlier this year:
“Pollsters typically present their poll results in press releases that do little to educate journalists as to the meaning of the numbers they contain….If Zogby and Bennett don’t talk about margins of error or methodology, why should anyone else?”
Amen.
Bonus finding: Yes, Ana Marie Cox, aka Wonkette, has (had?) a little-known double life as a statistics geek. Who knew?
CORRECTION (9/24): Oy. The comments below are correct. I completely misread Al Hunt’s quote. He was just using a different range. Double apologies to Mr. Hunt since he not only got it right, but was one of the few to actually raise the issue of sampling error in the coverage the last week. My bad.
Second, I obviously goofed up my own example. I had initially written it using a +/- 3.5% margin of error (since that is more typical with the smaller samples of likely voters reported recently) but then decided the rounding made it too hard to follow. For some inexplicable reason I didn’t change all the numbers. My bad again. In the spirt of the blogosphere, I’ve corrected the numbers leaving my ugly mistakes in place.
Thanks to Sasha and Phil for the fact checking. Apologies to all. One more thing to atone for.
So why did you disagree with Al Hunt? He took the margin of error, and doubled it.
Isn’t that what you suggested?
There are also reasons to believe that when we aggregate polling data, i.e., combine polls and take the average, that the “bell shaped curve” becomes tighter around the average percentage for one candidate. Of course, we should take predetermined time span in which to do the aggregation and not a time span the “favors” our candidate. One method I like is to take moving averages over a two week period for all polls. Then when we see a moving average outside the newly computed confidence interval, we know that a new trend is emerging. The sampling error is larger than what it would be if a single poll increased its sample size, but the sampling error is lessened when we combine polls nevertheless.
Excellent article!!
Good comments and analysis. My problem is not so much the polling but the reporting. The last few weeks show that the more unusual results, such as Gallup, got headlines while four or five other polls that showed a close race were buried paragraphs into the reporting. It is unfortunate that an accurate headline of POLLS SHOW MIXED RESULTS is not as sexy as a headline of CANDIDATE X JUMPS TO BIG LEAD (in only one poll).
Jim3737
Sorry, but I agree with the first commenter; you’ve completely messed up the example about margin of error.
Al Hunt got it right when he said that if the MoE is 3 percent, either candidate might be up by a half dozen. Let’s assume that each candidate is polling at 47 percent. This means that each candidate could have as little as 44 percent support and as much as 50 percent support (with 95% confidence, of course). If one of the candidates has 44 and the other has 50 (both at the edge of the MoE), then one candidate will be up by a half dozen percent. Not coincidentally, a half dozen is double the margin of error for the poll.
You point out that the spread between the two candidates has a potential *range* of double that (or four times the MoE), but its potential *value* is only double the MoE. In my 47-47 example, this means that the race could be anywhere from Bush +6 to Kerry +6, so the spread between the two candidates could be anywhere in that 12 point range. The actual value of the spread can only be 6 points and still be within the MoE of the poll, though, as Al Hunt wrote.
Of course, you came up with a 14 point potential range of the spread rather than 12, but I’m pretty sure that’s due to two arithmetic errors. First, a margin of error around 42% should be 39% to 45%, not 39% to 46%. Second, 52% minus 39% is 13%, not 14%.
People who don’t know a lot of statistics, but who want to understand the margin of error can (in my opinion) learn a lot from spending about 10 minutes in excel performing an “artifical” poll.
1. In cell A1, type “=100*RAND()”. This will give you a random number that is uniformly distributed between 0 and 100 — that is there is a 10% chance that the number will be less than 10, a 20% chance that the number will less than 20, etc.
2. Decide what you want to assume about the “truth” in the population that you are (pretend) polling. For example, you might want to assume that true population is evenly split: 45 Bush -45 Kerry, with 10% other or undecided; or you might want to assume that population is 52 Bush to 43 Kerry with 5% other and undecided. The thing to notice is that you are deciding what the “truth” is here, and then we are going to see how well the poll measures this truth.
3. Let B be your Bush “true” % and K be your Kerry “true” percent. In cell B1, write: “=if(A1 (100-K), 1,)”. Of course you need to fill in actual numbers for “B” and “100 – K”. If your “truth” is 45 B, 45 K, and 10 Undecided, you would write “=if(A1 55, 1,0) in C1. This records whether the “person” you “contacted” supported Bush (1 in B1), Kerry (1 in C1), or was undecided (0 in both B1 and C1).
4. Decide how many “people” you want to poll. For example if you wanted to poll 600 people, Copy A1 through C1, and paste into A2-C2, A3-C3, … A600-C600.
At the bottom of the row, add up the 1’s in column B and 1’s in Column C, transform these into percentages. For example if you “polled” 600 people, you would write in cell B 602: “=100*sum(b1:b600)/600” and in cell C602: “= 100*sum(c1:c600)/600”
5. Now comes the “margin of error” part. If you make a copy of these three columns: (A, B, and C), and then paste 99 additional copies in other columns on your spread sheet, you will have the results of 100 identical polls who all polled the exact same situation with the exact same “truth”. What you will see is that some of these polls show Bush way ahead, others show Kerry way ahead — even though (in my example) the true population is exactly evenly split 45-45. The “margin of error” (assuming this is based on a 95% confidence level) of +/- 3 points means that there is only a 5% chance of Bush being polled at more than 48, or at less than 45. (I’m not sure that 3 points is the margin of error for a 600 person poll. This is just for explanation purposes.) You can look at your 100 polls and see what the maximum Bush number is, and what the second highest Bush number, etc. (If you did 1000, or 10,000 polls, it is likely that your results will be a more accurate representation of the margin of error.)
6. If you increase the number of people you “poll” your margin of error will decrease.
Interesting trivia. In proportional estimation, such as percent favoring x, the sampling error is independent of the size of the population being sampled. The formula is squareroot of (pq/n) where n is the size of the sample and p is the proportion for and q the proportion against. Note that the size of the population being sampled is not in the formula.
The New York Times Magazine discovers that bloggers are geeks
The teaser for Matthew Klam’s cover story on political bloggers: The bloggers covering the presidential race are maverick, funny, mostly partisan and always hypercaffeinated. Are they ruining political journalism or recharging it? That’s a great questi…
1.Statements of sampling error say how likely this sample is given some true state of the world of interest. eg. How likely would 47% of this sample of future voters say they will vote for Bush when 50% in the population actually would say so if this sample, drawn this way, were, say 100 times larger (or effectively as large as the population who would be queried by these methods.)
2.Our electoral rules give the victory (within states,anyway) to the candidate with the plurality of votes, even if that is just, 5% of eligible voters. Why don’t our polls report THEIR results in those terms?
Every sampled person will 1)vote Bush or Kerry, 2)vote for nobody/someone else, or 3)won’t/can’t tell the pollster. Pollsters try to make the *won’t/can’t tell* irrelevent by sampling adjustments, keep trying to identify the vote fore nobody/somebody else; and then estimate the condidional likelihoods of votes for Bush and Kerry. So they keep mixing in modeling error with sampling error, and only tell us about sampling error. There would be less mystery about differences between polls if for every poll we were also told the % who didn’t say, (including the unreached); and the % who report they won’t vote or will vote for someone else. When you see how large those components are and how much they vary from poll to poll, you’ll be less surprised at variation in %Bush and %Kerry. With typical polls, we would be comparing fluctuating numbers in the 9-13% range (because of low voter- participation and the inability of pollsters to reach eligible voters for the sample). We would, as citizens and analysts, be reminded how much the election outcome depends on mobilizing the (can’t/won’t tell+won’t vote)fraction.
Please Guide me to choose a title for my dissertation related with Exit pole. I love to c your article. Very nice presentatiton.
J Jerald Inico
INDIA.