As promised, here are comments from Mark Schulman founder and president of Schulman, Ronca & Bucuvalas, Inc. (SRBI) in response to my query about the disclosure party ID results on the Time magazine survey.
I should note that in my original email to Schulman, I wrongly concluded that the Time/SRBI poll "routinely" omitted the results for party ID, because those results were not reported on the release for the most recent survey conducted in January. In fact, a review of the marginal results from their 2004 preelection studies available for download at the SRBI archives confirms that they regularly included full results for party ID (see the link for "election trend frequencies" at the bottom of each poll analysis).
Schulman clarifies their policy:
Our policy is to post full marginals for each study. Thanks for noting that the demographics are missing from the January survey, just an oversight. Deadlines are very tight for the Time surveys. We’re often scrambling to get everything done in a very short time. Looks like the demographics didn’t get posted for that survey. I’ll make sure that full marginals are posted for that survey.
SRBI did report party ID distributions for registered and likely voters starting with the early September Time Poll and we continued reporting these distributions through the election. We posted the party data on our releases to anticipate queries about these distributions and in the interest of full disclosure to interested parties, since it had become a contentious issue.
You’ll remember that the party ID weighting issue surfaced with a bang in early September 2005 when we and some other media polls reported major Kerry horse-race slippage. At least some of the partisan pollsters pooh-poohed the reported slippage, arguing that we should reweight based upon party. They believed that we had somehow oversampled Republicans, hence the Kerry drop. I understand that the reweighting by party ID virtually wiped out the Bush surge…problem solved. I should add that some weighting schemes [used by other pollsters] do attach a minor weight to party ID based upon smoothing the estimation over time. We don’t do that, but I don’t have a real problem with that, since the impact is minor.
By way of background, our Time polls do collect full demographic data on the entire sample, registered voter or not, and we do weight the entire sample by multiple census demographics, adults in households, and number of phone lines.
Several of us challenged the party ID reweighting strategy on AAPORNET and on several blogs. In my postings, I warned that reweighting by party ID "can result in serious distortion." In media interviews, my stance was, if you think all is well with the Kerry campaign and that the slippage was just an artifact of ‘too many Republicans,’ then it should be "business as usual" for Kerry. However, if you believe the Kerry drop that we reported, then the Kerry campaign needs to rethink its strategy. (I was grilled on this point in an interview with Air America Radio, for example.)
Schulman also sent along a longer article he posted on the subject of weighting by party ID that originally appeared on the members-only AAPOR listserv in September. The full text appears after the jump.
Party ID weighting….September 11 Posting on various websites….Mark Schulman
Since we released last week’s poll with the Bush bounce, we’re gotten lots of inquiries about why our poll aand many of the other media polls differ from some of the partisan polls, particularly Zogby, which found little bounce. (I have not actually seen the Zogby poll, but have gotten second-hand reports.) The major reason for the disparity is that most of the media polls, including ours, weight by Census data. Zogby and some others weight on party id. I just penned a response to some academic folks who raised the weighting issue. I’ve attached it below, fyi. Please feel free to discuss.
———–
Weighting by party ID can result in serious distortion of the horserace numbers. Here’s why:
1. As an observer of party identification tallies day after day on our election surveys, it’s clear that we’re not measuring a constant factor. It varies slightly, sometimes even significantly, day by day, week by week.
2. Why does it vary? Most polls place the party ID question near the end of the questionnaire, so that it does not interact or contaminate the horserace measure and any other head-to-head candidate comparisons. David Moore has an excellent piece on the Gallup web site discussing the likely impact of question order on party ID measurement. The horserace always takes priority over party ID in question order, since that’s the topline number we report. As a result, respondents may tend to bring their party ID in line with their partisan choice, particularly after having gone through an extensive battery of election items. It’s simply "cognitive consistency." Hence, a Bush surge, for example, might elevate the number of voters later in the survey identifying themselves as Republicans.
3. Since party ID is a "variable" and not an enduring constant, as is age or gender, it varies!
4. Voting behavior literature from the 1950s and 1960s (Campbell, Converse, Miller and Stokes, The American Voter, for example), used to posit party ID as anchoring partisan choice, as if it were a constant. It’s likely that party ID was a more enduring "constant" in the 1950’s, but, that was then, and this is now. Voters are just not as tied to party as in the past. Let’s get over this likely out of date notion that party ID is a constant that anchors the vote. The causal arrows here are unclear, which influences which? We can construct several models of party ID as both a dependent and an independent variable. The traditional model posits party ID as an independent variable. We now see it likely as both an independent and dependent variable, with all sorts of interactions.
5. Hence, weighting by party ID, and it’s party ID, not party registration, can seriously distort the horserace data. Weighting by party ID would damp down the Bush surge over the past few weeks. Yes, there may be some "at home" selection bias when we interview during party convention periods. However, not all that many folks watch the conventions and the networks provide little convention coverage.
Finally, my choice, and the choice of most the major media polls, is to weight by factors that we know are real, such as age, gender, region, education, number of adults in household, number of voice phone lines, etc. While you can argue about the reliability of Census data, I’ll place my bets with the Census rather than party ID.
David Moore has a good discussion of this issue as well on the Gallup web site.
That’s the short version of my views. I really do believe that we need to put this issue to rest and stop pretending that there’s legitimacy to party ID weighting. I look forward to further comment!
Very best wishes,
Mark Schulman
Schulman, Ronca & Bucucvalas, Inc.
Bottom line:
Actual outcome: Bush by 2.4 (50.7-48.3)
Final polls of companies that weighted on Party ID:
Rasmussen, Bush by 1.7
Zogby, Bush by 1 or 0.3 (depending on source)
TIME’s final poll: Bush by 5
Simply put, TIME’s discrepancy of 2.6 is larger than for either of the companies that weighted. Granted, TIME’s final poll is listed on the Mystery Pollster chart (link below) as being from 10/19-21 (about two weeks before the election). But, we have to go with whatever we have for a company’s final poll…
Sources:
http://www.mysterypollster.com/main/2004/11/first_impressio.html
http://www.tipponline.com
“Granted, TIME’s final poll is listed on the Mystery Pollster chart (link below) as being from 10/19-21 (about two weeks before the election). But, we have to go with whatever we have for a company’s final poll…” Really Alan?
That’s the same type of thing Tom Bevin and John MacIntyre of Real Clear Politics tried to pull when they claimed a couple polling companies missed the election result “big time” even though they didn’t poll in the final two weeks before the election. I couldn’t believe they would impugn the reputations of polling companies for polls taken more than two weeks prior to the election.
If you listen to John Kerry (who had democracy corps and internal polling on the issue), the Bin Ladin tape had some effect on public opinion the final 96-hours before the election, and he even suggested it swung the election to Bush.
I suggest that the comparison all pre-Friday (pre-Bin Laden tape) polls are not “reliable” and should not be compared to the election result.
I openly acknowledged in my previous message that TIME’s poll was from 10/19-21 and in that sense, I believe I was totally transparent. If anyone thinks I sandbagged TIME, I apologize, but I don’t think I did.
I sincerely want to give TIME every benefit of the doubt. Perhaps they did another poll right before the election and it was accidentally omitted from the Mystery Pollster chart. Or perhaps the firm’s computers had glitches in the days immediately prior to the election (computer glitches have certainly happened with exit polls).
But, if indeed TIME stopped polling two weeks before the election, one has to wonder why an outlet that regularly issued polls during the stretch drive would not follow through until election eve. The main way we have of holding pollsters accountable is to compare their election-eve polls to the final outcome.
I completely agree with Rick B. that the best comparison with actual election results would be to use polls from the Saturday-Monday right before election day.
Lost in all this discussion (probably of my own doing by bringing in the TIME poll) is that the two firms that weighted on Party ID — Rasmussen and Zogby — did pretty well.
Alan, I meant no offense. I was just concerned that your statement “But, we have to go with whatever we have for a company’s final poll…” was implying that the TIME poll was a reliable poll for testing against the election result.
Two more things you said that indicates to me that you may be on to something: “The main way we have of holding pollsters accountable is to compare their election-eve polls to the final outcome.”
and
“Lost in all this discussion (probably of my own doing by bringing in the TIME poll) is that the two firms that weighted on Party ID — Rasmussen and Zogby — did pretty well.”
Now, are you considering the confidence intervals associated with each of these polls? If the polls that weighted Party ID and the polls that did not were not statistically different from each other or the election result, can we really say that any one poll did better than another?
I don’t know if the differences were significant, I haven’t run the tests. We could run a difference of proportions test to see if the polls were even different from each other. We can also test the proportions of each poll to the established standard (election result). It could be that sampling error alone could account for the appearance that Rasmussen and Zogby did well (I don’t even know that they did do well – I thought that Zogby really blew it this year).
Tell you what Alan, if you track down both the predicted Kerry and Bush proportions and sample sizes of each poll that you want tested (include whether they weighted by sample size), I’ll test it (or send you the formula to test it) and we’ll see if the polls that weighted party ID did statistically better than those that did not.
This analysis has probably been done by now, but I haven’t seen it. Have you Mark?
The problem is that, regardless of the outcome from these tests, I’m not sure we can know the value/appropriateness of weighting by party ID for near-election day polls. There are other possible ways to introduce non-sampling bias. What about picking the likely voters? What about weighting the undecideds? These are all very important factors.
So, unless we can somehow hold constant for these factors and unless we have an established standard to test against, I don’t know how we can statistically determine whether weighting by Party ID is a valid method.
In my view, the best way to tell if an organizations methods are sound overall is that it has a track record of consistently predicting election results within the sampling error. Taking a single election, picking a couple of polling orgs that did “well” and shared a characteristic and comparing them to other polling organizations with a different characteristic that didn’t do “as well” doesn’t really tell us much.
Mark’s victory is that he has convinced at least one polling org to start releasing information on their party ID weights and we can only hope that more will follow.
If more data regarding methods were made public by more organizations, perhaps enough data can be collected over time for rigorous analysis. Maybe it’s already available and has been tested in some professional journal, but I’m ignorant of the literature. I don’t know.
Okay, I see that Mark had compiled the survey data in his 11/3 post you linked to above. Does anyone know which organizations weight by party ID and which do not? Also, it appears that some orgs allocated undecideds, while others did not. The statistical tests I suggested above require knowing if the undecideds had been allocated. I could guess which ones allocated undecideds by looking at the “other” proportion, but Nader wasn’t the only “other” candidate.
I would exclude all the polls below the Democracy Poll because they were conducted before 11/29. However, from “eyeing” the Bush-Kerry spread columsn, it looks like EVERY poll but the FOXNews, GW Battleground and maybe the TIPP poll did “well” in that the margin of error of each poll appears to overlap the election result spread (I’ll have to look at that more carefully, I may be double dipping here as the confidence interval is applid to individual proportions and not the spread).
I’m assuming that weighting by Party ID is intended to improve the validity of the sample (make it more representative of the population). Kind of like adjustments for “likely voters” (have to do this because you are trying to find out how voters will vote, not how registered voters would vote if they voted), but not like the allocation of undecideds, which is not really a weight, but an assumption about how the undedcideds will vote. I thought that a few orgs did allocate undecideds, while most did not, but am not sure.
Also, there is another layer of data here that might be useful – the state polls. Maybe this is where Zogby “blew it” as I suggested above. It’s been a while since I’ve reviewed the data.
Rick,
I appreciate your effort to really probe the “ins and outs” of the pre-election polling data. I agree with many of the points you made. Given that we’re generally talking about only a couple of percentage points’ difference here and there between different polling firms’ results, the differences may well not turn out to be statistically significant.
Instead, I would suggest using historical track record, as you also mentioned. Zogby is widely regarded as the pioneer of weighting by Party ID. In addition to being only a percentage point or two away from the national popular vote results in 2004, here is his prior record:
1996
Actual: Clinton 49, Dole 41, Perot 9
Zogby: Clinton 49, Dole 41, Perot 8
2000
Actual: Gore 48, Bush 48, Nader 3
Zogby: Gore 48, Bush 46, Nader 5
You made a very astute point, Rick, about how even if a pollster who weights on Party ID does an excellent job of matching the actual popular vote, the success could be due to some other methodological facet used by the same pollster (e.g., defining likely voters, allocating undecideds). In other words, the sample weighting could be *confounded* with another methodological feature used by the pollster.
Bolstering — somewhat at least — the argument that it is the Party ID sample weighting *per se* that contributes to the good results is that Rasmussen also did well with party weighting in 2004. Rasmussen uses automated telephone polling (also known as “robo-polling”), which sets him apart from most other pollsters in a major way. I don’t know how Rasmussen defined likely voters and handled other decisions, but it’s unlikely he did it *exactly* the same way Zogby did. Yet, after adopting party weighting for the first time in 2004, Rasmussen’s performance in forecasting the election improved dramatically from 2000. This provides some degree of cross-validation across pollsters.
In fact, I know of no pollster who weighted on Party ID in ’04 (or any other year) at the national level and badly missed the mark.
I invite you (and anyone else) to visit my sample-weighting website (link below) and read it top to bottom. It starts with a brief post-2004 election summary at the top, then has the main essay discussing the topic, and then has a compilation of web links for further information (at the very bottom is an appendix on how to implement sample weighting with SPSS software, but that’s not so relevant to our present discussion). Among my compilation of links is information on various firms’ sample-weighting policies, if you want to do more research into this area.
http://www.hs.ttu.edu/hdfs3390/weighting.htm
There goes my Ready-Fire-Aim personality again… I just realized how much work MP has really done on this subject. (Sorry Mark).
Alan, I skimmed your site and I skimmed many of MP’s former posts on “Weighting by Party” and I need to read them all again more carefully before I say any more.
“Simply put, TIME’s discrepancy of 2.6 is larger than for either of the companies that weighted.”
Both true but completely without any significance. We are talking about polls with margins of error in the 3-4 point range. That one poll was “off” in the gap between the candidates by 2.6 as compared to another that was “off” by 2.1 or 0.7 tells us nothing at all about how accurate the underlying electorate model was. The camera’s focus simply is not precise enough for that sort of picture.
Adjusting polls for party identification
There has been some discussion about adjusting public opinion polls for party identification (for example, see this page by Alan Reifman, which I found in a Google search). Apparently there has been some controversy over the idea as it was…