It is nearly impossible to blog on political polling this week without discussing the subject of weighting by Party ID. Ruy Teixeira of the Emerging Democratic Majority and Chris Bowers of MyDD have been arguing forcefully (here and here) that major polls are oversampling Republicans. Yesterday, Mickey Kaus posted a retort by an emailer ("Y”) who described the call for weighting as the "new, cocoon-building liberal analyst meme.”
I have a lot of respect for Ruy Teixeira, and I think his critiques of Gallup Poll’s likely voter model have merit. However, on the issue of weighting, I agree with Kaus and "Y” (whose post is worth reading in full).
Let me add a few thoughts. Party Identification is one of the longest tracked and closely examined questions in political polling. "Generally speaking, do you consider yourself a Republican, a Democrat, an independent or what,” has probably been asked on more surveys over more years more consistently than any other question.
The most important thing to remember is that Party ID is an attitude, not a demographic. People can change their views of political parties. They cannot change their age, gender, race, years of education and locale (unless they’ve moved).
However, for many years political science students were taught that Party ID, like religious affiliation, rarely changes among adults. This belief was the result of two famous panel (or "longitudinal”) surveys conducted by the University of Michigan’s National Election Studies (NES) that interviewed the same respondents three times over a four-year period. These studies convinced researchers that Party ID was highly stable at the individual level.
One limitation of those studies was that they were conducted during periods (1956-1960 and 1972-1976) when overall levels of Party ID remained virtually unchanged. Later closer examination of the data showed that even then, one of every four respondents had changed their answer to the basic Party ID question over the span of four years. Most of the change involved shifts between independence and one of the political parties (See Samuel L. Popkin, The Reasoning Voter, pp. 53-55).
Partisanship has weakened significantly over the years. The percentage of adults that describes itself as independent has grown sharply (from 23% in 1960 to 36% in 2002, according to NES).
Moreover, more recent studies have shown evidence of significant short-term change in Party ID. The 2000 Annenberg National Election Study (NAES), like the 2004 study now underway, was a daily tracking survey that ultimately included more than 58,000 interviews over the course of the year, roughly 5000 interviews per month. NAES observed that the percentage of the electorate identified as Independent "was not stable over time.” In a chart on page 61 of Capturing Campaign Dynamics, Daniel Romer and his colleagues showed the percentage of Independents falling steadily from roughly 31% to 27% during the conventions, then spiking 8 points to 35% just after the Democratic convention in early September, then falling off again steadily back to roughly 28% on election day, then plummeting sharply to below 25% a few days later. No surprise that they concluded:
Surveys that are weighted by party identification may be operating under some misconceptions about party identification. Party identification may not be as stable as once thought and could be considered an indicator of the respondents’ attitudes toward candidates at a given moment of the campaign. (p. 61)
There is another good reason to be weary of automatically weighting survey results by Party ID. Most of the national polls ask Party ID toward the end of the survey. Campaign pollsters can tell you that Party ID can vary with the content of the questions that precede it. Include a long battery of items on the health care or the environment and you are likely to get more Democratic identifiers by survey end than if you include a long battery on fighting terrorism.
I am not arguing that a pollster should never weight by party – there may be times and conditions when it is appropriate (I’ll try to suggest a few in a future post). But I’m very wary of automatically weighting every survey by the results of a four-year-old exit poll, given everything we know about the potential for short term variability of Party ID.
I am also not arguing that the Republican bump in Party ID seen in the Newsweek, Time, CBS News and ABC/Washington Post polls is likely to persist. Nor do I think George Bush has a massive lead. I agree with Ruy and Chris (and Mickey for that matter) that the race is far from over, that Kerry can still win. The post convention bump may have already faded. I hope it has. But I don’t know that it has. And that’s the point.
My first boss, Harrison Hickman, used to say that a pollster’s job was to bring reality to the table. Whether we work for a news outlet or a political campaign, our task is to describe voter attitudes as they are, not as we might wish them to be.
Is that what is happening here? Consider this. Teixeira says in a post yesterday that recent polls showing a "sudden tilt” toward a Republican advantage of 4-5 points or more in Party ID are implausible, when past exit polls and national surveys have typically showed a 3-4 point Democratic edge.
Just three months ago a poll by the Los Angeles Times showed a similarly abrubt "tilt” in party ID (to a 13 point, 38% to 25% Democratic advantage) that helped give Kerry a seven point lead. Matthew Dowd, Bush’s pollster argued that the LA Times Poll was "a mess” because it had not been weighted by Party ID.
And what was Teixeira’s take? He defended the LA Times Poll. Money quote:
There are ample grounds for thinking there is, in fact, a surge toward the Democrats and their positions and away from the Republicans and their positions among the broad electorate. A growing Democratic party ID advantage is a logical consequence of that surge, since party ID does not remain stable as political conditions change….Conclusion: there is no good reason to ignore the results of this poll (unless you’re Matthew Dowd, of course, who has his own reasons for doing so).
Apparently Party ID isn’t the only thing that can change in a few month’s time.
If the Democrats want to delude themselves, why should the rest of us stop them? There’s some value in discouraging the more conspiratorial theories if ‘why Bush won’, but isn’t it really the job of the Democratic leadership to be clear-eyed about this stuff?
It’s all due to the free agency in NFL and MLB. It opens the door to people not having an incentive to root for their team (party.)
Instead they root for individuals and a small coat tail for the party.
SEE BRETT FAVRE INCREASING PACKER SYMPATHY.
I like Brett Favre, therefore I will associate a bit more as a Packer fan.
Sigh. I wish you were wrong. That last Ruy quote is devastating.
What puzzles me about the idea of weighting by party ID is that surveys are supposedly designed to sample from the population randomly. This assumption is also the basis of the margin of error they calculate. So if you sample randomly but then adjust the results because you didn’t get the percentages of D and R that you expected, you have thrown out the design of the survey.
Do you really consider that a fair presentation of Ruy’s point? I don’t.
What Ruy is saying is that viewing the polls that present Party ID breakdowns as shown in Gallup et al as predictive of the Electoral turnout is ridiculous. Do you disagree with that? Do you expect a 5 point edge for the GOP come November?
Can you point to a single Presidential election in the last 20 years when that occurred?
You know you can not. Thus, Ruy’s point is solid for my money.
“Thus, Ruy’s point is solid for my money”
Which thinking? His thinking now – weight by party ID or his thinking then – polling that oversamples one party vs the 2000 split indicates movement in party affiliation?
Or is it 3rd way thinking – if Democrats are oversampled relative to 2000 it is driven by movement in party affiliation movement but if Republicans are oversampled it’s because of sampling error?
Actually we don’t really need published polls. Just looking at how the campaigns are acting speaks louder than any published poll.
Thank you. I made basically the same points to the cocooners at Texeira’s site. You can’t take a random sample, introduce a systematic bias, and expect it to become MORE accurate. Party ID is part of the results, not the demographics. Adjusting the results to make the party ID fit a preconceived notion is really just saying that Candidate X is polling to high and it can’t possibly be right.
Mystery Pollster:
While your objection to weighting for respondant reported voter affiliation makes sense, wouldn’t it be at least theoretically possible to weigh the results for actual party registration?
Regardless of how respondants identify themselves to pollsters, all voters are registered as either Republican, Democrat, Decline to State, or some other minor party. That information isn’t subjective and is readily available. If the pollsters took a little extra time, couldn’t they cross reference with a percentage of actual party registrants for the current election and adjust the results up or down depending on the variation of the polling sample from the public at large?
Hey, don’t knock Texeira; his book is the main reason Kerry wasted a bunch of money in Arizona this year when anybody who knew the state could have told him it would be solid for Bush. He’s killing the Democrats with kindness, and I for one am loving it!
I remember back in 1984 when a lot of “Reagan Democrats” crossed party lines to vote for the Republican candidate. Will such a thing happen again this year? Perhaps, but probably not on such a scale. From what I’ve read, there are a lot of “9/11 Democrats” out there and I suspect a lot of them won’t be voting for John Kerry.
Near-constant party affiliation longitudinally is one of those bright ideas that works great until oneday it changes.
Like the earth, the data-world can appear flat and undynamic when looked at from a particular perspective over a short-to-intermediate time frame . . . . . and the world we live in is getting more dynamic with time, not less. A hundred years ago, people were born, married and died without moving around all that much – but populations today are much more transient.
Moreover, the conservative Southern White folks who favored the Democratic Party for a number of generations in the post-Civil War era have steadily been moving away from the Democratic Party and towards the Republican Party. Etc etc etc.
Texeira and the PartyID crowd are divorced from American social reality. How could party ID remain as strong as it was forty years ago when nearly every other form of social affiliation– to place, employer, to spouse and family– has weakened dramatically?
Think about it. Why would year 2004 Americans, many of whom have almost no loyalty to employers (and v-v.) and relocate every few years or so, and half of whom are divorced at least once, somehow have a strong affiliation to one of the two political parties? Do the parties provide more or fewer tangible economic benefits today than they did in 1960? Do they offer more or less ideological clarity today than in 1960?
I’m not a sociologist, but I’d be very surprised if an increasingly flexible, mobile, fragmented American society sees much value in party affiliation. Teh marketing people “cluster” Americans into, what, 40 or 50 distinct demographics? How on earth can two political parties offer a message, platform and orgnizational apparatus coherent enough to comprise the vast majority of such an atomized society?
Now, this is less true of that tiny minority of US workers who belong to politically-active unions, or of those who live in the college towns-with-a-foreign-policy, and probably also those white shoe suburban enclaves where people still marry their high school sweethearts and take over their father’s country club memberships.
But not in the population overall. These pollsters are as clueless about America today as network newspeople are.
Regardless of how respondants identify themselves to pollsters, all voters are registered as either Republican, Democrat, Decline to State, or some other minor party.
Sorry, no. There are states that do not require registration by party, until very recently, Washington was one of them (and there are a lot of people here who want it to be that way again).
Another problem with weighting based on the 2000 turnout of 39/35 is that those numbers were in turn based on exit polls – and many of those same exit polls wildly overstated the Democratic vote totals. Remember how the nets called FL for Gore based on such exit polls? Or how those same nets waited hours to call states like CO and NC, which Bush won comfortably? They waited because their exit poll data showed a much closer race.
Academics tend to be Democratic. Academics have theories about how things are, or should be, and tend to get upset when reality doesn’t cooperate, i.e. there must be a conspiracy involved, or someone isn’t playing fairly, etc. The last option: perhaps my theory was wrong.
Even if I were a committed/rabid Democrat/Republican and the candidate of my party was a squirrel/turkey, I would at least seriously consider voting for the other guy/gal. If I were not so committed/rabid, I’m gone.
I see the theory of Party ID being a key predictor of a future vote as becoming more and more a fallacy.
I have been arguing for more than 20 years that until we coherently explain exactly what parties are doing (aside from trying to win elections), that is, what actual function they perform in the process, PID won’t make much sense. I tend to see parties as information transactions (it works for me), and thus PID is a function of what kind of info voters believe they need. This can be at least partly demonstrated by looking at historical events such as widespread radio, tv broadcasting and the rise of tv ownership, cable tv, and now the web.
When voters have more info options, PID should be less stable. Parties that don’t understand this lose elections. They’re not performing the function the voters want or need them to. At least not the non-diehard voters.
Bush up 52% to 41% in new CBS poll
http://www.cbsnews.com/stories/2004/09/23/opinion/polls/main645322.shtml
and this time CBS weights by party affiliation (see last page of pdf)
http://www.cbsnews.com/htdocs/CBSNews_polls/poll_0923.pdf
First time CBS has weighted its poll by party, to my knowledge.
“Constant party ID” is “no black swan” logic: the assertion that things not seen in the past will never be seen in the future. It’s a way to bet based on past evidence, but it’s NOT natural law. When a thesis gets proved true often enough, however, we tend to think of it as a law.
And then things change. Events change: 9/11, war in Iraq, the economy, etc. Parties change: gone are both the Gingrich and New Democrat revolutions. Yet somehow people are supposed to remain constant in response to these changes? I don’t think so.
And party ID does NOT equate to candidate choice. It’s correlative, not conclusive.
Correction by party ID is just wishful thinking.
I really don’t like the idea of blindly weighting by a fixed party ID. It seems guaranteed to hide shifts in party ID that quite obviously do vary (even if they usually don’t vary by much) from year to year. Political re-alignments among key demographics *do* happen.
It also doesn’t seem obvious to me that the methodology of Gallup, CBS, Time, and Newsweek would suddenly start going screwy just at the time of the GOP convention. If there is a pro-GOP bias to their model, this shouldn’t have been brand spanking new.
However, the evidence that does convince me that *something* is wrong with many of the recent polls is the reported internals in one or two of them that on the question of who the respondent voted for in 2000, Bush beats Gore by several points.
When you take that with the weirdness in party ID, it certainly makes it plausible that polls are oversampling Republicans for some not-understood reason.
Just as one can oversample the elderly, white people, rich people, catholics, and many other demographic groups that are routinely corrected for, it is certainly possible to oversample Republicans or Democrats.
As such, I tend to agree with Mystery Pollster. I don’t like the ID of ignoring polls that don’t weight party ID, but I still think the results of those polls are currently excessively pro-Bush in their results.
1) How many Republicans here liked the “oversampling argument when Dowd used it?” I thought so…
2) About 5 polls a week have are released- 3 usually show a close race, 2 (usually Gallup) show Bush pulling away. Clearly someone’s methodolgy is wrong. Do we go with the majority polls showing close or the minority showing Bush ahead
3) Finally as I understand Ruy’s point, LV’s are a theory, not science. Gallup is weighting by their pet methodology, which may or may not be right, to come up with LV’s. Their RV result however, agree with the majority of polls showing a close race, if not tied race. We won’t know if Gallup’s pet LV theory is correct until November 2. I would point out that in 2000, Gallup’s pet theory was the least accurate and that Harris & Zogby were the most accurate. Further Harris & Zogby still show the race close.
You GOP folks keep counting those chickens while they are still eggs and don’t go looking at the 1980 election.
One significant weakness in your argument, as you present it, is that you don’t get at the core of Ruy’s argument, which assumes that the DIFFERENCE between party identification for Republicans and that for Democrats has been stable.
The particular research you’ve quoted demonstrates something VERY different, namely that the number of INDEPENDENTS has varied a great deal. This is completely consistent with the idea that the difference between Republican PID and Democratic PID are stable. And, indeed, it stands to reason that a convention might drive independents equally in both directions, polarizing them.
Indeed, one of the very studies you quote says that the largest amount of variation in PID comes from movement in and out of independence. It seems very plausible to believe that certain events might polarize independents, leaving the difference essentially intact.
If you have any evidence that the DIFFERENCE between Rep and Dem PID varies as wildly as we’ve seen across these polls, that would be very good.
Otherwise, you’ve proved exactly zero.
You also didn’t address the very reasonable idea put forward by Charlie Cook (as reported on Ruy’s blog) that pollsters should use a three month moving average of party id to weight their polls. This eliminates the problem of old exit data and recognizes the possibility that party id is not as stable as previously thought.
Instead of asking for party identification, do any polls ask “In which party primary, if any, did you vote this year?” Does it seem to anyone else that it could be useful, or at least interesting, to have this question at the very beginning of a poll? I think it would, particularly if this question were rotated with the “party identification” question.
For that matter, since party registration is public knowledge in most if not all states, have any studies been done comparing registration and how people have responded to the party ID question?
Zogby has always weighted based on party affiliation, and I never heard the Republicans complain about this. In fact, they’ve been known to tout Zogby as the most accurate pollster in history, since he predicted the Republican takeover of the house.
But, rather than the high number of Republican IDs in recent polls, should we be concerned by the high number of people reporting they voted for Bush? For example, in a recent New York Times/CBS poll, 36% reported the voted for Bush, versus 28% for Gore. Given that the actual vote was pretty much 50/50, are we to assume that 8% of respondents were lying? Or that NYT/CBS just happened to get lucky and get significantly more Republicans?
Greg, asked after the fact, some people will incorrectly report that they voted for the winning candidate in the prior election. The fact that Bush, but not Gore, is running again this year may enhance this tendency So it’s entirely possible that 4% of the New York Times respondents were lying. when That would be enough to move from a 32-32% tie to 36-28%.
That doesn’t mean that Mystery Pollster is right to say that weighting by party is undesirable. That may have been true when he started polling, but truly random telephone sampling gets more difficult every year, and Internet sampling has its own set of biases. Market research has the same problem, and the solution is weighting, preferablly by the variable or variables that will have the most influence on the results. In political polling, that will almost always be Party ID, and sooner or later weighting by Party ID is going to become the preferred approach.
I think Keith makes a key point here. Much of this discussion assumes that these national polls are in fact obtaining a representative sample. Let’s assume that a growing number of people do not respond to telephone pollsters for the reasons noted above (i.e. telemarketer fatique, cellphone only use). It stands to reason that during particular time periods, certain potential respondents will be more or less likely to speak to pollsters and it may very well be that when the GOP is seen as “riding high,” more folks sympathetic with their positions will be likely to respond, while those opposing the GOP will be less likely to respond. This selection bias would explain the variation on party ID we see in the recent polls.
Assume that Keith is correct and that PID is the best, or least inaccurate, way of correcting for sample bias/error.
But what happens to this corrective power in the future if (when) PID continues to decline?
Every trend points toward increased social fragmentation– or, if you prefer, empowerment of individuals instead of hierarchical organizations and amateurs instead of professionals. You can sell your house without an agent, invest without a broker, get all the news you need without ever turning on a TV or looking at WaPo or the NYTimes or any media organization.
Likewise, independent amateurs can rapidly raise vast sums of money for a candidate without ever contacting that candidate’s or his party’s organization.
Why would anyone assume that the political parties will NOT decline in influence as activity by independent amateurs continues to increase?
Assume that Keith is correct and that PID is the best, or least inaccurate, way of correcting for sample bias/error. But what happens to this corrective power in the future if (when) PID continues to decline?
Every trend points toward increased social fragmentation– or, if you prefer, empowerment of individuals instead of hierarchical organizations and amateurs instead of professionals. You can sell your house without an agent, invest without a broker, get all the news you need without ever turning on a TV or looking at WaPo or the NYTimes or any media organization.
Likewise, independent amateurs can rapidly raise vast sums of money for a candidate without ever contacting that candidate’s or his party’s organization. Why would anyone assume that the political parties will NOT decline in influence as activity by independent amateurs continues to increase?
Assume that Keith is correct and that PID is the best, or least inaccurate, way of correcting for sample bias/error. But what happens to this corrective power in the future if (when) PID continues to decline?
Every trend points toward increased social fragmentation– or, if you prefer, empowerment of individuals instead of hierarchical organizations and amateurs instead of professionals. You can sell your house without an agent, invest without a broker, get all the news you need without ever turning on a TV or looking at WaPo or the NYTimes or any media organization.
Likewise, independent amateurs can rapidly raise vast sums of money for a candidate without ever contacting that candidate’s or his party’s organization. Why would anyone assume that the political parties will NOT decline in influence as activity by independent amateurs continues to increase?
Assume that Keith is correct and that PID is the best, or least inaccurate, way of correcting for sample bias/error. But what happens to this corrective power in the future if (when) PID continues to decline?
Every trend points toward increased social fragmentation– or, if you prefer, empowerment of individuals instead of hierarchical organizations and amateurs instead of professionals. You can sell your house without an agent, invest without a broker, get all the news you need without ever turning on a TV or looking at WaPo or the NYTimes or any media organization.
Likewise, independent amateurs can rapidly raise vast sums of money for a candidate without ever contacting that candidate’s or his party’s organization. Why would anyone assume that the political parties will NOT decline in influence as activity by independent amateurs continues to increase?
Party ID: The 2006 Edition
We have devoted much attention recently to the flood of new national surveys showing small declines in the Bush job approval rating and modest Democratic gains on the generic House ballot question since mid-September. Until today, I had not looked…
Mark is on the mony that party ID is not 100% static. In mid 1980’s I aided in a pre- and post study (panel-back) for the AMA’s PAC nationally. I was surprised to to find, as Mark asserts here from other data, that fully 1/4 of the voters had shifted their party allegiance in the roughly 10 week period between the pre- and post survey. And don’t we all remember at one point over 80% approval ratings for Presidents Bush 41 and 43? I think what is really being talked about trying to weight data by district partisan performance, using a party ID as a means to do so. Tricky business, that.