Ohio: Back to the Future

There is one poll I have been dying to write about since I started this blog. It has a surprisingly strong track record and, as luck would have it, happens to survey only voters in Ohio, one of the most important battleground states in the nation.

Four years ago, four highly respected academic survey methodologists took a close look at this survey and concluded:

It has been strikingly accurate in forecasting election outcomes since 1980, with an average error of only 1.6 percentage points. They have been substantially more accurate than telephone polls forecasting the same races conducted by the University of Akron (average error = 5.4 percentage points), the University of Cincinnati (average error = 4.9 percentage points), each of which has average error rates reported [for most pre-election polls nationwide] by Crespi (1988) and King (1993).” [From Visser, Krosnick, Marquette and Curtin (2000), p. 227]

As it happens, this survey just released its final pre-election numbers on Saturday. It shows a dead heat between Bush and Kerry (50% to 50%).

So why haven’t you heard more about this highly accurate survey? Because the poll I’m talking about is the Columbus Dispatch Mail Poll, and like the late Rodney Dangerfield, it gets no respect.

I have a hunch that may be about to change.

For years, politicos and pollsters scoffed at mail-in polls. After all, we know the history of the Literary Digest mail-in surveys that wrongly predicted that Alf Landon would defeat Franklin Roosevelt. And most political sophisticates considered mail surveys too slow and their response rates too low in comparison to telephone surveys. Well, a lot has changed in 20 years.

Unlike the mail-in polls of old, the Dispatch poll draws random probability samples from the most recent registered voter list available from the Ohio Secretary of State. Unlike telephone surveys, the Dispatch can sample from the voter lists without concern for unlisted or missing telephone numbers. Then send out over 10,000 packets by U.S. Mail, each containing a cover letter on Columbus Dispatch letterhead, a questionnaire and a postage paid return envelope. They design the paper questionnaire so it closely resembles the look of the actual ballot. They do several surveys during the course of an election year, but on the final survey, they omit the undecided category, thereby forcing voters to chose the way they do in the voting booth.

The methodologists who studied the Dispatch Poll (Visser, Krosnick, et. al. 2000, pp. 227-228) have several theories for its greater accuracy:

• It involves a very large sample size. The survey released over the weekend had 2,880 respondents and sampling error of 2%. No other Ohio poll released this week has even half as many respondents.
• It is a better solution to the problem of “modeling” likely voters: “The mail survey procedure recruited samples of respondents that more closely resembled the actual voting population, perhaps because the act of completing a self-administered questionnaire is in many ways comparable to the act of voting” (emphasis added).
• The lack of an undecided option: “Having to allocate those undecided voters introduced error into the telephone polls.”
• The mail survey was a closer facsimile of the ballot than the questions pollsters typically use to ask about vote preference.

Let me add two more theories:

The mail survey’s “low” response rate is no longer a disadvantage. The Dispatch reported a response rate of 25% for the most recent mail survey. Telephone surveys rarely publish response rates, but according to another study (also by Jon Krosnick and a different set of colleagues), response rates from 20 national news media telephone surveys averaged 22% in 2003 (and ranged from 5% to 39%).

Of course those telephone response rates count a lot of non-voters who agree to be interviewed. Only about 1% of the Dispatch respondents return their ballots indicating they will not vote. Telephone respondents typically over-report voting due to the “social discomfort” (embarrassment) of admitting non-voting to a stranger. So why do 99% of Dispatch Poll respondents report they will vote? You can either believe that Ohioans are so easily embarrassed by non-voting that many feel compelled to fill out and mail back a paper questionnaire, or you can believe that the Dispatch Poll’s 25% response rate is artificially low, since the non-respondents include the non-voters who have essentially screened themselves out.

Also, since the paper questionnaire simulates the actual experience of voting so closely and eliminates the option of being undecided, the incumbent rule does not apply. Most undecideds “break” on the last poll just as they will on Election Day.

One more benefit to consider: None of the big problems looming for telephone surveys — cell phone only households, number portability, caller ID — are an issue for a mail survey.

The Dispatch mail polls done earlier this year had one big shortcoming. They were based on older lists from the Secretary of State that missed most of this year’s new registrants. Not this time. Let’s let Darrel Roland’s poll story in Saturday’s Dispatch explain:

One difference between the latest poll and the one published four weeks ago is the inclusion of more newly registered voters in the sample, whose names were in the latest available data from the secretary of state’s office. About 88% of the new voters – including those from Ohio’s largest counties – were among the potential poll participants.

And which candidate did those new voters prefer? “These newbies now represent one in eight Ohio voters, and they support Kerry by nearly a 2-1 margin [65% to 34%].”

As a result, Kerry has moved from a seven-point deficit in late September to a 50% to 50% tie. Kerry actually “led” on the most recent survey by “a mere eight votes out of 2,880 ballots returned in the mail survey – the tightest margin ever in a final Dispatch Poll.”

Roland also adds the following:

However, in the past four weeks Kerry has surged from a 7 percentage-point deficit into a tie with Bush. And several signs indicate the Massachusetts senator has gained the momentum in Ohio.

Kerry is ahead by 14 points among independent voters. He has a narrow lead in northwestern Ohio, the state’s most reliable bellwether media market. And he has brought black voters home, gaining 91 percent support among black respondents.

Meanwhile, the poll contains troubling signs for Bush. Only 44 percent say things in the nation are headed in the right direction. Fewer than half approve of his handling of Iraq and the economy. And his overall approval rating is 49 percent, a measure that many political experts say represents a ceiling on his support Tuesday.

So here is what I see: Ohio’s most historically accurate survey is telling us that the race is a dead heat. The Dispatch Poll is not infallible – it still has sampling error of 2% – but it has come closer to reality more consistently than any other Ohio poll and involves the best “model” of likely voters I know of. Meanwhile, the RealClearPolitics average for Ohio stands at this hour at 48.6% for Bush, 46.8% for Kerry. Seven out of eight surveys have Bush ahead, by margins of 2-4%.

There are two ways (at least) to interpret these latest results. You can believe that the race is now breaking Bush’s way in Ohio. The Dispatch poll was fielded between October 20 and October 29. However, evidence of a trend is inconsistent across individual surveys: Three polls (SurveyUSA, the Ohio Poll and Mason-Dixon) show movement to Bush, two surveys (Fox and Gallup) show movement to Kerry and two surveys (Rasmussen and Strategic Vision) show no change in the margin.

Or you can believe that the incumbent rule is alive and well and that the two results are essentially consistent in pointing to a very close finish: an incumbent with 48-49% of the vote on a telephone poll is headed for a very close finish with a slight edge over the challenger.

You can make a case for either scenario. My instincts are telling me that the Dispatch poll is right, that the incumbent rule holds and that Ohio will be incredibly close. I also have a hunch that the traditional likely voter models are underestimating the impact of the new registrants just enough to have Bush about a percentage point too high. So I still think Kerry will win Ohio, even though 7 of 8 most recent polls out tonight seem to say otherwise.

Go figure.

I’ll have more later tonight…

Offline sources on the jump page

—————-
A Special Request: Have you found this site useful? Please help me keep it alive by taking a few minutes to complete this brief (3-5 minute) about Mystery Pollster. Respondent identities will not be tracked and, as such, your participation will remain completely anonymous and confidential If you have any problems with the survey please email me. Click here if you’re asking, “why a reader survey now?” Thank you!!

Penny S. Visser, Jon A. Krosnick, Jesse Marquette and Michael Curtin (2000). “Improving Election Forecasting,” in In Election Polls, the News Media and Democracy, Paul J. Lavrakas and Michael Traugott (eds.), Chatham, N.J.: Chatham House Publishers.

Penny S. Visser, Jon A. Krosnick, Jesse Marquette and Michael Curtin (1996). “Mail Surveys for Election Forecasting? An Evaluation of the Columbus Dispatch Poll.” Public Opinion Quarterly 64:125-48