Internet Polling: Unfulfilled Promise

An update on the topic of Internet based political polling: Last week, I noted that some influential academic research on public opinion and the Iraq war had been conducted on the Internet.  It involved  a "panel" of respondents maintained by a company known as Knowledge Networks and recruited using conventional random digit dial (RDD) telephone sampling.  The company offers free Internet access to willing participants that not already online.

However, Knowledge Network’s use of "probability sampling" is unique to Internet based surveys.  Many other research companies now offer to poll panels of self-selected respondents that volunteered to participate in their surveys, usually for some sort of monetary incentive.  These non-probability panels are all the rage in commercial market research, but for now, have seen little application in political polling. Why?

In a guest column that appeared in National Journal’s Hotline two weeks ago, Doug Usher, a pollster with the Democratic firm the Mellman Group, offered some sensible analysis:

When it comes to quantitative data for political applications — statistically reliable answers that are projectable onto the entire population — telephone surveys remain critical. And for good reason: when it comes to cost, population coverage, and predictive accuracy for nearly all political polling, telephone surveys remain the gold standard…

Telephone surveying today — as always — has significant problems, and critics are correct for pointing them out. Moreover, the Web may be a critical tool for supplementing telephone polling – indeed, the time may be ripe for the transition. But there are very real impediments to its use in polling in most campaign contexts. Until we overcome those impediments, telephone surveying will continue to dominate the political polling landscape.

Usher goes into depth on three "oft repeated falsehoods about the tradeoffs between Internet and telephone polling."  These are worth reading in full but — until today — were trapped behind The Hotline‘s subscription wall.  However, both the author and the folks at the National Journal have kindly consented to allow MP to reproduce Usher’s column in full.  It appears after the jump.  Read it all.

As it happens, The Hotline today also unveiled a web version of its very cool "Blogometer" feature that is now free to non-subscribers.  It’s definitely worth a click.   

The following originally appeared in The Hotline:

POLLING ANALYSIS


The Internet’s Unfulfilled Promise For Political Polling

By Doug Usher, Hotline guest contributor
Thursday, June 30, 2005

The Internet has dramatically changed the face of most aspects of
today’s political campaigns. Key elements of campaigns, including
fundraising, grassroots network development, and GOTV, now rely on Web
innovations for a large and growing portion of the work they do.

At the same time, the Internet has not lived up to its early promise for one key part of campaign consulting: polling. 

And this may not change in the future. Unless major changes are made in
e-mail registries, Internet privacy laws, and the way we interact with
the Internet, it may never be the same tool that the telephone has been
quantitative research for over 50 years.

At the Mellman Group, we’ve been harnessing the power of the Web in
many of our qualitative research applications. Web-based ad testing
allows us to very quickly gain input from hundreds of voters in a
matter of days, giving us (and campaigns) new flexibility to test ads
on the fly.

Online qualitative tracking allows pollsters to advise campaigns about
changes on the ground, from all over candidates’ home turf. This direct
give-and-take helps clients learn about surprise advertising and direct
mail, in addition to other changes on the ground that can have an
impact on public opinion.

But when it comes to quantitative data for political applications —
statistically reliable answers that are projectable onto the entire
population — telephone surveys remain critical. And for good reason:
when it comes to cost, population coverage, and predictive accuracy for
nearly all political polling, telephone surveys remain the gold
standard.

The argument for moving to the Internet for quantitative survey
research usually comes down to a cataloguing of the reasons why it is
harder to reach people over the phone: caller ID, growing reluctance to
answer the phone, younger people don’t have landlines, etc.

But this begs the question: is the Web an adequate replacement for
telephone surveys? Here are three oft-repeated falsehoods about the
trade-off between Internet and telephone polling.

MYTH #1: Telephone polling is no longer accurate, because of low response rates, cell phones, caller ID, and other factors.

It is true that it is becoming harder to reach people via the
telephone. Response rates are down (although the exact meaning of
"response rates" often lies in the eyes of the beholder). Some voters
only have cell phones, which pollsters are not allowed to call. And
yes, some people will never take a phone call. All of those are
problems, for sure. And they’re getting worse.

There was a lot of huffing and puffing about the hard-to-reach
cell-phone wielding Deaniacs in Iowa. Around election time, newspapers
spilled a lot of ink (and blogs a lot of bandwidth) on the problems
that this lack of reachability was going to have.

Despite all of these pre-election complaints, did polling mislead us
last election cycle? Were there any results in the last election that
deviated in any substantial way from pre-election polling? Naturally,
we Democrats would have liked to see Kerry win Ohio and win the general
election. But the results were all within the margin of error of public
polling (almost exclusively telephone surveys). In the Senate races,
Republicans probably hoped that Pete Coors could beat Senator Salazar,
and Democrats hoped that Dan Mongiardo could carry out his improbable
effort to unseat Jim Bunning. But a review of the pre-election polling
shows the actual outcomes to be, frankly, unsurprising.

The nice thing about polling ballot items (as opposed to
issue-based polling and corporate market surveys) is that we have a
check. Not only was our internal surveying accurate, but the public
polls (with just a few exceptions) were as accurate as they have ever
been this cycle.

MYTH #2: The high rates of Internet usage mean that Americans are now reachable via the Web.

It is certainly true that more people are using the Web than ever — in
every demographic category. According to the Pew Center on the Internet
and American Life, 70 million voting aged Americans go online on an
average day, up from 51 million in 2000. Overall 136 million people use
the Web — 67 percent of those over 18.

But many make a fundamental error in interpreting this
statistic: just because somebody uses the Web does not mean that they
are reachable via the Web. And unless the architecture of the Web is
changed dramatically, voters may never be adequately reachable via the
Web for useful quantitative research for political campaigns.

All telephone numbers are knowable. Even unlisted numbers can be found,
via random digit dial technology. More important, because of area code
and exchange standards, phone numbers are knowable within relevant
political geography — including states, counties, and most
Congressional Districts. Skilled phone match vendors also do an
excellent job getting phone numbers for voter files, which allows
pollsters to reach voters at the more local levels of political
geography.

E-mail addresses are not universally knowable — because of the
structure of the Internet and e-mail, there are infinite permutations
of addresses and domain names. Spammers are continuously trying to
circumvent this by generating random e-mail addresses and sending bulk
e-mails. While that may be a good way to sell cheap prescription drugs
to a small segment of the population, it is much less effective in
reaching a representative cross-section of an electorate.

Even with ever-increasing online activity, this problem may never be
solvable. Assuming you are able to get through to people, how would you
know that they live in the district of interest? There is no connection
between e-mail addresses and home addresses, rendering them unreliable
in most applications for political polling.

At this point, the only truly "reachable" online population is
those who have opted in to survey research. Some have argued that these
samples are adequately representative of the country, and even for some
large states, when weighted to demographic characteristics. Assuming
that this is true, that still leaves nearly all relevant political
geography uncovered – most states, and nearly every Congressional
District, not to mention every state legislative seat, county executive
seat, and mayoralty. And how about finding likely primary voters who
are actually properly registered?

Opt-in surveys may be adequate at some point in the future; indeed,
they may be adequate now for looking at a very limited political
geography. However, they are not now usable for most political polling.
And as we move forward, it does not seem particularly likely that their
usability will expand to the bulk of political geography in the United
States.

MYTH #3: The Internet now allows us to reach voters whom we can’t reach by phone.

This is one of the most promising aspects of Internet survey
research. If we could supplement telephone survey with Internet
research that covers those who are unreachable, we might be able to
develop a better research instrument. But the jury is still out on
whether it can serve this purpose.

It seems reasonable to assume that "cell-phone only"
twenty-somethings spend time online.
Additionally, many of those who use technology (like caller ID) to
avoid phone calls from people they don’t know also have some connection
to the Internet.

But – as with the voting population more generally — the fact that
they are online does not make them reachable online. Indeed, it seems
unlikely that a person who uses technology to guard their time from
telephone surveys will then turn around and spend their time online
participating in the very same activity. Do some take surveys online?
Probably. But are they representative of the larger population? We just
don’t know.

For the only population that is truly unreachable by
traditional phone methods – those without a landline – Internet polling
may provide some inroads. But fundamental questions will remain, as
discussed above: are you sure that the people you are speaking with are
a) without a landline, b) from the appropriate political geography, and
c) are available to you in an adequately representative way? If you can
be reasonably sure that those three questions are answered, then
Internet polling may provide a nice supplement.

What’s the future for political public opinion research over the Web?

The Web is a growing force in qualitative public opinion research in
politics — an invaluable tool for testing ads and exploring the nature
of individual public opinion. Our Web-based research is now part of
many of the campaigns we work with, allowing them to do research in
time frames that were never before possible. In time, more applications
will be available online, allowing us to explore different populations
quicker than we’ve ever been able to in the past.

And there are grave concerns about the future of telephone
polling, as response rates continue to decline, and reachability
narrows.

Yet ten years after Netscape’s IPO, the Internet has not lived
up to its billing as the panacea for quantitative research. Indeed, I
have yet to hear of a single candidate facing a competitive race that
used Internet polling as the basis for strategic decisionmaking.

As of now, there are a number of ways for certain types of
quantitatively precise surveys to be conducted over the Web. Many
organizations, including universities and corporations — use the
Internet to conduct surveys of their employees, students or members.
This makes sense, as those organizations tend to have comprehensive and
accurate e-mail lists, and are recognized by the membership as credible
invitees.

Additionally, at least one national panel has been formed which
recruits survey respondents and provides them with access to the
Internet in exchange for survey participation. This helps overcome some
of the obstacles discussed above, but is also expensive and time
consuming. Moreover, its accuracy for quantitative research is largely
limited to national efforts.

Telephone surveying today — as always — has significant
problems, and critics are correct for pointing them out. Moreover, the
Web may be a critical tool for supplementing telephone polling –
indeed, the time may be ripe for the transition. But there are very
real impediments to its use in polling in most campaign contexts. Until
we overcome those impediments, telephone surveying will continue to
dominate the political polling landscape.

© 2005 by National Journal Group Inc., 600 New Hampshire Avenue, NW, Washington
DC 20037.  Any reproduction or retransmission, in whole or in part, is a
violation of federal law and is strictly prohibited without the consent of
National Journal. All rights reserved.