More on the CT Exit Poll Experiment

Today I want to catch up and fill in a few details provided by our intrepid reader/reporter Melanie about that experimental CBS/New York Times exit poll conducted last week in Connecticut.   As regular readers may recall, Melanie first brought the exit poll to our attention after being interviewed last Tuesday.   I asked her what she remembered about the experience, starting with the interviewer.  Here is her report: 

The poll person was a young guy, maybe 20 – looked like a college student (tall, a little messy, scraggly hair, very diffident). I think he had some sort of ID around his neck or clipped to his pocket, but I didn’t notice what it said  (he was in the midst of a discussion with the woman running the voting site about where he could be set up – she said she had just gotten a faxed letter from someone giving this young guy permission to be there, wherever he wanted to be as long as he didn’t interrupt anyone, so she told him he could stay out of the sun – he was about 5 feet from the door of the site).

Melanie’s experience gives us a unique window into the real world challenge of trying to select voters randomly as they exit a polling place.  She happened to overhear the interviewer’s most important interaction of the day, the one that enabled him to stand just outside the door of the polling place.  Had he been forced to stand farther away, his ability to sample exiting voters randomly would have been severely compromised.  The post-election report provided by the two companies that conducted the 2004 exit polls (Edison Research and Mitofsky International**) showed that errors in John Kerry’s favor were more than twice as large (-12.3) when interviewers were forced to stand 100 feet or more from the door of the polling place than when they could stand right outside the door (-5.3, p. 37).

At first I assumed this conversation occurred first thing in the morning, just as the polling place opened.  But I asked her to clarify and she said:

I got there about 9, which is early for me, but not for them (they had opened at 6 – the folks inside told me about 10% of the possible voters had already been in by then).

So put it all together and notice the reference to a faxed permission letter that the polling place official "had just gotten."  From the description, it appears that when the interviewer first arrived, the polling place officials would not let him stand near the exit door.  So he presumably called his supervisor and they found a way to fax a letter to the polling place official, and just as Melanie was voting, they relented and allowed the interviewer to stand near the door.  Up until that time, the interviewer had to try to intercept 10% of the day’s sample from a less advantageous position. 

Second, notice the clear impression that the interviewer’s appearance made on Melanie.  He "looked like a college student" with "messy scraggly hair" and a "diffident" attitude, yet she did not notice what appeared on his ID.  Now dear reader, ask yourself what you might guess about the politics or personality of that interviewer.  How would you react to an approach from such a person?  Is it possible that your choice — whether you make eye contact, approach with interest or walk briskly in the opposite direction — might have some relationship to your politics?  The data in the Edison/Mitofsky report from 2004 strongly suggests that it does. 

I will let Melanie continue with her description of how she happened to be interviewed: 

He was alone — he actually didn’t approach me — I saw a couple of other folks doing his poll and when I walked over to see if he’d ask me to do it too he just handed me the device.  (My voting site is pretty small — just one voting machine — with about 700 registered democrats — in a part of a smallish suburb east of New Haven — we never get exit polls here). 

An exit poll interviewer is usually instructed to select exiting voters at random using a predetermined selection rate.  In other words, they are given a number (usually between 1 and 10) and told to select every third voter or every fifth.  Some are told to select every voter that exits the polling place.  We do not know the interval used here, although with 700 voters likely a number greater than one — that is, not every voter should have been selected.  We will also never know if Melanie was one of the voters who should have been selected, but it is awfully interesting that in this instance she approached the interviewer rather than the other way around.

The data from the Edison/Mitofsky report on the 2004 exit polls showed that errors in Kerry’s favor were nearly three times greater when where interviewers had to approach every tenth voter (-10.5), than in smaller precincts where interviewers were instructed to interview every voter or every other voter (-3.6, p. 36).   The clear implication is that where there was more potential for the random selection procedure to break down (either because of more chaos at the voting place or just more room for error), Kerry voters were more likely than Bush voters to volunteer to participate. 

So here we have another anecdote that helps convey the most important challenges in conducting an exit poll.  Some argued the collection of "almost perfect" random samples outside a polling place is easy.  It is not. 

Now let’s hear about the most novel aspect of the CBS/New York Times Connecticut experiment, as per Melanie’s report, that they used an "electronic device" to conduct the interviews.  Exit polls have traditionally been conducted with a paper "secret ballot" given to respondents on clipboard so they can fill it out and drop it into a "ballot box" without revealing the answers to the interviewer.  In Connecticut, CBS and the Times experimented with something new: 

The device was sort of like what the UPS guys use, except horizontal — about 11" wide and 7" high. There was a LCD screen making up the middle of it, and the questions were in the center of the screen — you just tapped the answers with a pencil to register your choices. He didn’t get involved at all — just handed it to me and took it back when I was done.  He seemed to have 2 or 3 of them.  He also had a little thing that looked sort of like a Palm pilot or blackberry that (I think) he inserted into the back of the poll device after it was filled out — I assumed that was the way each set of info was transmitted to the collection folks, but I could be completely wrong about that.

The downside of the paper ballot is that it forces the interviewer to stop intercepting voters several times during the day to talley up responses and call in respondent level data, reading off each answer to someone on the other end of the line that enters everything into a computer.  The NEP exit polls (and those done by their forerunner, VNS) typically had interviewers call in only half of the respondent level data in order to cut down on phone call bottleneck.  This new technology automates both the tabulation and transmission so interviewers can cover the polls constantly all day, and 100% of their collected data gets transmitted immediately. 

How hard was it to use the electronic device?  Melanie continues:

The last question on the poll was about how easy or difficult the text on the screen was to read, so clearly they are experimenting with that part of it too (it was a little tough to read, especially with sunlight reflecting off the LCD screen — or maybe that’s just my aging boomer eyes).  But it was very user-friendly — I think it was much quicker than a paper ballot would have been.

One last footnote:  In my speculation about the exit poll problems in 2004, I have given great weight — probably too much, in retrospect — to the presence of the network logos on the survey questionnaire, the exit poll "ballot box," and on the interviewer name tags.  Note that Melanie did not notice that network identification until she was holding the device in her hands.  I asked her specifically when she first noticed the logos. Her answer:

I noticed the logo after I had started to read the questions on the device — it was about 2-3" square, at the top left of the device — he didn’t say anything about it and I just happened to notice it after I had read the first question or 2.  It was the usual CBS logo, but it wasn’t a particularly bright color — it sort of blended in to the dark gray or black color of the device, so it wasn’t splashy. I assumed that was on purpose, as was his failure to tell me whose poll it was.

Of course, Melanie was just one respondent and amounts to a sample size of one.  Of course, as a colleague of mine once told me, the plural of anecdote is data.

**An MP source reports that Edison-Mitofsky, the company that does the pooled exit polls for the NEP consortium did not conduct the CBS/New York Times exit poll in Connecticut last week.