As of last week (before yesterday’s indictment news) what portion of Americans believed “Bush administration officials” did something “illegal or unethical” in connection with the leaking of a CIA operative’s identity? CNN’s summary of the latest CNN/USAToday/Gallup poll may have left some readers a bit confused. A quick lesson in survey methodology may help clear up the confusion.
The CNN story leads with the following sentence:
Only one in 10 Americans said they believe Bush administration officials did nothing illegal or unethical in connection with the leaking of a CIA operative’s identity.
But then three paragraphs later, the story tells us:
The poll was split nearly evenly on what respondents thought of Bush officials’ ethical standards — 51 percent saying they were excellent or good and 48 percent saying they were not good or poor.
Huh? Fifty-one percent (51%) of Americans rate the ethical standards of Bush administration figures as “good” or “excellent,” yet only 10% say they did nothing “illegal or unethical” in the Plame case. How can that be?
One possible if unlikely explanation is that the more general question came first. Perhaps, even before the announcement of the Libby indictment, Americans were evenly divided on the general ethical standards of Bush administration officials even though only 10% believe they did nothing “illegal or unethical” in the specific example of the Plame case.
A better and more likely explanation has to do with the way Gallup asked the question that produced the 10% result and the way CNN chose to present it. Let’s take a look at the question:
As you may know, several members of the Bush administration have been accused of leaking to reporters the identity of a woman working for the CIA. Which of the following statements best describes your view of top Bush administration officials in these matters – some Bush administration officials did something illegal, no Bush administration officials did anything illegal, but some officials did something unethical, or no Bush administration officials did anything seriously wrong?
39% did something illegal
39% unethical, not illegal
10% nothing wrong
12% had no opinion
This question forces a choice between three alternatives with one category (not illegal but unethical) clearly in the middle. Survey methodologists have long debated the merits of presenting an expclit middle alternative choice in survey questions. On the one hand, the true attitudes of the respondents may sometimes fall in the middle. On the other, some respondents may use a middle category as an “easy out” rather than thinking hard about the issue at hand. The academic research on this issue is mixed (see O’Muircheatough, Krosnick and Helic, 2000, for a summary), but even then the academic research MP knows is not exactly on point with this example, as it tends to focus on very concise answer categories, such as whether the government should spend “more, about the same or less” on various programs.
What the academic literature tells us more clearly is that “easy out” behavior by respondents (something methodologists call “survey satisificing”) appears to occur more often when questions are wordy and confusing. Ironically, a recent working paper based on experiments by the Gallup Organization (Holbrook, Krosnick, Moore and Tourangeau, 2005) on “order effects” (whether the order of answer categories affects the results) reached the following conclusion (pp. 22-23):
We found that response order effects were more common among questions that were more difficult to comprehend, when the response options were complete sentences instead of single words or phrases, and among questions involving response options that were not mutually exclusive. These findings are consistent with a large body of evidence from previous research that have found response order effects to be stronger for longer questions, questions with longer response options, and those with more difficult language (Bishop and Smith 2001; Payne 1949/1950; although McClendon 1986a found no such relation).
The authors of the report concluded with this advice for pollsters (p. 27) :
Our findings have a number of implications for survey research practice, though they are hardly surprising. Because response order effects are stronger in questions that are more difficult to comprehend, researchers should strive to keep their language as simple as possible. Advice of this sort is offered in numerous research methods textbooks, but a cursory examination of many survey questionnaires suggests that there may still be room for researchers to heed this advice more faithfully. It is not uncommon to see questions laden with social science jargon and long sentences involving many multi-syllabic words. Because survey researchers are usually well-educated, they may find such questions to be easily understandable.
With all of that in mind, let’s consider that Gallup question again. Is it likely that a large number of Americans considered the actions of “Bush administration officials” in the Plame affair unethical but not illegal as of last week? Certainly. So posing a question with a middle category in this case seems reasonable. However, the question they wrote is a it long and wordy side. For example, it repeats the phrase “Bush administration officials” three times in one sentence, throwing in “officials” a forth time. Why not simply ask whether Bush administration officials “did something illegal, did something legal but unethical or did nothing seriously wrong?” Another complication is that the middle category poses something of a double-negative: not illegal but unethical.
MP wonders how many respondents found this question confusing or hard to follow. MP also wonders whether some heard the question as having four choices rather than three. If one does not listen carefully the middle option may sound like two distinct choices: “no Bush administration officials did anything illegal, but some officials did something unethical.” Is it possible that some respondents replied that they preferred the second option because they thought it said only that Bush officials “didn’t do anything illegal?”
MP encourages readers to try this experiment: Read the question aloud to someone and ask them to answer it. Better yet if that person lacks a college degree. Are they confused? Are they confused? Do they ask you to read the question again?
In a perfect world of unlimited budgets and no time constraints, survey organizations would conduct extensive pre-testing to check against such confusion. MP has no knowledge of how carefully Gallup pre-tests its surveys, but his experience is that most pollsters lack the time and budget to pre-test as thoroughly as they would like.
So what can we make of results of this question? Here is what we know:
- On another simpler question on the same survey, 51% rated the “ethical standards of top administration officials” as excellent or good, 48% as not good or poor.
- On the same question more than half the respondents chose one of the two middle categories: 11% rated the ethical standards excellent, 40% good, 19% not good and 29% poor.
- When Gallup tabulated responses to the longer question by party identification, the “did something illegal” response follows the pattern we would expect: much higher among Democrats (55%) than Republicans (19%) with independents somewhere in the middle (45%).
- However, preference for the middle choice – unethical not illegal – was similar among Democrats (36%), independents (36%) and Republicans (44%).
MP’s admittedly subjective conclusion: The middle category of the three way illegal/ unethical question may probably included many respondents without well formed opinions in any direction. The 39% who believe Bush officials did something illegal is the most reliable and newsworthy result from this question, while the lines between other categories seem more blurry. The USA Today story and the Gallup release both put appropriately greater emphasis on the 39% result. CNN would have done better to follow their lead, rather than hyping the “only 10%” result.
Wilson: There Have Been Threats
In his first interview since the indictment of I. Lewis Libby, Joe Wilson tells Ed Bradley that ther
You seem to think there is an inconsistency in the responses. However, it is not inconsistent to believe that an some administration official did something unethical or illegal in this particular case but that the ethical standards overall among all officials are good.
If you took a poll asking if American soldiers did nothing unethical in the Abu Graib incident, a majority would say there was something unethical. However, a majority would also say that American soldiers ethical standards are excellent or good. Unless you think ethical criticism of any incident by members of a group implies that the group members are generally unethical, this is not inconsistent.
Perhaps, the people being polled were actually able to make this distinction.
I remember taking an anonymous survey about drug use in highschool. They gave us 5 options for each drug . . .
A) I use it daily
B) I use it weekly
C) I use it monthly
D) I use it rarely
E) I have never used it
After the 15th drug I remember getting bored with answering E and I filled in some C’s and D’s for heroin, PCP, peyote, etc. Anyway, I think there is a little peer pressure out there to come up with a middle of the road answer. Even in an anonymous survey you don’t want to lable yourself as an extremist. I remember my uncle telling me that I went to the most drug ridden school in the city soon after the results of the survey got published in the newspaper . . . 80% of freshman had snorted cocaine off a hooker’s ass, etc. Needless to say, I don’t believe federal numbers on teen drug use anymore. Whoever came up with the survey I took was trying to get inflated numbers.