Earlier this week, ABC News released poll results concerning the federal investigation of the leak of CIA operative Valerie Plame’s identity. The poll showed a sharp decline in the percentage of Americans who say that the White House is "fully cooperating" with the investigation and a large number who want Bush advisor Karl Rove to "lose his job . . . if investigators find that Rove leaked classified information."
On Wednesday, Jay Cost (aka the Horserace Blogger) excoriated the poll in comments posted on RedState.org. MP is instinctively skeptical of ad hominem attacks wrapped in overheated rhetoric. Cost’s piece — which labels the ABC poll variously as "patently absurd," "lousy or should I say tendentious," "done with the ostensible purpose of adding lighter fluid to a story," "screams in an unequivocal voice, ‘I am garbage, please debunk me!" — certainly seems to fall into that category. However, Cost is a well-read student of political science. While his conclusions about the ABC poll are questionable, his arguments are worthy of our consideration.
Let’s begin with the first question ABC asked about the Plame investigation:
As you may know, a federal prosecutor is investigating whether someone in the White House may have broken the law by identifying an undercover CIA agent to some news reporters. One reporter has gone to jail rather than reveal her source. How closely are you following this issue – very closely, somewhat closely, not too closely or not closely at all?
Cost has three complaints about this question: He doesn’t like the opening clause ("as you may know") because, he says, it is a "priming or cueing mechanism" intended to get the issue "to the front of people’s mind." He is troubled by the amount of information provided and the apparent narrative it creates. He argues that ABC "puts all [the] pieces together" in a way that "frames" them "into one sensible story." He concludes:
ABC News is playing a subtle psychological trick with the public here — trying to make them respond that they are paying attention when they are not actually paying attention.
Let’s start with some explanation. Pollsters frequently use the phrase "as you may know," to introduce unfamiliar information. The phrase typically serves as both a transition from the previous question and a polite softener to avoid insulting knowledgeable respondents. An interview is a conversation, and this clause is a nice way of saying, "yes, I realize you may already know these details; no, I don’t think you’re stupid, so please bear with me." MP grants that it is a bit odd to use the phrase in a probe of awareness, but if research exists to prove this phrase inflates awareness, MP has not seen it.
Second, Cost has a point when he warns that "informed" measures may exaggerate reported awareness (Cost refers to this as a "self selection" problem). In this case, the pollsters have two ways to ask about awareness. They can ask a purely open-ended (or "unaided") question: "What are some of the stories in the news you have been following lately?" Or they can ask a closed-ended "aided" question (How closely are you following this issue?) that provides just enough detail to trigger the respondent’s memory.
Both approaches have drawbacks. The open-ended question may tend to underestimate true awareness, as some less verbal respondents may hold back opinions. Others may have genuine memories that require a "trigger." (Imagine: "Oh, the story about the reporter who went to jail? Oh yes, I remember now…"). On the other hand, as Cost suggests, the social pressure of the interview can induce some respondents to want to seem better informed than they are. Thus, unaided questions may slightly understate awareness while aided questions may slightly overstate it. The truth usually falls somewhere in between.
Having said all this, MP tends to agree with Cost that the ABC awareness question includes an unusual amount of detail. The issue is whether the combination of (a) the introductory phrase "as you may know," (b) the level of detail and (c) the supposed narrative "framing" produce a meaningfully higher level of reported awareness than a more bare-bones question.
Fortunately for us, the Pew Research Center included just such a question with nearly identical answer categories on a survey fielded on the very same dates (July 13-17). The Pew question was:
Thinking again about news stories: How closely have you followed reports that White House adviser Karl Rove may have leaked classified information about a CIA agent very closely, fairly closely, not too closely, or not at all closely?
The question has no "as you may know" introductory clause and describes the issue in just 14 words (compared to 37 for ABC). Yet the results are remarkably similar. Pew shows 23% say they followed the story "very closely," 25% "somewhat closely" (for a total of 48%). ABC shows 21% "very closely," 32% "fairly closely" (for a total of 53%). The ABC survey gets a slightly lower percentage for "very closely," but a larger response in the second category (only the second difference appears to be statistically significant). However, since ABC labels their second category "somewhat closely" compared to Pew’s "fairly closely," we cannot be certain what caused the slight difference — the answer category or the text before it.
Regardless of the explanation, the differences in the results are trivial and, in MP’s view, not worth all of Cost’s huffing and puffing. We would reach the same substantive conclusion about awareness of the Plame story from either poll’s results.
But what about his s point that informed questions slightly overstate the true level of awareness? What do we make of that? When analyzing these sorts of results, an astute survey consumer should always ask, "compared to what?" In this case, how does awareness of the Plame leak compare to awareness of other issues. The numbers become much more useful and meaningful as we put them into context.
Once again, the Pew survey provides an answer: At 23% very closely, awareness of the leak ranked well behind the terrorist bombings in London (48%), the war in Iraq (43%) and the recent Gulf Coast Hurricanes (38%). It ranked at the same level as the O’Connor retirement (24%) and ahead of "the move by a Chinese firm to buy the American oil company Unocal (11%).
In preparing this post, MP also stumbled on an amazing resource for these sorts of comparisons buried on the Pew Research Center website. Pew has been asking awareness questions about public issues using the same methodology for nearly twenty years. On this page, they provide familiarity ratings (the "very closely" percentage) for over 1,100 different stories they have asked about from 1986 to 2004. That’s a lot of context!
Cost has more complaints about the ABC survey that are also worth of some discussion, but my blogging time is up for today. I’ll come back to the rest in a subsequent post.
UPDATE (7/22): MP wondered about why the Pew News Interest Index reports the percentage that say they follow an issue "very closely," so he emailed Scott Keeter, the director of Survey Resarch at Pew ans asked. Here is Keeter’s answer:
We tend to find that the “very closely” category is more sensitive to change and to differences across items. It is also probably less subject to social desirability pressures, since respondents have at least two other categories of attention to use if they feel the need to show they are not completely tuned out.
Also, Jay Cost responds in the comments section.
The Pew write-up also provides a useful comparison to other insider controversies:
http://people-press.org/reports/display.php3?ReportID=250
To quote:
“The survey finds that attention to the Rove story is comparable to interest in past Washington controversies, including the 1997 ethics charges against House Speaker Newt Gingrich, former Senate Majority Leader Trent Lott’s resignation in 2003 following remarks at Strom Thurmond’s birthday party, and charges in 1991 that then-White House chief of staff John Sununu used military aircraft for personal trips. But the story has much greater resonance than the recent ethics complaints made against House Majority Leader Tom DeLay.”
Minor but fun point: Though Pew and ABC found roughly the same level of attention, Pew framed that figure as “only about half of the American public” whereas ABC framed it as “a fairly broad level of attention.”
On the subject of praising Pew, I’d also like to commend them for making their raw data available to the public (each dataset has a relativly brief embargo period). I’m not certain that everyone is fully aware of just how rich Pew’s aggregated data are.
Dear Mystery Pollster,
That was a very interesting and well-reasoned response, and I am flattered that you took the time to read what I had written. I am prepared to cede a few of your points here, and am interested to see if you have more to say on the subject. My response is as follows:
(1) You are right that I was being a little hyperbolic in the post. However, I would not say that I was engaging in ad hominem attacks. I certainly did not mean to, at any rate. I thought I was careful not to directly impugn anybody’s intentions beyond what the inferences I was drawing allowed, i.e. either sloppy or tendentious. But, in retrospect, I might not have been.
(2) Your point is taken about the “as you may know” introductory clause, though I would add that if you are granting the oddness of using it on this type of question, you should also take note that this question is, as far as I could tell, the first question of the poll. So the transition justification is not in play here (assuming that I am right, which I might not be).
I would say generally that (a) I do not like the phrase used in any context in a poll and (b) you would not hear me say that a poll is a conversation. First off, I just do not think it is, as the reader has not designed the questions (usually) and cannot deviate from the script even if he/she has. Second, I think that getting oneself into the mindset that a poll is a conversation is a bad idea. Conversations potentially interject the pollster into the data, and potentially in systematic ways (e.g. a white pollster feels more comfortable being “conversational” with a white respondent than a black respondent). This is a minor point and certainly not essential to your over all rejoinder, but I thought I would mention it (although in retrospect, I wonder if it it was worth all the sub-headings I’ve placed here).
(3) In terms of your potential remedies for how to cue knowledge for recall questions, it seems to me that you are trying to salvage ABC’s poll here by couching them in terms of industry standards. That is fair enough, though I think the oddness which I pointed out still, in large measure, stands (if I am right in my (2), it stands more than you had previously argued). I also would respond that I have great problems with the industry, as many in my profession do. *Everything* that I have read on public opinion and political behavior indicates that the public is generally ignorant about politics, much much more than the average political junkie presumes. Thus, I am instantly skeptical about polls that seek to ask about summer scandals of what are, in my judgment, about small potatoes. Given the known ignorance of the public on matters much more important than this, one should enter into question design with the assumption that they do not know the details of the case. And therefore one should presume that a question about one’s level of attention is really not about one’s level of attention in an objective sense. It is little more than what the respondent perceives of himself . Anything beyond this entirely subjective sense absolutely requires a whole host of secondary questions designed, as I said, to test the public. This would cost ABC a ton of money, and with very little payoff, as it would only confirm/disconfirm the data they were going to present anyway (rather than be data they directly presented).
Thus, the fact that this compares to other results on similar questions means two things: first (as I will argue in (5)) they were a little lucky; second, they — like other media pollsters — are engaging in a practice in which they should not be engaged.
Perhaps now you are seeing why I was so hyperbolic in the post. I imagine that, at the end of the day, you and I will probably have to agree to disagree here. But my years as a poly sci graduate student have really, really, really aggravated me when it comes to commercial polling. I have problems with the methods, the (lack of) publicity in terms of both methods and results, the perceived agenda (i.e. frequently it seems like the goal is not to tell the truth, but to tell newsworthy truth — a problem which transcends polling, imo), and the way they are interpreted.
I have specific problems with the ABC poll, particulary the nature of these questions — but the hyperbole really stems from their participation in a practice which I find very frustrating.
(4) You understate my claim when you say that the issue for me is “social pressure.” I am not simply saying that somebody will want to seem to be informed, though I think that is the case. I am, rather, saying that somebody will *genuinely* think they are informed. Self-selection is, in my judgment, a big no-no UNLESS one has some additional questions that could provide some objective confirmation. This is, as I mentioned in the column, what burned Verba, Nie and Petrocik in 1979. And that was on an issue of political ideology — a BIG issue. People cannot accurately self-select themselves into conservative, moderate and liberal. Why should one think they can do so on this?
(5) The area where I will genuinely back down is in terms of the results. By and large I like Pew, and their question is good relative to ABC’s. At the same time, though, that does not, in my opinion, exonerate ABC from sloppy questioning. They got lucky with results that replicate Pew’s (and, as I mentioned before, I do not think questions like this should ever be asked). If I, as a student working on my dissertation, applied for a grant so I could conduct a poll on this issue, and I submitted to the University of Chicago those ABC News questions, they would turn me down cold. Period. John Brehm would probably revoke my ABD status and make me retake Data Analysis. That question should never have found its way to ABC’s sample.
(6) I will be interested to see the second part of your response — as I would like to know if there are other polls that produced similar results on the other questions. The Pew question on understanding seems to exonerate the results of the ABC one, but the other Pew questions on the Rove/Plame subject are sufficiently different to defy direct comparsions with the other ABC questions.
I will conclude by adding briefly to my original post in Red State — in part as a way to anticipate what you have to say (the kind of inside baseball that follows was not all that relevant for my original piece and its audience) . My theoretical problems with the other questions go much deeper than the problems I had with this self-selecting “how much have you been paying attention” question. I mentioned the cues that I think exist for the other questions, but I did not mention that, in my judgment, even if one eliminated all of the particular problems with the rest of the ABC poll, one should still not ask these questions.
Personally, I really think that questions on matters like these are doing little more than picking up the kind of noise that Zaller (1992) finds. This is just RAS in action — people recently Received information/opinion from elites on the subject, they Accepted it, and they Sampled their bundle of recent information for these answers. This is not an indication of independent thinking on the part of the public, or at least independent thinking along the lines of what ABC News is driving at here.
There are, of course, more optimistic ways of explaining public opinion (the on-line model or Alvarez and Brehm 2002’s value referencing model) and I really like them much more than Zaller’s model (they certainly are more optimistic), but I do not think they are really in play for topics like this. On-line works best for something like opinion of a presidential candidate, value referencing works best for something like variation on issues about abortion.
I just do not think this poll *means* anything, and I find myself very aggravated to see money used to conduct it and a bit to-do made over its results. Again, I would admit that ABC caught a little more ire than you might think they deserved. In my defense, I guess I can just say that I was so aggressive because they did something that is usually done poorly (from a theoretical standpoint) in a particularly poor (from a methodological standpoint) manner.
Best wishes,
Jay Cost
The ABC News poll states in its “as you may know” section that Patrick Fitzgerald is investigating “whether someone in the White House may have broken the law by identifying an undercover CIA agent to some news reporters…”
Not quite accurate. Fitzgerald is investigating whether someone–in the White House or not–broke the law. By confining the “as you know” part to “someone in the White House”, ABC is a) misstating the facts, by putting a limitation on Fitzgerald’s investigation that doesn’t really exist; and b) misstating the facts in such a way as to put focus on the White House, rather than another part of the government or the media.
The “as you may know” section should have been phrased “whether someone may have broken the law…”
I don’t know Jay Cost, and I’m sure this is unnecessary for me to point out to him, but some readers might enjoy being reminded that “this minor Rove matter” may compare to a certain minor campaign burglary in media attention, precisely because both acts point towards cultures of secrecy and unaccountability found in both the Nixon and Bush administrations far beyond the norms of modern presidencies.
I don’t know, but maybe because the press has always known about and not been able to do much about the overreaching Bush Admin secrecy they feel an obligation to report on a story that points to this dangerous presidential flaw. A flaw they have seen before and that led to disastrous ends.
This context may still not mean much to Jay Cost, but I think the news media is right to act when a dangerous administrative trait gets exposed to the public. Besides this story is the nexus of seeming WMD lies, an unpopular Iraq war, and the president’s political “brain.” Too sexy to ignore when coupled with a special prosecutor, a grand jury, and potential perjury in the White House. This too the media has seen before.
Of course the Bush Admin may be innocent of these suspicions, so we shall see.
I just posted a follow-up that responds generally to some of the points Jay raises. I wanted to briefly comment on his admittedly “minor point” about my references to survey interviews as a form of “conversation.”
I might have done better to say that a survey interview is a *form* of conversation. Yes, it is certainly a standardized, scripted conversation from which the interviewer should deviate rarely if at all. And yes, the respondent can only answer the questions asked in the manner allowed.
However, standardized though it may be, a survey still generally involves two human beings talking to each other. Many of the norms and etiquette of a conversation — such as being polite — still apply. Ideally, those who write standardized questions try to keep those norms in mind. That’s where transitions like “as you may know” come from.
I was probably thinking of the classic, Conversations at random: survey research as interviewers see it, (Converse and Schumann, 1974), which explores these issues at length. See:
http://www.worldcatlibraries.org/wcpa/ow/c528df3630d4a3e2.html
In response to Alex:
I would agree that the Bush Administration is particularly adept at being secretive, though I do not think it is appropriate for you to place it in the normative terms which you do.
All presidencies, which is to say all presidents and their administrations, have an inherent interest in being secretive — which is to say they have an interest in only disclosing information that benefits them and withholding information that harms them.
One can argue that the president should not have the ability to be as secretive as he wants. And I certainly agree with that. So, for that matter, did the Framers. Congress works as a check (even, btw, a Congress of the same party. See Chapter 2 of Mayhew’s Divided We Govern). Ditto for the Courts. Ditto even for the bureaucracy (the CIA leaks to the NY Times at the end of the election season are a good example).
In this context, I think your criticism of the Bush administration is actually quite unfair. Presuming that they are not doing things that are illegal (I am guessing you are not inclined to that point of view, but just presume it for the sake of argument), what you are really criticizing them for is their superiority. They are better at being secretive than the horribly undisciplined Clinton administration, the Bush 41 administration, the second term of the Reagan administration, the Carter administration. In terms of keeping things under its hat, the Bush Administration is surpassed only by FDR’s administration — which (I am guessing) you do not find to be morally noxious. I certainly do not.
At the end of the day, perhaps we should have a debate about the power of the executive branch *in light of* Bush 43’s ability to keep things under his hat, so to speak. Perhaps this indicates that the executive power, as it is currently constituted, is simply too vast — and that a president who knows how to wield it can move things beyond what we as a nation consider to be the proper limit. That sounds like an interesting debate, and you would probably find me on the anti-executive side (personally, I like a good, strong Congress).
However, I do not think that one can cast moral aspersions on Bush 43’s use of the executive branch. Well, one can, but they really boil down, in my judgment, to many of the ridiculous moral aspersions that conservatives put forth against Clinton: all puff and no stuff. He would only be acting immorally if he was violating the limits on executive power inherent to our current political settlement, as Nixon did. If that is the case, where is the fire? Where are the key indictments that concern abuse of power? All there is now is innuendo and deductive reasoning from people who are predisposed to think the worst of the man. Again, I would compare the suspicions of Bush 43 to many of the suspicions that conservatives had of Clinton.
In other words, excesses of the Bush 43 administration indicate how the law must be modified, not how the Bush 43 administration has trampeled upon the law. It indicates how political morality must be modified, not how Bush 43 has violated current political morality.