The Zogby Poll of Troops in Iraq

February 28, 2006October 1, 2019 Mark Blumenthal39 Comments

[3/1 – Note to regular readers: I have added additional thoughts at the end of this post].

MP has received email from several readers asking about a just released survey of troops currently serving in Iraq conducted by Zogby International and noted in Nicholas Kristof’s column in the New York Times this morning.

According to the release, Zogby conducted the survey in collaboration with the Center for Peace and Global Studies at Le Moyne College. Those who click through to the Zogby summary will find the following methodology statement:

The survey included 944 military respondents interviewed at several undisclosed locations throughout Iraq. The names of the specific locations and specific personnel who conducted the survey are being withheld for security purposes. Surveys were conducted face-to-face using random sampling techniques. The margin of error for the survey, conducted Jan. 18 through Feb. 14, 2006, is 3.3 percentage points.

I wrote about another survey of troops in Iraq conducted last year by the Military Times newspapers (here and here) and know that polling active duty troops is no easy task. So I sent an email to John Zogby this morning asking if he could describe the survey a bit more in general terms if not in specifics.

Zogby returned my call this afternoon. While not exactly a fan of this site (to put it mildly) he was courteous enough to provide a more in-depth explanation of what his company did and why he is unwilling to disclose more publicly. Unfortunately, the ground rules for our conversation prevent me from sharing much of what he told me. But I did come away convinced that Zogby has good reason to withhold the details of how he was able to interview U.S. troops the way he did. More disclosure could put the interviewers’ lives at risk.

Here is what I can say:

~~The Center for Peace and Global Studies paid Zogby to conduct the study but otherwise played no role in conducting interviews or gathering the data.~~ [Correction: In our conversation Zogby indicated that an "anti-war" sponsor paid for the survey but played no role in conducting interviews or gathering the data. I wrongly assumed he meant the Center for Peace and Global Security, whose faculty according to a story in yesterday’s Syracuse Post Standard, did help "develop and word the poll’s questions." In fact, the funder was "a wealthy war opponent who [Zogby] would not name." Thanks to reader Bob for pointing out the Post Standard story. Apologies for the error. See this update for more details].

According to the procedure Zogby described, respondents were intercepted randomly (e.g. they were not self selected) at multiple locations throughout Iraq (e.g. not just in the so-called "Green
Zone") and interviewed using a paper questionnaire that they filled out with the assistance of an interviewer.

Zogby was willing to share the specific geographic locations where they collected data on the condition I not repeat them. I passed on the offer as my knowledge of Iraq and military operations there is cursory at best, but I have no doubt his offer was genuine.

Zogby provided Nick Kristof and others reporting on the poll full details about his methodology on an "off-the-record" basis.

So in short, I can tell you that Zogby found a creative solution to the difficult problem of polling troops in Iraq, but I promised to say no more than that. I asked Zogby what advice he would offer data consumers who find this all puzzling. In this case, he said, "you have to trust me."

PS: I neglected a hat tip to the reader who blogs at Fickle Minded and first emailed me about the Zogby poll.

PPS (3/1): Readers should not interpret my commentary above as either an endorsement or criticism of Zogby’s methodology. However, in response to some of the very reasonable questions posed by readers in comments and email, I want to clarify a few points.

First, to be clear, the ~~Center for Peace and Global Studies is in effect a "partisan" sponsor in that, according to Zogby, they opposes the war in Iraq.~~ [Correction: As noted above, the survey had a partisan sponsor that opposes the war in Iraq. However, that sponsor was not the Center for Peace and Global Studies, which collaborated on the study but did not fund it. Apparently, the Center, despite its name, is a non-partisan academic institution that takes no formal position on the Iraq war. The confusion on this point was mine. Again, apologies for the error].

Second, while Zogby says his interviewers selected respondents randomly at various locations, he makes no claim of random selection with respect to the locations involved. I apologize for being so vague, but the most I can say is that the method Zogby used to gain access to those locations constrained his ability to make random selections.

So to evaluate this survey, one important question is whether the troop populations accessible at the locations Zogby selected are representative of all troops in Iraq. This issue is analogous to the question of whether precincts selected for an exit poll are representative of all precincts in a given state (although exit poll precinct selection is usually random). Zogby believes the locations involved provide a reasonably representative sampling and, as noted above, he offered to share the names of specific locations on an off-the-record basis. I declined largely because I lack the knowledge and resources to make an independent assessment. So for me, this question remains open.

Third, even if consumers of this data knew all that I know about how Zogby’s interviewers "walked up to troops" (as commenter Karen puts it), they would still have questions about the impact of such an interaction might have on the kinds of troops most likely to agree to participate in the survey. Consider the exit poll example again. Even though exit pollsters have disclosed the procedures they use to train interviewers and select respondents, we still debate the effect of those procedures on the kinds of voters that choose to participate. Disclosure in that case cannot resolve all questions, but it at least enables an informed debate. Unfortunately, such discussion and debate is impossible in this case.

The survey procedures Zogby described to me involved compromises analogous to those used in surveys of Katrina victims conducted by the Washington Post/Kaiser/Harvard and CNN/USAToday/Gallup. A survey of troops in Iraq would be impossible without some sort of methodological compromise along these lines, and Zogby’s approach may be the best available. Yet my "bottom line" on this survey remains uncertain. I hope that those with more military expertise will assess how well it represents the troops in Iraq in terms of the age, rank, branch of service and any other similar characteristics (and no, I did not ask Zogby whether he weighted or adjusted his results).

The issue of disclosure is more difficult. Note that for the Katrina surveys, news organizations clearly characterized the results in terms of the limitations of the survey design. For example, the Washington Post reported results from "evacuees living in shelters." USAToday reported on the opinions of "residents who sought help from the Red Cross." Both organizations spelled out the methods used to contact respondents so inquisitive consumers could reach their own conclusions about the value of the data. Yes, Zogby has good reason for withholding the details in this case, but the secrecy limits the ability of consumers to evaluate the data and of news organizations to report it.

The toughest question here is whether it is appropriate for news organizations to report on a survey with partisan sponsorship that requires readers and viewers to place an unusual degree of trust in unspecified methods and procedures. I will admit I do not have a good answer for that one — different news organizations will apply different standards — but big "grains of salt" are certainly in order.

PPPS: In addition to Democracy Project (which has track-back links below), see also the posts by The Officer’s Club and Murdoc Online for a different perspectives on the Zogby poll.

Also a reader emails with a very good question that I probably should have put to Zogby:

What is the problem with releasing the demographics on the poll? Were it truly
representative, it would have a breakdown of service branches proportional to
the military in Iraq. Same with the breakdown for gender, age, etc. I can’t
imagine any security risk in releasing the demographics, or motive for keeping
them hidden other than they would expose the lack of validity of the survey.

Neither the poll release nor (I’m told) the full report available for sale provides statistics on the demographic composition of the sample.

Mark Blumenthal

Mark Blumenthal is political pollster with deep and varied experience across survey research, campaigns, and media. The original "Mystery Pollster" and co-creator of Pollster.com, he explains complex concepts to a multitude of audiences and how data informs politics and decision-making. A researcher and consultant who crafts effective questions and identifies innovative solutions to deliver results. An award winning political journalist who brings insights and crafts compelling narratives from chaotic data.