Tuesday’s post on a question wording experiment by Rasmussen Reports yielded some interesting comments and email. Here is a sample.
The Rasmussen questions poses four answer categories (strongly approve, somewhat approve, somewhat disapprove or strongly disapprove) while the traditional job approval rating question poses just two (approve or disapprove). According to their report, the Rasmussen experiment showed that the four category version produced a smaller “don’t know” response and a hgher approval rating.
My observation that the word “somewhat” softens the choice and often makes the “somewhat” positive choice (somewhat approve, somewhat favor, etc), more appealing provoked an excellent question from reader Jon Willits in the comments section:
Couldn’t this work both ways? Might there be some people who would feel troubled saying “Disapprove”, but would feel ok about saying “Somewhat Disapprove?” Or is there some psychological reason to think this would only work in the positive direction?
The short answer is, yes, there is a psychological reason why “somewhat approve” may be more attractive than “somewhat disapprove,” particularly among respondents that are expending the minimal effort to answer the question. Survey methodologists have long noticed a phenomenon we call “acquiescence,” which is the tendency to choose the positive response regardless of the content of the question. See this 1999 article by Prof. John Jon Krosnick from the Annual Review of Psychology for a brief review of the “voluminous and consistently compelling” evidence as well as the psychological context.
Now, the theory behind acquiescence does not guarantee that asking respondents to choose between four categories of approval will always yield a bigger approval percentage than asking them to choose between just approve or disapprove. From Rasmussen’s description, it appears it did in one experiment, but we have not seen the actual data. Moreover, a survey based experiment — like any survey — is always prone to random variation.
Along those lines, I also heard from Doug Rivers at Polimetrix, who reports on several similar wording experiments conducted online using their online panel. They conducted a very similar test and found that a version of the approval question virtually identical to Rasmussen’s resulted in a “not sure” response of less than one percentage point compared to 4% for the the classic two-way choice (“approve or disapprove”). In this case, the lower “not sure” response did not translate into a stronger “approval” rating. In fact, both the approve and disapprove categories were slightly higher on the four-way choice.
Finally, I also heard from Emory political science Prof. Alan Abramowitz, an occasional contributor to DonkeyRising, who reports discovering a seemingly similar effect in the 2004 exit polls. Abramowitz was trying to compare self reported ideology among primary and general election voters. He reports this discovery:
The questions were slightly different [see reproductions below]. The general election exit poll simply asked respondents to describe themselves as liberal, moderate, or conservative. The primary exit polls give them five categories to choose from: very liberal, somewhat liberal, moderate, somewhat conservative, and very conservative. The percentage of moderates was considerably higher in the general election exit poll than in the primary exit polls for the same parties from the same states, but I think that this is probably not because Democratic primary voters are more liberal and Republican primary voters more conservative than Democratic and Republican general election voters, but because of the differences between the 3-option and 5-option questions [emphasis added].
Abramowitz may be right, of course, but the difference in the question makes it impossible to know for sure. And that is the point. Differences in question language or in the number or language of the answer choices can produce different results. Caveat pollster.
***
The exit poll ideology question from the 2004 primaries:
The exit poll ideology question from the 2004 general election:
