Debates: Focus Groups as Reality TV

Ok, so apparently, MSNBC has pulled the plug on its plan to have Republican Pollster Frank Luntz conduct a live focus group on MSNBC among uncertain voters tonight before and after the debate. According to Roll Call (subscription required), they have abandoned the plan to do the focus groups altogether. For the reasons I’ll provide on the jump, that’s probably a good thing. Nonetheless, since another network may well be planning something similar, it’s probably worth a providing a little background on focus groups, on their limitations why I don’t have much faith in focus groups as reality TV. Click for more:

Few tools of survey research are as widely used, much derided and frequently misunderstood as the focus group. A focus group is a free flowing, in-person discussion among 8-12 participants led by a specially trained moderator. Researchers usually conduct focus groups in special rooms equipped with microphones and a one-way mirror that allow others to watch and record the discussion without being seen (though all of this is always fully disclosed to the participants beforehand).

The advantage of a focus group is that it allows the researcher to go beyond the limits of standardized survey questionnaires. Participants can speak freely and the moderator can improvise, probing unexpected issues as they arise. Because of the in-person format, focus groups also allow for show-and-tell. Market researchers use focus groups to test every imaginable consumer product, while political pollsters use them to gauge reactions
to television, radio and direct mail advertising.

Focus groups do have important limitations that are not well understood. Although focus group recruiters try to make the participants as representative as possible, the focus group is not a projective random sample, like a poll. Participants usually live near the facility. As the response rates are miniscule given the time commitment, participants usually receive monetary incentives (usually $50-$75) to encourage participation. Recruiters also seek to fill specific quotas for specified demographic characteristics (a mix of ages, for example). Thus, we simply cannot count answers in a focus group to estimate the reactions of a larger population. In other words, if 20 of 30 “undecided voters” react a certain way to the debate tonight, we cannot conclude that 66% of all undecided voters nationally feel the same way.

A second limitation is what researchers call “group dynamic.” In a focus group, participants are often influenced or cowed by the opinions of others in the group. If one dominant personality loudly stakes out a position, others tend to hide or modify their contradictory views. To control this dynamic, we typically ask focus group participants to write down initial impressions at intervals throughout the groups to root them in their initial reactions. Moreover, the role of the moderator is critical to countering the loudest and most vocal while encouraging the more timid participants to share their true feelings.

Finally, the artificial nature of the focus group is often a poor way to judge how the information from advertising (or the fallout of a debate) will be processed in the real world. For example, focus group participants often express genuine antipathy for negative advertisements and reject the information contained in them as false and unfair. Yet in the real world, as the recent campaign has demonstrated powerfully, such advertising can still communicate negative information with ruthless effectiveness. Also, People no doubt watch advertising much more closely and critically in focus group than in their living rooms. Finally, as has noted about debates in recent days (here, here and here), the media coverage in the days after the debate typically has more impact on the race than initial reactions to the debate itself.

All of this brings us back to the growing practice of conducting focus groups as live reality TV. Unlike the traditional focus group, the networks put people on a soundstage in a brightly lit studio, where the participants surely know they are on live television. I am not aware of any formal research on TV focus groups, but it seems that if peer pressure from ten strangers leads a participant to hide or alter an opinion, what is the effect speaking freely to several million? I would also imagine that those willing to participate in a live broadcast differ from those who might shy away. Having watched these in the past, it seems that many of the participants come ready to perform rather than just share opinions. They often look like they are imitating the schtick of the cable news pundits, rooting themselves in a position and arguing it, rather than reacting as they might while watching the debates at home.

The appeal of these groups to the television producers is obvious. The groups allow for immediate reactions, unlike instant polls that take hours to conduct and analyze. And I suppose it is better to get immediate reactions from ordinary people, no matter how artificial the setting, than to listen to journalist/pundits prattle on and on.

Later today, some additional thoughts on other ways you might see the networks try to poll debate reaction tonight…