Focus Groups – What They’re Not

Yesterday, the Hotline On Call blog reported on focus groups conducted by Republican pollster Frank Luntz recently among Democrats in Iowa and New Hampshire.  Since MP assumes those observations will get noticed in the blogosphere (and since…well… they asked) now is a good time to talk a bit more about the pros and cons of focus groups. 

The Wikipedia has a good explanation of focus groups that includes this basic description: 

In traditional focus groups, a pre-screened (pre-qualified)
group of respondents gathers in the same room. They are pre-screened to
ensure that group members are part of the relevant target market and that the group is a representative subgroup of this market segment.
There are usually 8 to 12 members in the group, and the session usually
lasts for 1 to 2 hours. A moderator guides the group through a
discussion that probes attitudes about a client’s proposed products or
services. The discussion is unstructured (or loosely structured), and
the moderator encourages the free flow of ideas. Although the moderator
is seldom given specific questions to ask, he/she is often given a list
of objectives or an anticipated outline.

Client representatives observe the discussion from behind a one-way
mirror. Participants cannot see out, but the researchers and their
clients can see in. Usually, a video camera
records the meeting so that it can be seen by others who were not able
to travel to the focus group site. Researchers are examining more than
the spoken words. They also try to interpret facial expressions, body
language, and group dynamics. Transcripts are also created from the
video tape.

In traditional focus groups, a pre-screened (pre-qualified) group of respondents gathers in the same room. They are pre-screened to ensure that group members are part of the relevant target market and that the group is a representative subgroup of this market segment. There are usually 8 to 12 members in the group, and the session usually lasts for 1 to 2 hours. A moderator guides the group through a discussion that probes attitudes about a client’s proposed products or services. The discussion is unstructured (or loosely structured), and the moderator encourages the free flow of ideas. Although the moderator is seldom given specific questions to ask, he/she is often given a list of objectives or an anticipated outline.

Client representatives observe the discussion from behind a one-way mirror. Participants cannot see out, but the researchers and their clients can see in. Usually, a video camera records the meeting so that it can be seen by others who were not able to travel to the focus group site. Researchers are examining more than the spoken words. They also try to interpret facial expressions, body language, and group dynamics. Transcripts are also created from the video tape.

MP, like most pollsters, considers the focus group an invaluable tool in the measurement of public opinion.  The great advantage of the focus group is its wide open and unstructured format.  While a survey must follow a standardized structure and fit within tight time constraints, a focus group can be free-wheeling and spontaneous.  Participants can answer in their own words.  If the initial questions are confusing, the moderator can immediately explain or revise or take the conversation in new and unforeseen direction.  Also, the in-person format allows for the moderator to play "show and tell" with video clips, advertisements or new products – something that would be impossible over the phone.   

Traditionally, focus groups have been used most often as a "pilot test" of language, concepts and theories before conducting a formal survey.   Political consultants often use focus groups after conducting formal survey research to pilot test television spots and other forms of advertising.

Another reason for their great popularity, especially in the corporate world, is that research consumers find opinions expressed in a focus groups easier to understand and relate to than numbers in a table or chart.   As Wikipedia puts it, the results are "believable" and have "high apparent validity."

However, that ease of understanding can often lead to misuse.  Thus, it is important to remember the limitations of focus groups, especially the idea that a focus group is not a survey.  To use the research lingo, focus groups are qualitative not quantitative.  That is, they do not allow for projective, quantitative estimates for some larger population.  Put another way, you cannot count answers to a question posed in focus groups (hypothetically, 10 of 20 in an Iowa focus group like chocolate ice cream) and use them to make estimates about the views of a larger population (50% of Iowans like chocolate ice cream).  The reasons are that a sample size of 10 is tiny and more importantly, given the time and travel required, many in the population of interest will lack either the time or inclination to participate.   

Focus group recruiters typically offer a cash incentive (typically $50 to $75 but sometimes much more), a necessary practice that can create its own challenge.  Focus group researchers must deal with the "professional respondents" who would be happy to participate in focus groups several times a week.  (To hear how this process can break down, MP highly recommends the story that aired in March 2002 on the NPR program Marketplace — the focus group story begins at about 4:10).

Another limitation is the challenge from what researchers call "group dynamic," and everyone thinks of as "peer pressure."  If one highly opinionated participant makes a compelling or emotional argument, others in the group may have a hard time expressing contradictory opinions.  To get around this reluctance, researchers try to keep the demographic composition of groups as homogenous as possible (all female or all white, for example).  They will try to weed out those who might have an "expert" opinion or who typically speak with authority (such as teachers in a group about education issues).  They will also use written exercises to get respondents rooted in their opinions before the discussion starts. 

Unfortunately, the same format that makes focus groups easy to understand also makes them challenging to interpret objectively.  Focus group analysis can be a lot like interpreting a Rorschach inkblot.  It is all too easy to see what one wants to see in a focus group or to make too much out of too little.  That is why most serious researchers recommend following up focus groups with a quantitative survey to try to confirm their apparent findings. 

In the political context, focus groups can also be very sensitive to the kinds of people recruited and the nature of information presented.  Recruit only undecided voters and those unsure about their vote preference, and opinions are likely to change during the group.  Include strong partisans or those strongly committed to a given candidate, and opinions about candidates will be more resistant to change.  Moreover, since a focus group will often expose voters to more political information and discussion in a single evening than they typically experience all year, their reactions can sometimes be highly artificial and deceptive. 

All of which brings me to Frank Lutz’s latest focus group project.  It is hard to evaluate the findings since the Hotline report is second hand and includes no information about "size/demographic balance, etc." of the groups.   However, in general terms, what the Hotline report describes is more or less what an internal campaign pollster would do in Iowa or New Hampshire on behalf of a client running for president.  The focus group moderator would first discuss all the candidates, probing for awareness and existing attitudes.  At some point they would probably ask participants to "vote" for their favorite.  Then they would present more information or video-tape of some or all the candidates followed by yet more discussion and possibly a second vote. 

In this case, it appears that Luntz played just one video clip of each candidate (it would be hard to do more given the time constraint).  As such, the reactions he reports depend greatly on the clip he chose to play.  Those clips may or may not provide a decent simulation of the sort of exposure those candidates will get over the next two years.  So while Luntz observations are interesting, it is hard to know what to make of them.  But I know that won’t stop political junkies from speculating, so…enjoy!