An article in yesterday’s New York Times on interviews conducted among Katrina evacuees raises some very important questions about how to use a survey-like study that is not quite a projective survey. In this case, in MP’s view, the Times study was an appropriate effort to provide the best information available about a population essentially out of reach of true random sampling. But the example raises some useful general questions: When do we cross the line between the appropriate and inappropriate use of a study that cannot project the views of the population of interest with statistical precision? When do unavoidable compromises so degrade the sampling process that, as one thoughtful MP reader RS put it, "the concept of ‘best available’ ceases to be meaningful?" Unfortunately, these are not questions with easy answers.
A bit of background: A few weeks ago, in discussing the Zogby survey of US troops in Iraq, I wrote the following:
In the business world, commercial market researchers sometimes use non-random sampling (including many Internet based "panel" surveys) when rigorous probability samples are impractical or prohibitively expensive. However, the most ethical of these market researchers do not attempt to dress up such "convenience" samples as more than they are. Their clients pay for such projects on the assumption that the information obtained, while imperfect, is the best available.
Although obviously not market research, the Times Katrina evacuee study is just such a project. As described in the article and a "How the Study Was Conducted" sidebar, the Times researchers had access to the Red Cross database of 160,000 evacuees. They drew a random sample of that database and "conducted interviews by telephone" and used "standard survey methods in asking the questions and recording the answers."
Yet the Times researchers made it clear that they did not consider this "interview project" to be a "scientific poll:"
The study differs from a scientific survey in that the total population of evacuees is unknown and therefore could not be sampled precisely or randomly.
Three of every five of those interviewed lived in New Orleans before the hurricane; the rest are originally from the area surrounding the city. Almost two-thirds of the study’s participants were black and about a third were white – similar to Census Bureau figures for the city’s population. But the participants were older and slightly more likely to be women than the city’s population. (The precise racial, age and sex breakdown of all evacuees is not known.)
Because the answers from the 337 respondents in the study cannot be projected to a definable population, no margin of sampling error can be calculated. For that reason, the accompanying article avoids giving specific figures in most cases.
One important point: When a survey can be projected to a larger population, it is appropriate to use the results to describe that population ("48% of Americans believe," "53% of voters approve," etc.) But in this case, the Times authors are mostly careful to characterize the results as describing only those interviewed, not the larger population of those interviewed. For example:
Fewer than a quarter of the participants in the study have returned…
[M]ost of those interviewed favor…
The blacks interviewed were more likely…
[W]hile a majority of whites and blacks reported…
4 in 10 of those interviewed said…
Another quarter of those interviewed…
Yet oddly, for all the caution exhibited in the body of the article, the lead of the Times report does tend to go ahead and characterize all evacuees:
Nearly seven months after Hurricane Katrina flooded New Orleans and forced out hundreds of thousands of residents, most evacuees say they have not found a permanent place to live, have depleted their savings and consider their life worse than before the hurricane, according to interviews with more than 300 evacuees conducted by The New York Times.
Is the Times article an appropriate effort to provide the best information available in a situation where a true random sampling was impossible or, as reader PZ wondered, did they go ahead and use a "non-survey" inappropriately to "[draw] lots of conclusions on Katrina victims as if it was a survey?"
In this case, MP thinks the Times made the right call. They tell us that "fewer than a quarter" of their respondents "have returned to the same house they were living in before the hurricane." Based on these and other findings, it seems reasonable to conclude that "most evacuees" have not found permanent housing, especially since the study’s methodology probably underrepresented the poorest evacuees . As the Times article points out:
Some of the poorest people in New Orleans were not included in the project, in part because they did not have access to the database or posted the telephone numbers of emergency shelters where they left no forwarding information.
Although I think the Times handled this issue well, the general questions about the use of non-random sampling are much harder to resolve. Because the Times did not include "specific numbers" in their report, their study is less likely to produce a percentage that takes on a life of its own (as often happens when a dramatic number enters the public discourse). Yet even in this case, those who read the Katrina study and cite its findings are not likely to make the same fine distinctions about its ability to "project" the views of all evacuees as the Times authors.
Earlier in the week, before the Times released its Katrina study, I received an email from reader RS who asked a very perceptive question about when the concept of the "best information available" ceases to be meaningful:
I encounter this [problem] all the time because my research often involves C-suite executives, ultra high net worth individuals, corporate directors and the like. Sampling is a bitch with populations like these and I almost always recommend that my clients use qualitative research [focus groups] instead. As you can imagine, however, I often come under significant pressure to field a quantitative study – with my client promising to take the results "only for what they are worth" – only to find myself discussing cross-tabs the size of which wouldn’t fill a closet. So, I am not a real believer in "best available" data, because I don’t think clients (let alone journos) really can be expected to appropriately limit its use.
I wish I had an easy answer for this one. The question gets to the heart of an issue that the survey research profession — and consumers of its data — will face more and more as declining coverage and response rates further degrade our ability to interview random samples by telephone. In many ways, this question underlies many of the controversies we discuss here at MP. The best general advice I can offer data consumers is to get into the habit of reading the methodology section first and asking: Who was interviewed, how where they selected and how well did the selection process represent the population of interest?
[Typos corrected]
