The Wall Street Journal Online‘s Carl Bialik is on the case of another non-random sample survey conducted online. In this week’s Numbers Guy column (free to all) Bialik writes about a study done by the salary compensation firm salary.com that attempts to estimate how much it would cost to replace a stay-at-home mom with salaried professionals. While Bialik focuses mostly on the questionable ways this study attempts to "valuate the ‘mom job,’" as Salary.com puts it, readers should not overlook the non-random way they collected the data used to make those estimates.
Bialik tells us that Salary.com conducted an online survey that asked 400 mothers about "how many hours they spend on each job function each day." How did Salary.com select those 400 women? You won’t find the answer on the Salary.com online release, but Bialik tells us that to get these 400 interviews, "the company emailed 30,000 people who use its Web site."
That’s a long way from a random sample of anything. At best, it may represent women who registered to use Salary.com – it cannot possibly represent all mothers nationwide with any statistical precision. Now in fairness, this "study" differs from the other non-random internet survey that Bialik looked at recently, in that it makes no explicit claim of being "scientific" or possessing a "margin of error." However, Salary.com was more than willing to make the implicit claim that their study represents all mothers nationwide.
Check the "facts" they highlight from the "Mom Salary Study." Salary.com is perfectly happy to use their 400 non-random interviews to project statistics about "moms" in general:
"Working moms get less sleep. Working Moms reported getting only 6.4 hours of sleep per night, versus 6.7 for the Stay at Home Moms."
"Working Moms and Stay at Home moms both spend roughly 4 hours per week nurturing the emotional needs of their kids in the "mom job" of psychologist."
"Moms work an average of 90 hours a week whether they are a Working Mom or a Stay at Home Mom"
And so on.
This criticism may seem like overkill given the questionable methodology Salary.com used to translate these time reports into salary estimates (see Bialik’s piece), but it involves the most important bedrock principle of survey research. In order to use a "survey" of 400 people to estimate the characteristics of 30 million or more, you need to start with a random sample.
That may seem obvious, but it’s a principle that a number of otherwise smart reporters seem to have missed.
PS: An opinion piece by Wendy McElroy of Fox News yesterday also criticized the Salary.com study yesterday, although her column included this odd bit of information:
Salary.com’s press release cleared up one issue.
The site had conducted a survey, not a study, as the majority of the
media reported. A study is a scholarly or scientific investigation that
uses controls to prevent bias and error. A statistical survey collects
data by interviewing or asking questions of individuals. A survey is
less rigorous but, depending on its methodology, it can produce
valuable results.
Um..really? MP is not familiar with that particular distinction (though we heard something similar recently). The last time I checked, "statistical survey" implies a random sample that is projective of some larger population.
By the way, McElroy also takes issue with the fact that sample of stay-at-home mothers is tiny fraction (".00357 percent") of all stay-at-home moms. Again, the basic principle of survey research: The issue is not the size of the sample, but the way it’s selected. A sample of two hundred can be used to make estimates of a 5.6 million (within a predictable range of statistical sampling error) provided that the sample is chosen at random. This one was not.
Pointing out the obvious again Mark? More than a dozen paragraphs crafted to tell us the survey shouldn’t be taken seriously. You’re not paid for this “expertise” I hope.