Microsoft is crowing about a study that it says shows Hotmail having better spam filtering than Gmail or Yahoo. Really? If that's the case, then there's been a huge turnaround in Hotmail's spam-filtering accuracy. Is there any Devil in the detail? Of course there is, outlined here in The Long View...
By Richi Jennings.
On Monday, Microsoft's Dick Craddock, the group program manager of Hotmail, published a blog post crowing about Hotmail's spam filtering performance. In it, he claims that the typical effectiveness of the filter is better than 97%. Aside from the fact that 97% is hardly a state-of-the-art effectiveness rate, there are many aspects of the study methodology that just don't add up.
Where's the legitimate email?
Spam filters don't just filter out spam, they also filter in legitimate email. It's about correctly categorizing email of both types, not just removing the bad stuff.
However, there seems to have been little effort made to have test accounts receive email from legitimate sources. And no effort was made to solicit non-bulk email of any kind.
Artificial environment
If I deliberately give my email address to a spammer, it's statistically dumb to categorize email received from that spammer as legitimate email. That's because spam filters are supposed to work in the real world -- a world where those senders are sources of spam.
The vast majority of the sites where the researchers seeded test email addresses were at-best shady, and at-worst blatant spammers. This sets up a totally artificial environment for the spam filter to work in.
In the real world, most people give their precious email addresses to legitimate senders, not to websites such as cragmedic.ru or cash-loans.name.
False positives
The effectiveness of filtering spam from the inbox is just one dimension of spam filter accuracy. The other dimension is ensuring that all the legitimate email gets delivered.
As statisticians would say, you need to minimize the false positives as well as the false negatives. This study focuses only on the number of false negatives (or, as Microsoft puts it, spam in the inbox).
The thing is, false positives are just as important -- perhaps even more important. A few spam messages in your inbox are irritating, but losing an important message in your spam folder can have concrete business consequences. And from a pure productivity standpoint, the cost of a spam message in your inbox is seconds, but the cost of a false positive can be minutes -- and that's just the ones you find out about!
N00b shibboleth fail
I just can't resist one final jibe. If you want to be taken seriously in the spam-filtering world, do make sure you capitalize correctly. SPAM is the name of a canned meat product, trademarked by Hormel foods; unsolicited bulk email is simply spam.
To summarize: Testing spam filters is hard
Look, comparative tests like this are extremely hard to get right. I'll even say it's impossible to run a perfect test -- one that reflects reality. It's possible to get reasonably close to perfection; but this Microsoft test is far from that.
In order to have a fair test, there's countless pitfalls to avoid. We all have to start somewhere, but newcomers to the spam technology scene always get this stuff wrong (for suitably big values of "always").
As I hinted before, there's plenty more that's wrong with this test. In a future article, I plan to write about other common pitfalls of comparative spam-filter testing, as well as how to do it right (or at least, less wrong).
What do you think? Comment below...
Richi Jennings is an independent analyst/consultant, specializing in blogging, email, and security. As well as The Long View, he's also the creator and main author of Computerworld's IT Blogwatch, for which he has won ASBPE and Neal awards on behalf of IDG Enterprise. A cross-functional IT geek since 1985, you can follow him as @richi on Twitter, pretend to be richij's friend on Facebook, Encircle richij on Google Plus, or just use good old email: TLV@richij.com. You can also read Richi's full profile and disclosure of his industry affiliations.