No, we're not losing the spam war

Forgive me, but I simply can't let David A. Milman's post from yesterday go unchallenged. We're not losing the spam war; the sky isn't falling. Let's take The Long View... Let me quote a few of David's points and respond to them. I do so not to criticize my fellow CW blogger, but to illustrate several common misconceptions about spam and how the anti-spam community fights it. 

First, the tease:

There's a spam war raging ... and computer users are losing.

Users aren't "losing" the war against spam. I've been tracking the growth of spam and spammers since 1985. The typical North American end-user's inbox experience is far better than it was, say, five years ago.

There's a simple reason for that. Spam filters are much more accurate than they were, and up-to-date filters are more widely deployed.

So what proportion of email is spam? In any discussion about spam, we're almost contractually obliged to quote these stats:

The numbers show the truly frightening expansion of spam. ... Spam is wide spread and growing ... impossible to stop. ... 88% of all inbound email sent to corporations ... 91.9% of email ... 30% increase.

However, spam volumes are a drop in the ocean of Internet traffic. Statistics such as "90% of email is spam" exaggerate the problem, for two reasons.

  1. Email network traffic volume is a tiny part of the bandwidth use on ISP backhauls; it pales into insignificance compared with applications such as YouTube, Hulu, and BitTorrent.
  2. More than 80% of the spam is never actually transmitted, because it's rejected due to poor sender reputation.

But wait, what about spam filters?

With many email programs allowing for strict filtering by word or even country of origin, there are protections in place against spam. ... The sheer proliferation of spam makes it easy for spammers to find users.

Spam filtering is far more clever than people often think. The "protections in place against spam" are much more sophisticated than "filtering by word" or "country of origin".

Spam filtering today is a sophisticated blend of connection analysis, behavioral analysis, and statistical content analysis. (As I mentioned, 80% of messages from senders with poor reputation get rejected, so these more sophisticated techniques are only employed on the remaining 20%.)

Filter accuracy is usually extremely high, with a very small false positive rate. Again, the end-user experience is in stark contrast to what they used to have to put up with. This is due to spam filters getting better and to there being more mailboxes protected by good spam filters.

Should we blame users, for encouraging spammers?

According to the Messaging Anti-Abuse Working Group's (MAAWG) 2010 Email Security and Usage Report, "half of email users in North America and in Western Europe have opened or accessed spam."

In fact, users aren't opening spam -- or at least this survey doesn't say they are. While MAAWG's surveying and statistical methods leave a little to be desired, the press reporting of them has often been shoddy.

Spam wars war

David A. Milman
Spam wars 2010

Richi Jennings
We're not losing the war

David A. Milman
Losing time is losing the war

Richi Jennings
The inbox is ground zero

David A. Milman:
Spam wars 2010: The last word

This is the second year I've seen these MAAWG data. In fact, I worked with MAAWG on the conclusions from the 2009 survey and was disappointed with the wording of some survey questions. They appeared to be worded so as to produce 'newsworthy' results -- I'm unclear whether that was by accident or design. Sadly, I was brought onto the project too late to influence the survey design; the 2010 methodology has been improved in some areas, but not in others.

The often-quoted "half of users open spam" factoid is a classic example. Although this statistic is often parroted, I simply can't draw this conclusion from the MAAWG data.

In fact, you get a very different picture if you analyze the actual question asked and the responses to the followup question -- "Why?"

What respondents actually said was that they can't always tell if a message is spam before they open it. Stop the press. I don't think that conclusion would surprise anyone -- but it's really not as newsworthy as, "ZOMG! Half of users open teh spams!!1!"

Botnets are a big problem:

95% of all spam is generated by botnets, infected machines that are used by cyber criminals as automated and anonymous spammers. ... [One] be rented for as little as $67 a day.

Yes, botnets are a big problem, but the good news is that we're getting better at fighting them.

Part of the message that MAAWG is trying to push home to ISPs is that they should be helping customers clean up after a malware infection. Indeed, the survey clearly shows that users expect their ISP to provide that service.

In addition, companies such as Microsoft are leading the charge to neuter botnets, by legal as well as technical means.

These two activities -- ISP remediation and botnet takedown -- are a growing part of today's industry response to the botnet menace. I'd have liked to have seen the process start sooner and grow faster, but we can't change the past.

In conclusion, we need to educate users:

Many computer users have been trained ... to identify spam and avoid it. However, even in 2010, not every computer user follows that training. ... One uniformed or ignorant employee using Facebook or opening a spam email at work can compromise an entire network. ... The war against spam is won a single inbox at a time.

Sadly, user education is all-but pointless. Decades of trying and failing to educate non-technical users in homes and businesses have proven that.

The war against spam is not "won a single inbox at a time." Tackling spam at an individual user level simply does not work -- experience has taught us that. Sure, you might be able to convince some of the end-users to guard against today's spammer social-engineering tricks, but what about tomorrow's, or the next day's? Fuggedaboudit.

I agree that a single user's actions can have big consequences for a network. But that's all the more reason not to rely on user action to perfectly conform with perfect training.

What's needed is a three-pronged approach:

  1. widespread use of good spam filters,
  2. ISPs detecting and disconnecting zombies,
  3. accelerated botnet takedown.

Anyway, I've rambled on enough for today. I plan to write more about those three topics next month.

What do you think? Do comment below...

Richi Jennings, blogger at large
  Richi Jennings is an independent analyst/consultant, specializing in blogging, email, and security. A cross-functional IT geek since 1985, you can follow him as @richi on Twitter, pretend to be richij's friend on Facebook, or just use good old email:

You can also read Richi's full profile and disclosure of his industry affiliations

Copyright © 2010 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon