Matt Kesner has been in IT long enough to take silence as a compliment. "People don't often come down to IT to say, 'Nice job,'" says the chief technology officer of Fenwick & West LLP, a national law firm. "The best you get is that they don't come down at all when things are running well."
But then Kesner tackled the firm's spam problem, and suddenly he found himself a hero. After he outsourced the problem to a managed service provider, the law firm's partners (whose time is worth US$350 to $600 an hour) were no longer spending more than an hour a day wading through 300 to 500 spam messages to get as many legitimate messages. "We got quite a few pats on the back and attaboys after putting the spam filter in place. Users saw the difference instantly and are dealing with hundreds fewer messages a day. They actually got excited about it."
Unlike the invisible foe of Y2K, the scourge of spam -- which plagues receptionists and CEOs alike -- is painfully evident to everyone. Now that spam accounts for 40% to 60% of most organizations' e-mail traffic, you scarcely need to mention that Ferris Research says spam will cost U.S. businesses at least $10 billion this year, or that Nucleus Research Inc. estimates that companies forfeit $874 per employee annually in lost productivity alone. Nor do you have to bring up the fact that spam clogs e-mail systems and siphons IT resources away from legitimate business projects. Spam is a royal pain in the server, and we all know it.
As Kesner has discovered, the sheer ubiquitousness of spam affords chief information officers (CIOs) a rare opportunity to look good. Although receiving some spam is inevitable (and employees' expectations should be set accordingly), there's plenty you can do to make things better. In fact, there's plenty you should do, since the problem is only going to get worse, and you can't count on antispam legislation to save the day. (Criminalizing spam would simply drive more spammers to send their messages through offshore Internet service providers.) Solve the spam problem -- or even just put a big dent in it -- and you too can be a hero. Here's a look at how otherwise mild-mannered CIOs are leaping into the spam fray to help keep e-mail viable for users.
The spam balancing act
What makes it so hard to write antispam laws or antispam software is that there's no such thing as a universal litmus test for spam. "One person's spam is another person's newsletter," says Eric Ogren, a senior analyst at The Yankee Group. "There's no magic widget the CIO can put in front of the e-mail server and spam goes away."
End users have to be involved in deciding what is spam, he explains, because what's unwanted can vary widely not just from one company to the next, but from one person to the next. What looks like spam to the rest of the world could be essential business communication for certain employees. Colorful language might be important to a customer service agent (displeased customers often lose their tempers, after all), anatomical references may be work-related for a doctor in a research hospital and Viagra messages could very well be germane to someone in the pharmaceutical industry.
Case in point: When John Zarb, CIO of Libbey Inc., a manufacturer of glassware, china and flatware, tested the Guenivere (a virus and subject-line filter) and SpamAssassin (an open-source spam filter), he had to shut them off after 10 days because they were rejecting important legitimate e-mails. The filters bounce mail with a spam score of 7.5, yet they were automatically assigning 7 points to e-mails from an Asian country in which Libbey has business relationships. Another rule assigned what Zarb calls "bad points" for using all capital letters. Since using all caps is common practice in that Asian country, messages from those business partners easily racked up more than 7.5 points and therefore got zapped. "If the message is a transport document, ouch," says Zarb. His group tweaked the default settings so that Asian e-mails wouldn't automatically accrue so many points. Today, the filters block about 70 of Libbey's spam, and Zarb says the false positive rate is far lower but not zero. Because some messages are too critical to miss, he decided to exempt a few employees who deal with international issues from the SpamAssassin filter.
As Zarb quickly discovered, once you start filtering mail, you run the risk of blocking legitimate e-mails because they look like spam. Avoiding an unacceptable level of "false positives" requires a delicate balancing act. Although most vendors will claim they capture at least 90% of spam, going above 90% will probably result in too many false positives, says Matt Cain, a senior vice president at Meta Group Inc. "You could crank it up and catch 98% of spam. But you'd get an unhealthy amount of false positives," he says. "And if you go down to 85%, you'll have very few false positives, but too much spam will be getting through."
At printing ink manufacturer Flint Ink Corp., Vice President and CIO Don Barnowski has been trying out Symantec Corp.'s Norton antispam product. After initially filtering on 300 to 400 keywords, false positives were a daily occurrence. "We started to get calls from people not getting e-mail they were expecting," he says. "That was a red flag; you don't want people questioning the integrity of e-mail delivery."
Cutting the keyword list in half cut the false positive rate in half, but it also let more spam through. "I've accepted the fact that we can't prevent all spam from reaching employees," he says. "Finding out five times a day that I can improve my mortgage rate is irritating but not offensive. There's a big difference there. It's more important to reduce the number of false positives than it is to smother all spam. You can't have it both ways, unfortunately."
Walter Smith, director of the global IT infrastructure services group at Advanced Micro Devices Inc. (AMD), decided that outsourcing spam control to a vendor that has multiple solutions would be the best approach for his company.
To combat false positives, make sure you choose a spam solution that gives you a quarantine area for probable spam that users can access to check for legitimate messages. Users can be alerted in the form of an e-mail digest of all blocked spam subject lines or be directed to a Web mailbox. Outsourcers generally maintain quarantine areas on their servers so that companies don't have to tie up their own networks with suspected spam. Giving end users the ability to add addresses to trusted sender lists (often called whitelists) also ensures that legitimate senders won't get blocked.
"We took the approach of putting in very coarse controls at first, then tightening them up, rather than going with the 'big bang' theory and begging forgiveness for weeks," says Gene Fredriksen, vice president of information security at financial services company Raymond James Financial Inc. "It's absolutely a strategy I'd recommend. You have to build trust in your system first." Fredriksen uses Syntegra's managed service to filter spam for the company's 14,000 mailboxes.
It's also smart to test before you buy, particularly if blocking any legitimate e-mails would harm your business. At Fenwick & West, Kesner created shadow e-mail boxes for some of the firm's biggest e-mail users, into which he put duplicates of all of their messages. He then used those shadow boxes to test antispam products. Because some of the language used in the firm's large commercial transactions -- buy, sell, price, dollars -- tends to show up in spam, he was dismayed to discover false positive rates of 1-to-1,000 and even as high as 1-to-100.
"In our business, every e-mail from clients is really crucial. We can't block a high percentage of legitimate e-mail," Kesner says. "We needed to be below 0.05 percent, which seemed near unattainable with a filter."
After trying out more than 18 antispam products, Kesner decided to go with Postini Inc.'s antispam service. With Postini, his false positive rate approaches 1-to-10,000, in part because users can put trusted senders on a whitelist, meaning messages from those senders automatically bypass the filters and get delivered.
Kesner's cautious approach of testing on duplicate messages allowed him to get a real-world read on false positive ratings without worrying about losing any legitimate messages.
The outsourcing option
Kesner's testing convinced him that the ability to filter out most spam while maintaining an extremely low false positive rate was worth the risk of outsourcing. "I was cautious of an outside service," he says. "But (being an outsourcer) allows them to respond to spam outbreaks faster than their competitors." Sending out a spam update to thousands or millions of remote users is taxing, so spam software makers tend to roll these updates into packages and send them periodically. A service provider can simply add an update to a few servers in a couple minutes and have the update apply to all customers nearly instantly.
Postini also lets Fenwick & West IT employees choose how much of each kind of spam they want to filter out by setting filters for each of four subcategories of spam: explicit content, get rich quick, too good to be true and racially insensitive. Kesner pays a per-user fee, which turned out to be about half of what he'd budgeted for. And because he's now blocking at least 99% of incoming spam (5,000 to 7,000 messages a day get trapped on Postini's servers), Kesner has been able to delay the purchase of four new servers (costing $10,000 to $20,000 each) by more than six months.
Indeed, using an outsourcer can be cheaper than managing the spam problem internally. Water Pik Technologies Inc., which manufactures personal health-care products, pool products and heating systems, also found that to be the case. "We looked at the cost of doing it internally, and it was staggering," says CIO Wallace Miceli. "We're talking one or two people full-time," he says. Miceli pays FrontBridge Technologies Inc. $1.50 per month for each of his 1,000 users, which he says is cheaper than buying and maintaining an onsite filter.
Outsourcing, however, won't work for everyone. Large companies, those with multiple locations whose mail doesn't all pass through one or two points, and those that use both private and public networks, may find it tricky to outsource. And the obvious downside of outsourcing is that it requires giving someone else the authority to decide what e-mail enters your organization. "For a spam filter to work very effectively, it has to look to a certain extent at the body of the message," says John Mozena, a cofounder of the Coalition Against Unsolicited Commercial Email. "Something -- even if it's just a piece of software -- is reading your company's mail. For some companies, that is not acceptable." Law firms and hospitals, for example, might be wary of exposing confidential client or patient e-mail to a third party.
If you choose to outsource, make sure your service provider will give you timely access to quarantined messages. When Rush Enterprises Inc., a truck, construction and farm equipment dealer, tried outsourcing, Rush's e-mail administrator couldn't see what was being filtered and therefore couldn't tell if the company was missing good e-mails. "When you outsource, you generally lose control," says CIO Scott Kressner. If there was a problem, or if a user needed to be able to receive an important message, it took hours or even a day or two to resolve the situation. Kressner ended up purchasing the antispam appliance (a server loaded with the outsourcer's software that sits in front of the real mail server) and now uses it in conjunction with Symantec Gateway. Although the appliance was more than two or three times the annual cost of the service, Kressner says it's been well worth it to regain control.
A spam cocktail
A year or two ago, subscribing to a list of known spammers (known as a black-hole list or a blacklist), or relying on a signature approach (comparing the patterns in a new message against the fingerprints of known spam messages), or using reverse domain name system lookup to check whether the sending domain was legitimate might have worked. But companies can't rely on just one type of blocking anymore.
"I'd strongly argue that you need a spam cocktail -- a variety of approaches that work together to generate a probability as to whether a message is spam or not," says Meta's Cain. The most reliable products and services subject each e-mail to numerous tests that yield a probability score indicating how likely the message is spam. Companies can then set up rules that, for example, delete messages with a spam score of 95 percent or more, quarantine messages in the 85 percent to 95 percent range, and deliver (with a "suspected spam" warning) messages with scores between 75 percent and 85 percent.