Last time on Security Levity, I talked about blocklists, particularly DNS blocklists (DNSBLs). This time, I'll talk about how the industry moved to reputation services and how they differ from traditional DNSBLs.
(If you missed part 1, here it is.)
Criticisms of DNSBLs
The main criticism leveled at DNSBLs were that they were often run in a haphazard, inconsistent manner. They were often pilloried for not responding quickly when mistakes were pointed out to them -- usually this boiled down to not being able to cope with the volume of changes or the level of accuracy required. Basically, because they weren't run as a business, they had insufficient resources to maintain the lists.
Another common criticism was that the maintainers of some DNSBLs were 'mere' volunteers or netizens who sometimes turned aggressive and acted antagonistically to ISPs and email senders.
This became a general 'amateur-hour' stereotype, which was one of the catalysts for the industry to get serious and build professionally-run reputation services that encompassed DNSBL functionality and more.
What are reputation services?
The simplest reputation services are simply DNSBLs rebranded and run as a business, with consistent operational procedures. But the better services are so much more than that.
The main difference is that a true reputation service will be more nuanced in its opinion of the 'goodness' of a sender. The service can have a scoring system that illustrates the likelihood that email from this sender is spam. That allows the service to express opinions that are 'shades of gray' and allows the receiving side to decide how aggressive it wants to be in rejecting spam.
For example, if a sender's reputation is said to be 70%, what should a receiving spam filter do with its email? One idea is to ensure the email passes more onerous, rigorous anti-spam checks than email from 'known good' senders. Examples of such checks include greylisting, extreme greetpause delays, and tarpitting. (I have half a mind to write a followup blog post explaining anti-spam techniques like this; let me know if you'd like that.)
Sadly, most of the so-called reputation services out there are little more than glorified DNSBLs.
Why is this important?
Reputation services are the first link in the chain for most anti-spam systems. As their accuracy improves, they're able to reject upwards of 75% of all spam without needing to examine the message in any way. This gives a terrific performance benefit, allowing a spam filter to better cope with ever-increasing email volumes.
However, as reputation becomes a bigger and bigger part of the spam filtering technology mix, the need for it to be accurate becomes greater. The ever-present specter of 'false positives' lurks in the shadows, waiting to pounce on any reputation service that gets too aggressive in its opinions. That's why it's important for the service to be professionally run, and adhere to clear, consistent rules for listing.
Also, services must have a way for senders to request delisting, if the service makes a mistake. The process for getting delisted should, again, be clear and consistently implemented by the service.
Moving away from DNS
DNS is an excellent protocol for high-performance, distributed, key/value pair database access such as this. However, it has some limitations.
Reputation services are often integrated into commercial anti-spam products, so they'll often use the same update mechanism as used by the rest of the anti-spam engine, rather than using DNS. Using a purpose-built protocol also makes it easier for spam filters to send feedback that's automatically incorporated into the reputation service's database.
Also, DNS lookups impose limitations on how innovative a reputation service can be. A fundamental tenet of DNS is that replies to queries are global: that is, that the answer you get is the same, no matter who's asking or where you are. (Coincidentally, Paul Vixie, the inventor of the first DNSBL wrote about this very issue recently.) This is a problem with more sophisticated reputation services, which may give different replies, depending on your location or preferences. I'll talk more about this 'regional reputation' idea in Part 3.
Next week: the future!
That's all for this week. On January 6 -- in the final part of this series -- I'll look ahead to the future of sender reputation, including domain reputation and vouching. Read part 3 here.
I want to make this an interactive place: where I can answer questions and cover topics that you suggest. Feel free to add comments and ask Amir!