DNS blocklists and reputation services (part 1: background)

In the next three Security Levity posts, I want to talk about 'sender reputation' and how it's used to filter spam and other undesirable email. What is reputation, what's a reputation service, and how is it different from good old-fashioned blocklists? These three posts should tide us over the holiday period. This week, we'll cover a little history and background. That will ease us into a description of reputation services, which I'll talk more about next week. Later, in the third post, I'll look ahead to the future of sender reputation.  

The old guard: IP blocklists

From the early days of spam filtering, email administrators have maintained lists of email senders that were known to send spam. Specifically, these lists contain known spammers' IP addresses and IP address ranges. When a spammer attempts to make a connection, the receiving mail server simply looks up the source IP address in the list and drops the connection if it matches. Pretty soon, these same administrators hit false positive problems, so they also built lists of exceptions to their blocklists, or allowlists. (Note that these lists were often known as blacklists and whitelists, but many people are uncomfortable with the racial implications of these words. These days, they're usually referred to as blocklists and allowlists.)  

DNS Blocklists (and allowlists)

These administrators would even trade their discoveries with other, like-minded email admins. Blocklists became community spam-fighting resources.

Pretty soon, someone had the bright idea to set up a database that people could query in real time. After a false start trying to replicate the list to all its users, Paul Vixie was arguably the first to use the DNS protocol to access a blocklist. In its simplest form, the receiving mail server does a DNS lookup of the sending IP address and is informed whether that IP address is known to have sent spam recently.

Vixie started this work way-way back in the dark days of 1997, initially for AboveNet, and later for the now-defunct MAPS non-profit, spam fighting organization. He called the blocklist RBL, short for Real-time Blackhole List. (People often casually use the term RBL to refer to any DNS blocklist, but it's actually a registered trademark. The 'correct' terminology is DNSBL.)

After AboveNet, other ISPs quickly followed suit. Other than filtering spam, an emerging objective was to 'shame' other ISPs into cleaning up their acts. An ISP that was seen to harbor spammers could find that its legitimate customers' email was being filtered along with the spam.

In the same way as was done with the original, local lists, some blocklist operators also ran DNS allowlists. A DNS query for an IP address could return a code that indicated the address was trusted not to send spam.

List segmentation

The next logical step was to segment the lists, to reflect the type of email sender the IP address belongs to. This was either done by creating several new lists, or by making a master list return a range of response codes.

Instead of simply replying whether or not the IP address was known to send spam, these master list DNSBLs started to offer more nuanced responses. Examples of responses that such a list might return include:

  • Alleged 'mainsleaze': a well-known, respected organization, but which receives more than the usual 'background radiation' of spam complaints
  • Reformed spammer: on probation, was known to be sending spam, but complaints have fallen to an acceptable level
  • Consumer IP: not known to be sending spam, but IP address is in a consumer ISP's space, so has no business sending email
  • Legit: known good email sender, from whom the vast majority of recipients will want email
  • Legit (bank): this sender is a bank, so beware of mis-classifying its email as a phishing attack
  • Legit (unconfirmed opt-in): this is a legitimate sender, but does not practice confirmed opt-in

This idea allowed recipients to implement more finely-grained policies about what types of senders they wish to receive email from.

Next week, I'll cover common criticisms of DNSBLs, and how the industry morphed DNSBLs into reputation services. I'll talk about how reputation services differ from traditional DNSBLs, in terms of professionalism, philosophy, and technology. Read part 2 here.

I want to make this an interactive place: where I can answer questions and cover topics that you suggest. Feel free to add comments and ask Amir!

  When he's not measuring senders' reputations, Amir Lev is the CTO, President, and co-founder of Commtouch (NASDAQ:CTCH), an e-mail and Web defense technology provider. MORE...

Computerworld's IT Salary Survey 2017 results
Shop Tech Products at Amazon