DeepNet DNSBL Tools

This is a set of scripts for maintaining your own DNS-based IP and URI blacklists for use either in a cumulative score-based filter system like ​SpamAssassin or (if you trust anyone who has access to the system) hard-blocking in your MTA.

The API and UI for both the IP list and the URI list should be considered functional but incomplete. Some changes must be done directly in the database, or a custom script written.

These are NOT designed for a major publicly-available list similar to Spamhaus' ​zen, ​SpamCop, or Barracuda's ​BRBL. If nothing else, the extra bitmask handling slows things down enough that it would take an unreasonable amount of time to maintain a multi-million-record dataset. As of writing, a local dataset containing ~19000 IPs and ~12000 netblocks, resulting in ~20K internally-published entries, takes about 20 seconds to export to an rbldnsd-formatted file on otherwise lightly-loaded modern hardware. Past versions of the code on the same dataset may run as long as 10 minutes.

IP blacklist

For IP listing, the RIR allocation (from ARIN, APNIC, LACNIC, AfriNIC, or RIPE) and two levels of internal delegation are supported. This allows tracking of how many IPs in a given block have been reported, and automatically listing the entire block if that goes beyond a certain threshold. Additional levels of delegation could be supported with minimal changes.

The thresholds are defined in a database table. No UI is currently built to modify these. A fairly reasonable set of defaults (in production use for over a year) is provided in the initial SQL tabledef.

Multiple lists are supported via autodetection of the web URL that will load different database connection information. This allows you to maintain one list for general scoring, and a second with different autolisting thresholds for hard blocking.

Data can be exported for various DNS systems. ​rbldnsd is recommended as it supports CIDR netblocks rather than forcing divisions along classful octet boundaries (leading to having to add 128 lines for a /17, for instance).

Manually tagging a block or netblock owner/operator for listing is possible but no UI has been implemented.

An IP can have a bit in a bitmask set for any one of the following:

• IP count exceeds automatic threshold (threshold is set for each CIDR block size)
  • IP count in RIR allocation
  • IP count in first delegation
  • IP count in second delegation
• Netblock has been manually tagged to be listed (this is useful to tag blocks marked in WHOIS as dynamic IP space - these IPs should not be sending mail directly to your MX)
  • RIR allocation tagged
  • First delegation tagged
  • Second delegation tagged
• Netblock owner/operator tagged (useful to catch new IPs assigned to netblock owners who generate lots of spam)
  • RIR allocation tagged
  • First delegation tagged
  • Second delegation tagged

​Example !SpamAssassin configuration fragment

URI blacklist

URI listing is much simpler; one or more domains can be added at a time, each at a different listing level or type. Currently supported designations are "black", "grey", and "URL shortener". The supplied SpamAssassin configuration fragment uses these designations.

​Example !SpamAssassin configuration fragment

Releases

No stable release has been designated.

You can check out the current code from SVN at ​https://secure.deepnet.cx/svn/dnsbl/trunk, or download a development snapshot current as of writing from ​https://secure.deepnet.cx/releases/dnsbl/.

The IP blacklist has a ​demo. You can see the entire dataset ​here. This results in an rbldnsd data file like ​this (may be out of date with respect to the listing on browse.cgi).

Configuration

The web UI takes the hostname and root path to the installation, converts non-alphanumerics to underscores, and appends .conf - for instance, the demo site above looks for secure_deepnet_cx_demos_dnsbl.conf in /etc/dnsbl. This file contains the database connection information. The URI blacklist looks for <something>uribl.conf in /etc/uridb along the same pattern.

For export, the export script requires an argument to determine which configuration to pick up. I use symlinks with convenient short names linked to the longer filenames (dnsbl -> secure_deepnet_cx_demos_dnsbl.conf).

Thresholds for automatically listing a block are defined in the autolist table. The CIDR mask length is used on the masklen column to look up the IP count threshold from the ipcount column.

Manual tagging of a block or block owner is done by setting the listme column in the blocks or orgs tables respectively to y. The comments column gives you a place to put notes about the block - I've used this to note an "advisory" block added when I see a grouping of IPs that keep showing up in missed spam but which doesn't show in WHOIS, and to tie together apparently unrelated block owners. It's also useful to remind you why a block might be listed in spite of being part of a squeaky-clean network otherwise (typically for blocks labelled "dynamic" in WHOIS, but not listed on eg Spamhaus' ​PBL).

All further configuration relies on the key-value "misc" table in the database. Supported keys are:

• blzone (default "company.dnsbl")
  DNS zone to publish the data under. A non-public TLD like .dnsbl is recommended to insure you don't leak the data beyond your own usage.
• bladmin (default "systems.company.com")
  Admin contact for the zone's SOA record
• iplisted (default "$ relayed a reported spam")
  String to use when a TXT lookup is done on an IP that is explicitly listed. $ is replaced by rbldnsd with the actual IP on the fly.
• blocklisted (default "Netblock listed on one or more criteria")
  String to use when a TXT lookup is done on an IP in a block that's listed, but the IP itself isn't.

The latter two support some customization on a per-entry basis by replacing the literal string :ENTITY: with the IP or block being written.