Connection scoring beats spam filteringA simple device that prevents spammers from delivering junk to your mail server outperforms complex spam filtering appliances costing up to seven times as much, according to tests by the Windows Secrets Newsletter.
If your company is suffering from onslaughts of spam, our tests indicate that this new approach can halt more than 99% of your unwanted flow without blocking legitimate e-mail. Best of all, the new technology does this without creating a large “quarantine” of suspected spam that you or your employees must manually comb through.
Significantly, the innovative device we tested has never been reviewed by any computer magazine, despite the fact that it’s been on sale for months. The reasons for this are an intriguing part of our story.
The little box that stops spammers
The antispam appliance that inspired our testing is the Deep Six Technologies Spamwall DS200 (photo, left). This little gizmo is only 5″ by 6″ and just 1″ deep (11 x 13 x 2 cm). You configure it to receive your e-mail before the messages hit your mail server. The device uses “connection scoring” to accept transmission attempts from legitimate senders and reject attempts from servers that are sending spam. We found it to be extremely accurate in making the distinction between spam senders and “ham” (legitimate) senders.Since the DS200 is a hardware device that protects an e-mail server, it’s primarily useful to companies that operate their own servers. This includes most large businesses, of course. But also includes many small and medium businesses that have registered their own domain names, such as Example.com.
Home users, who receive their e-mail via an Internet service provider, such as AOL.com, may still see some benefit. The technology within the DS200 could easily improve these ISPs’ own spam rejection rates, helping their customers see less spam.
Testing against thousands of spams per day
To test Deep Six’s real-world performance, we invited major antispam appliance makers to send us whichever of their models they thought was the appropriate scale for small to medium businesses. We received units from all the invitees: Barracuda, Borderware, F-Secure, IronPort, and Network Box. The Deep Six DS200 unit we reviewed was provided by Tyrnstone Systems Inc., a small network consulting company in Seattle, Wash., that sells the device to the SMB market. Deep Six Technologies itself is an intellectual property development company in Tustin, Calif.
Invariably, the appliance vendors (other than Deep Six) sent us devices that combine antispam functions with a firewall, antivirus capabilities, or other features. I was assisted in running technical tests on the devices over a period of six weeks by Brent Scheffler, program director of WindowsSecrets.com. We tested all devices only for their ability to reject spam and accept ham, for the following reasons.
An antispam appliance that also offers antivirus filtering is not in itself adequade to protect against internal virus infections. Viruses can enter a LAN via a roaming USB drive, a laptop brought in from the outside, and many other ways. For this reason, companies need to run antivirus software even if an antivirus appliance is in place. "We’re a perimeter-based device, we’re not providing host-based security," explained Scott Rosen, Network Box’s president for North America, in a telephone interview.
By contrast, spam cannot enter a company except via e-mail. An antispam appliance on the network perimeter, therefore, can offer complete protection against spam. Adware, unauthorized server access, and other threats require their own specialized layers of defense. In our review, for this reason, we tested only the devices’ antispam performance. Firewalls, antivirus protection, and other security functions can and should be configured and tested separately.
Because WindowsSecrets.com doesn’t have a fully equipped test lab, we seldom rate hardware ourselves, leaving this to the publishing giants that can afford it. In this case, however, we do operate in-house a full installation of Exchange Server 2003 supporting five users on the SBS version of Windows Server 2003. We decided to see if we could dedicate this server to serious junk-mail testing.
Before we designed our test suite, we had thought we were targeted by very little spam. Our personal e-mail addresses were presenting us with only one or two spam messages a day. This is because we "spam-proofed" these addresses two years ago. (See our e-book about spam-proofing, above.) Our public, "editor" Windows Secrets e-mail address does receive several virus-infected e-mails a day. This is because we ask our readers to put our address into their "safe senders" lists, where (unfortunately) viruses easily find it. But these e-mails are reliably detected and quarantined by the server-managed antivirus software we run, so we never had to deal with these messages.
When we started building the test suite, however, we found to our surprise that more than 3,000 spam messages were actually being directed to our mail server every day. Most of this spam, we determined, was being sent to old e-mail addresses of mine that I never use any more. These addresses had been posted in plain text at InfoWorld.com, BriansBuzz.com, and other Web sites two or more years ago.
We’d never noticed this flow because our Exchange Server was already dismissing virtually all of it. The server had been correctly configured to accept messages only to the few e-mail addresses we currently use. Any spammers who did somehow get our real addresses were also mostly rejected. The IP addresses of almost all top spammers are published in the so-called SBL and XBL block lists by Spamhaus.org, a respected antispam organization based in the U.K. Our Exchange Server was rejecting any connections from the hardcore spam servers that managed to get listed in SBL or XBL.
Fortunately, we were able to set up realistic tests, despite the fact that our inboxes rarely showed evidence of any junk. Antispam appliances, by definition, must be placed "in front of" a mail server. With no access to our server’s rule base, these devices had to figure out by themselves which incoming connections were from spammers and which were legit.
We took several steps to make the testing fair. We devoted a day to each device to configure it according to its maker’s instructions. We then spent a full day "tuning" each device to reduce false positives (ham rejected as spam). Starting after Christmas, each appliance was then left alone to process a live, incoming mail stream for an entire work day (no weekends or holidays were used for live testing). More than 3,300 messages were processed by each device during its final, 24-hour test period.
Out of those thousands of messages, how well could these products separate out the 5% or so that were legitimate e-mails?
Zero false positives at an affordable price
The following table, sorted by false positives and then false negatives, shows that antispam appliances have become quite accurate. Three of the devices — from Barracuda, IronPort, and Deep Six — achieved a perfect score of 0.00% in rejecting legitimate messages, mistaking none of them for spam.
These three products also showed extremely good performance at filtering out junk. The IronPort let no spam into our inboxes, achieving a perfect false-negative score of 0.00%. The Barracuda accepted only 0.02% and the Deep Six accepted only 0.09%.
We consider the tiny differences between these scores to be statistical noise. All of the three top-rated devices essentially rejected no legitimate e-mail and accepted no significant amount of spam. (Any spam message that made it to our inboxes was considered a false negative. We did not allow grey areas, such as mail that "might be spam" but was placed in our inboxes anyway.)
Shown in Table 1 for comparison is our original configuration of Exchange Server 2003. This was the only strategy we found to be less expensive than the DS200. We configured Exchange to reject all mail sent to nonvalid e-mail addresses and block IP addresses found on the SBL or XBL lists. This scheme is essentially free (not counting our admin time and Exchange itself). But we found it allows significantly more spam to get through — 0.37% — which is more than all but one other contender in our tests.
Table 1: The Deep Six DS200 let through only 0.09% of spam but is low in cost.
The Deep Six device has a list price of only $999 for an unlimited number of e-mail accounts. This is a one-time investment and the device requires no ongoing fees. The IronPort model we tested is much more costly, listing for $2,999 to protect up to 100 e-mail accounts in its first year. The Barracuda lists for $4,899 in the first year for an unlimited number of accounts. All of the antispam appliances, other than the Deep Six, require the payment of ongoing license fees to continue the services after the first 12 months.
The bottom line: We consider the Deep Six technology to provide an antispam defense that’s as good as or better than the competing appliances, while costing only a fraction of the price.
How the Deep Six technology works
The Deep Six device operates completely differently than the other antispam appliances tested. The competing solutions are all modified PCs running Unix or some variant. They occupy either a mini-tower case or a 1U, rack-mounted server case. They include large hard drives to store configuration information, log files, and/or any “quarantined” mail that’s judged to be spam.
Because these devices are designed for use in a glass-house server room, they tend to be noisy. The fans on one unit, the F-Secure, were so loud that we had to raise our voices to converse in the otherwise-quiet office where the system was temporarily located.
The Deep Six DS200, by contrast, is simply a solid-state circuit board with no moving parts. As a result, it’s absolutely silent. This makes it a welcome addition to small offices and home offices, which don’t usually have soundproofed server cages.
More important is the theory that underlies the Deep Six technology. The implications of this concept have permanently changed some of my deeply held beliefs about spam.
Deep Six does not perform “content filtering” to compute a spam score based on the words found in a message’s body or headers. Instead, the DS200 performs "connection rating." It accepts or rejects any distant server’s attempt to make a connection (called a Simple Mail Transport Protocol or SMTP connection) solely according to the characteristics of the sending server.
One way Deep Six does this is by checking the IP address of the distant server to see if it is on one of several dozen “real-time block lists.” The DS200, however, does not disconnect a server merely because its IP address appears on a single list, as many antispam schemes do. Instead, according to a source close to Deep Six Technologies, the device is programmed to use a “network decision tree.”
The inclusion of an IP address on Block List A might not cause Deep Six to drop an SMTP connection attempt. But if the IP address is also on Block Lists C and E, then the sending server is considered to a spam bot. (Our source requested not to be identified by name, saying this technique is the subject of two U.S. patent applications and the details of the technology have not yet been made public.)
The DS200 also resolves "close calls" in an effective way. If a sending server might or might not be a spam server, based on the decision tree, Deep Six asks the sending server to re-try the SMTP connection a few seconds later. Legitimate e-mail servers do this automatically, following well-understood Internet mail standards. Spam servers, however, are programmed not to bother. Sending millions of pieces of spam per day is far more important to spammers than wasting any time responding to SMTP retry requests.
Because these re-tries occur infrequently, and only when a sending server falls into a grey area, I support this type of testing. I generally oppose “Penny Black” schemes, in which all senders, legitimate or otherwise, are required to expend CPU resources to “prove” their worth.
How the DS200 has changed my thinking
The success of the DS200 in our tests has forced me to change my positions on some controversial antispam techniques:
Before: I’ve previously written that antispam block lists should not be used to make a black-and-white, Yes/No decision about e-mail messages. That’s because these lists sometimes add an innocent mail server by mistake.
After: My experience with Deep Six has completely altered my opinion. Using dozens of block lists to create an intelligent decision tree seems to totally eliminate the false-positive problem.
Before: I’ve also written in the past that you shouldn’t delete messages ranked as “probable spam,” in case errors were made by faulty spam filters. Instead, I felt that a quarantine folder should be maintained and examined to retrieve legitimate messages that were falsely shunted aside by filters.
After: With the Deep Six technology, I believe a quarantine folder is no longer necessary. I have no qualms about using this device, given its accuracy, to reject spam connections without accepting and quarantining the spam or ever looking at it.
One of my opinions that’s grown stronger due to my testing is that holding spam and then ranking the content of the messages won’t work forever. I once wrote that the geometric increase in the volume of spam each year would make this storage-and-ranking process too costly for companies in the long run.
In a telephone interview, John Reid, a volunteer with Spamhaus.org, expressed a similiar notion. “Accepting every message that’s sent to you, and then churning through them — it gets very hardware intensive.”
Deep Six eliminates content filtering and quarantine folders altogether. This reduces the load on your mail server substantially. Best of all, there’s no need for you or your co-workers to ever slog through a “Possible Spam” folder looking for misfiled messages. That folder, after all, is certain to consist mostly of phishing attempts, phony pill offers, and worse. That’s exactly the kind of stuff you don’t want anyone in your company to spend time dealing with.
The DS200 was so effective in our tests that I have no concerns about rejecting SMTP connections from servers it deems to be spam bots. Even if some legitimate e-mail user somehow gets associated with a spam server, Deep Six’s effective feedback system minimizes false-positive problems. Allow me to explain.
How Deep Six’s feedback loop works
When Deep Six rejects an SMTP connection, it doesn’t just drop it. Instead, it responds with a standard error code known as a “550.” Companies that use the Deep Six device can include human-readable text in the 550 body. The sending server then displays this text in the e-mail program of whomever sent the message (if a real person was the sender). In our case, the text reads:
- “Our antispam system has rejected the IP address of your mail server. If this is in error, please use the contact page on our Web site to send us your message or call us at +1 206-282-2536.”
Spammers will never see or read this text. Even if they did, they certainly won’t type a spam message by hand into your contact form. But this provides an easy way for any accidentally bounced, legitimate sender to let you know. (Your site must have a contact page for this to work, but that’s a good idea anyway.)
It’s important to note that the DS200 does not send a “bounce” e-mail message to anyone. That would make it as bad as the spammers. Instead, the text of the 550 error is strictly contained within the electronic handshaking that your receiving mail server does with the sending server. No reply e-mails are ever generated.
Other antispam appliances can and do send error codes, of course. We simply feel that the DS200′s emphasis on using handshaking to convey alternate contact methods to hapless senders is particularly effective.
If someone ever does complain to you about a bounced message, the DS200 allows you to put the person’s spammy IP address on a “safe senders” list. Everything from that IP address will then get through. Rather than doing this, however, I believe you should ask the sender to virus-scan his or her server, in case it’s infected by a spam bot.
In reality, it’s very unlikely that an ordinary person sending innocent e-mails through AOL or Yahoo will have the same IP address as a spam bot. Major ISPs transmit their users’ legitimate e-mails from static IP addresses devoted to this purpose. If a spam bot infects a user’s PC, the program doesn’t spew its junk through an ISP’s static addresses. The risk of detection is too high.
Instead, the bot installs its own, tiny SMTP server and spews out junk through whatever dynamic IP address the person has been assigned by his or her ISP. These dynamic IP addresses should never be the origin of legitimate bulk e-mails. That makes them fairly easy for well-managed block lists to detect.
The Achilles heel of spammers is the fact that they must send their massive quantities of e-mails from somewhere. According to Spamhaus’s Reid, the top 200 spammers send out 80% to 90% of all spam worldwide, and the top 10 send out 80% to 90% of that. Whether the machines sending this spam are bot-infected PCs or bought-off Web hosts in the Third World, any IP address that sends millions of spams and little or no legitimate e-mail is going to stand out like a beacon. That’s why Deep Six is able to stop it.
It’s true that no record exists in a quarantine folder of any false positive that the DS200 may mistakenly bounce. But I believe our tests show that the count is effectively zero. Because the device is so effective — and blissfully silent — we put it back into service every time some other device’s testing was completed. That means that, after the DS200′s testing was complete, we ended up using it for more than 30 of the past 60 days. Not a single person has ever contacted us to say his or her e-mail bounced.
Considering how vocal my readers are, it’s inconceivable that no one would have notified me through my contact page about such a problem. I’m buying the reviewed DS200 unit and plan to continue using it to protect my office indefinitely.
Why you haven’t heard about Deep Six
I devoted eight weeks to hands-on testing of antispam appliances partly because Tyrnstone Systems said it couldn’t get major computer magazines to include its device in comparative reviews. In my opinion, the company’s small size is one reason this device has been overlooked. But it’s also because Deep Six’s approach is hard to test.
Spam reviews are usually conducted using a large “corpus” of spam and ham messages. One server sends the messages to another server, which is protected by a particular filtering product. The number of hits and misses are then calculated.
This method won’t work on the DS200. The device isn’t scoring the content of the messages, but the reputation of the sending server. Since the originating server in artificial testing is the same for every message, all the e-mails pass or they all fail.
The Deep Six technology can only be tested when placed in front of a live mail server, using a live stream of e-mails, and scoring live SMTP connections. This is the reason our tests took several weeks. No two devices could be tested on our mail server at the same time. They had to be scheduled one after the other.
I urge major computer magazines to devote the resources needed to test Deep Six against competing spam solutions. The DS200 technology may provide valuable insights into the spam menace and how it can be permanently stopped using technical methods.
To purchase a DS200 and test it on your own company’s mail stream, visit Tyrnstone Systems. For more information on the technology itself, visit Deep Six Technologies.
Both are tiny companies, so if their Web sites become slow or unresponsive from thousands of Windows Secrets readers visiting them, try again the following day.
The Deep Six site claims that the DS200 device is capable of handling peaks of “10 connections per second.” David Gerhart, CEO of Tyrnstone, says it’s his experience that the unit can reliably handle as many as 50 SMTP attempts per second. For larger volumes of mail, multiple DS200s can be employed. Each unit is given its own static IP address to balance the inbound load. Deep Six’s connection-scoring function can even be performed offsite as a hosted service. This allows even fairly large companies to try the technology for themselves.
If you do add one or more DS200s to your network, be sure to correctly set up your "secondary MX records." I described the procedures for this in my Executive Tech columns of Jan. 3 and Jan. 24.
I’ll be looking forward to any independent test results that come out. If you do any testing, or you’d like to send us a tip on any other subject, visit WindowsSecrets.com/contact. You’ll receive a gift certificate for a book, CD, or DVD of your choice if you send us a comment that we print. Thanks for your help.
Brian Livingston is editor of the Windows Secrets Newsletter and the coauthor of Windows 2000 Secrets, Windows Me Secrets, and eight other books.
