A Definite Spam Message

Understanding Spam Filtering in SmarterMail

The Problem

As a system administrator you know that spam is a real nightmare. What you might not know is that statistics show that well over half of the total email traffic, worldwide, is spam. When you factor in things like new system administrators who aren’t entirely sure how to run a mail server, ethically challenged users who exploit those system administrators, the fact that many mail servers on the market are running on old code or obsolete technologies and more, in some cases the percentage of spam coming out of a mail server can be over 70%. That’s a ton of unwanted email that a mail server must deal with. Spam is big business, both for those who are sending it as well as companies that attempt to block it.

When SmarterMail was first developed, there were simple methods that could could be implemented to help protect a mail server. However, as the spam industry continues on the path to becoming a multi-billion dollar industry, effectively blocking large percentages of spam have moved beyond what a mail server can do without the use of third-party tools.

The Goal

Our goal for the latest build of SmarterMail is to provide 70% spam protection, out of the box, without the use of any third-party solutions. In addition, we want to simplify spam protection for users and domain administrators along with increasing the efficiency of the default checks.

What We Changed

New Spam Header

We took great pains to review all of the RBLs and URIBLs that are used as the default checks within SmarterMail. We looked at scores, at the efficiency of the lists, re-reviewed what the lists did, looked at spam and non-spam messages to see which lists were returning scores, which were excessively aggressive and much more. Using that info, we then looked at the default Filtering options for Low, Medium and High spam: what the Weights were and what the Actions were on messages with those weights.

As a result, we came away with several changes. We changed both the weights for virtually evern spam check, RBL and URIBL as well as the Actions taken across the various spam levels. We removed lists that were ineffective and added in others that we found to be very effective. Using these, we established a concise set of Actions that efficiently handled spam WITHOUT having to use Cyren or Message Sniffer. Then, adding in either of those products, our efficiency increased.

We also changed the information contained in the header of a message: now, you’ll see the spam checks that were run and the Weight that was assigned to that check. This makes it very easy to see how a message was flagged, the Weight FOR that flag and then how the overall score adds up. Here’s an example of the new header information:

X-SmarterMail-Spam: SPF [Pass]: 0, HostKarma - Blacklist: 10, Surriel: 10, SORBS - Recent: 5, DNS Real-time Blackhole List: 5, Message Sniffer [code:52]: 30, DKIM [Pass]: 0, SURBL [count:2]: 10 X-SmarterMail-TotalSpamWeight: 70

We changed the default Actions for the total spam Weights for Low, Medium and High. Now, Low weights do nothing, but Medium and High weights will move to the Junk E-Mail folder. We didn’t want to muddy up messages with a Low probability of being spam with adding something to the message subject as that can just confuse users. However, with the weight changes, messages that are flagged as Medium or High are generally spam, so there’s no need for those to go to someone’s Inbox.

In addition, as a way to achieve the efficiency and ease of use of these changes, system settings are now propagated to Domain Administrators and users. When they’re allowed to by the system and/or domain administrator, domain administrators and users can easily change the behavior of Low, Medium and High messages as needed. However, as with most major email services, the weights for individual checks are solely managed by System Administrators.

What System Administrators Can Change

Now that SmarterMail gives system administrators a solid base to work from, what can they do to help the never-ending fight against spam? Well, here are some suggestions:

Understand the Changes for Users

Looking at how some of the larger companies handle spam protection, they just give you two options: a message is spam or it’s not. We’re providing system administrators more functionality than that, but eliminating some of the complexity of that functionality for users. For example, showing spam check Weights at the domain level caused confusion for some domain administrators because it showed every single spam option and its weight, but they had no idea what that meant. So Weights are now visible only to system administrators, but domain administrators and users can still change the actions taken across the various spam levels.

In addition, as domain administrators will no longer see any spam Weights, any custom Weights they’ve set for their domain will revert to the default Weights set up at the system level.

Reset Your Spam Settings

As the changes we made are the new defaults, you will need to use the Reset Antispam Settings to benefit from these changes. This will put you at the best level of spam protection, whether you’re using only the internal spam checks or using Cyren and Message Sniffer.

Propagate New Low/Medium/High to Domains

Once you’ve reset your spam settings be sure that the new Low, Medium and High filter Actions propagate to your domains. You will also want to make sure that the domain administrators on your server are aware of the changes, why they happened and the results you’re seeing.

Periodically Review Settings

Fighting spam is an ongoing battle. As such, it’s a good idea to review how things are progressing on a quarterly basis, if not more often. That means:

  • Review Scores: Check messages that are currently hitting Low, Medium and High scores. With the new headers, it will be easy to see where the effective scores are coming from. It can also show where some scores may be too high, or too low, and allow you to make adjustments as necessary.
  • Adjust Weights: Once the scores for incoming messages are reviewed, make sure the weights set up for the various checks are where they should be. Adjust higher or lower based on how often a particular check is being hit.
  • Adjust Actions: Make sure that any actions taken on a message are based on users and how they want spam to be handled.

One Size Does Not Fit All

Cyren Spam Messages Hourly

Taking all of these changes into account, fighting spam is really something that a system administrators need to vigilant about. As mentioned, there is no silver bullet. All of the checks available should be used and third-party products like Cyren, MessageSniffer or even additional checks should be added into the mix. During our testing, we noticed a big difference in the types of spam that were sent, based on the time of day: in the evenings and overnight, the default RBLs and URIBLs were very good at marking spam, whereas during the business day, MessageSniffer and Cyren were good at marking spam while the RBLs and URIBLs seemed to catch less.

And don’t forget: the defaults we suggest are just a starting point. Sure, they’re an efficient starting point, but the great thing about SmarterMail is you can add additional services, your own RBLs or URIBLs, you can set your own Weights and your own Actions based on those Weights. In addition, once you have an efficient base on one server, the spam settings can be exported FROM that server and applied to any other SmarterMail server in your environment.

In Conclusion

At SmarterTools, we’ll continue to provide our customers with as many tools as possible to help combat spam. However, the spam industry is just that: an industry unto itself. There are hundreds of millions of dollars to be earned, and spammers do everything they can to protect that revenue potential.

In addition, fighting spam is its own, separate industry, and many companies have grown very large, very quickly, building products and services to help combat the equally-growing spam industry.

From our perspective, SmarterTools strives to build the best mail server on the market, and that’s our specialization. Just as we have our niche, sometimes it’s best to give way to others who are attempting to build the better antispam service or the best antispam product. We do what we do, and they do what they do. The nice thing is, we can build SmarterMail in such a way that the integration with other products and services is included, or available, for users. It’s up to you all, as those system administrators, to decide what combination works best.