E-mail is one of the most important means of communication we have today and its use in organisations around the world is growing on a daily basis. Unfortunately, as e-mail use for business and personal communications increases, unscrupulous companies are also using this medium to send unwanted, unwelcome spam.
Spam is a complex problem. All too often, companies spend money on anti-spam systems and end up disappointed by the lack of results these applications deliver. It seems that whatever legitimate businesses do to protect themselves, spammers find some way to circumvent it and still get their spam into people's inboxes.
In their efforts to prevent this epidemic of junk e-mail, anti-spam application providers continually add new intelligence to their systems in order to more aggressively respond to spammers' shenanigans. Unfortunately, the more complex the spam filters become, the more they tend to misclassify and mislay more legitimate e-mail that is incorrectly assumed to be spam. This is naturally not acceptable.
In addition to losing business e-mails, certain legitimate companies can be prevented from communicating with customers because certain words in their branding message are commonly used by spammers - such as sexual health medication, the name of which cannot be included in this article because spam engines will classify it as junk mail if it is e-mailed. Not only is spam dragging the potential of the Internet down, it is also preventing it from realising its full business capabilities.
One important business e-mail lost is far worse than 1000 spam messages received. This is why a significant number of spam filtering applications are turned off within three months of installation. Companies can no longer sustain losses of business critical information and would prefer to deal with the irritation of spam if it means that important messages are not blocked.
Quarantine also misses the mark
So, while solutions that stop spam at the gateway - where they first enter the company's IT systems - is a worthwhile starting point, there are still problems gateway solutions need to overcome. To combat this problem of inaccuracy, some anti-spam vendors have created sophisticated quarantine systems to keep messages they believe are spam for user review (in case their filters get it wrong). This does not solve the problem, unfortunately, it simply moves it to a new place.
Some of the problems associated with quarantine solutions include:
* Nobody looks at quarantine lists to perform regular quality assurance as it is too time consuming.
* If quarantined messages are examined, it results in breaches of privacy or confidentiality.
* Quarantines become so large and cumbersome that finding any legitimate messages can be almost impossible.
* Quarantines require significant system resources to manage and store numerous mostly unwanted messages.
* The sender of a legitimate e-mail, which is deemed to be spam, is seldom made aware that the message has been quarantined. To make the sender aware the company has to reply to the sender address (bogus or otherwise) on every spam message.
In short, spam filters and quarantines represent classic bad practice in a spam management strategy. Mimecast has addressed this issue and developed a workable solution that ensures legitimate business messages are accepted and that unwanted messages are rejected.
Through its automated reputation management system, Mimecast does not quarantine messages, it either accepts a message for delivery to a user's mailbox; or it rejects it outright before the protocol discussion with the delivering mail server is complete. This leaves the delivering mail server; zombie machine or spam-ware with responsibility for its undeliverable messages.
The system is based on reputations. Outside senders with good reputations are allowed e-mail access to the recipients they wish to communicate with. Users without reputations are not automatically classified as spammers, but can use simple, specific methods to gain good reputations. These methods are inaccessible to spammers who rely on volume efficiency, anonymity and deception to deliver their messages.
Acquiring a reputation
Reputations are automatically created for known good communicating pairs (an auto-whitelist). For example: if firstname.lastname@example.org sends an e-mail to email@example.com, Jim acquires a reputation allowing him to communicate with John in future. During an average year over 95% of legitimate e-mails are likely to arrive with auto-whitelisted sender reputations and 5% of legitimate e-mails will establish their reputations by passing the tests described below.
From a spam-management perspective the relationship between the sender and recipient is more important than the message contents. By identifying reputable senders upfront and allowing them priority access, Mimecast saves a significant amount of processing, avoids potential inaccuracy and improves e-mail system reliability. Auto-whitelisted sender reputations are totally secure as the permutations of an auto-whitelist are too complex for a spammer to anticipate. It can not be breached.
Testing senders where no reputation yet exists
To do this it is useful to understand two fundamental differences between the behaviour of legitimate mail delivery and that of a spammer:
Spammers frequently use zombie machines (PCs they have hijacked through hacking or viruses that are connected to the Internet and can act as spam delivery agents).
Spammers and zombie machines do not hold down permanent IP addresses. They either operate on random dial-up addresses; or because the same IP address used for too long will rapidly appear on a realtime blocking list (RBL) it will become unusable for e-mail delivery on the Internet.
Mimecast tests senders without a reputation for these two behaviours. In order to deliver an e-mail through Mimecast the sender must comply with the following:
The e-mail server software must be fully RFC compliant. Most spam software is not; as to deploy it to hijacked zombie machines the software developers have taken several software shortcuts. Mimecast will reject the connection if the server and the message is not adequately RFC compliant.
The next test assesses whether the delivering mail server is capable of queuing and retrying. Most spam software runs in 'fire-and-forget' mode, if a recipient's mail server is unavailable, it simply skips to the next recipient and does not return. To all senders without a known good reputation, Mimecast initially appears unavailable. This technique alone will leave droves of spam discarded at its source.
If the spammer can retry an additional layer of reputation testing is applied. Mimecast tests if the delivering server can retry within a defined window using the same IP address (or C-class range). More spammers get stuck at this point since they usually keep changing IP address either to avoid detection and RBLs; or as a result of using thousands of different zombie machines. The longer the server has been prepared to retry the message using the same IP address range the more likely it is to be a legitimate e-mail (this process is also known as grey-listing).
Mimecast and Sender-ID
Sender-ID is a new approach which tests and validates whether the delivering mail server's IP address has been authorised by the domain owner to send e-mail for the domain it claims to be. Sender-ID is gaining widespread support and it is advisable for companies to register Sender-ID records to prevent their domains being misused by others. It is also advisable for their receiving MTAs to check Sender-ID records to ensure that they are not encouraging spoofing.
Mimecast performs these checks for its users. It checks to see if Sender-ID records exist for every domain that attempts to deliver to it. If the delivering server contravenes the records, the connections with that server will be terminated. Over time as more companies define Sender-ID records and more mail servers check these records before accepting e-mails for the domain, e-mail will become more trusted and spammers lives more difficult.
Mimecast has been involved in the evolution of sender policy framework (SPF) and subsequently Sender-ID since near inception. Mimecast authored and contributed the open source Java libraries for MTA to implementations of SPF/Sender-ID. The company was one of the first MTAs to check incoming e-mail for compliance with the stated domain's Sender-ID records on DNS.
Spam is the curse of the Internet age. Unfortunately, legislation is unable to stop it as it is relying on the ethical behaviour of spammers. New products, such as those from Mimecast have been designed to relieve corporations of the burden of spam without risking the deletion of critical business and personal e-mail communications. Companies have no choice, they either deal with the never-ending cascade of junk mail or they kill spam before it consumes bandwidth and employees' time.
Best practices for spam management
Auto-whitelist to manage reputations of legitimate sender/recipient pairs so that they are not subjected to server interrogation.
Never quarantine. If a message is not to be delivered to a mailbox, then reject it in protocol.
Do not employ gateway software to look at the content of the message. The spammers will fool your software and you will lose legitimate messages.
Only accept messages from mail servers that comply with standards.