How to deal with 98% spam? (corporate case)

Hi,

I have a small business. Every day we get 10-20 legit emails, and
(especially over a weekend) about 50x as many in spam.

The spam probably comes from email addresses ripped off our website
over the years, plus a load sent to admin@ and webmaster@ etc. I
realise one can stop some of this by having java code in place of
mailto: links, or use www enquiry forms (which a lot of people hate)
but it's too late to do that now.

This very high spam ratio means that no simple rules can be used
effectively. Blacklisting (DNS and addresses) works only partly; we
still get loads of emails clogging up our email clients. So I am
looking for a way to get rid of it more effectively, while not dumping
any legit incoming emails.

We've been using Mailwasher, which is pretty good but if set up to
work usefully, it is not 100% safe. It is also not a good solution for
multiple users receiving email - we have to run it first thing in the
morning, on one specific PC where it maintains its database, before
anybody reads their email. It also has no automatic way of adding to
the whitelist because it doesn't see outgoing emails.

It occured to me that if we could maintain a whitelist (by
automatically adding the To: header from our OUTgoing email to it) we
could have a very good working system which would never drop an email
from an existing contact. Moreover, we could get it started by
processing all our existing emails (going back to 1995) and extracting
the To: headers from them.

One can get services like Messagelabs but they cost a fair bit of
money. Also they can blacklist some addresses without us knowing about
it. I know a man who does "legit" commercial mailings and he has a
team of people working for him who spend most of their time working
out how to get around these message processing services!

So I have decided to set up an in house mail server on which we can
run antivirus software, do spam DNS checking, bouncing, etc, and
through which outgoing email will pass so it can be added to the
whitelist.

To complicate matters somewhat, we want to enable all emails from a
particular company, so if we send an email reply to e.g.
joe-(E-Mail Removed) we want to add *@flowsensors.co.uk to the
whitelist. But if we get an email from joe-(E-Mail Removed) we don't
want to do that! So there would be some rules; e.g. major domains like
aol, yahoo, btinternet etc would never acquire wildcards.

Does anyone know of any commercial software which would do this? We
plan to run FreeBSD on the server - it will also be a www/ftp server,
later running online shopping...

I gather sendmail can have plug-ins so this sort of thing could be
written. I know someone who could do it but he thinks what I am
proposing is an overkill.

Presently we have 64k ISDN dial-up access but will be getting BB soon,
so I am trying to get something sorted.

I would appreciate any suggestions as to how to do this perhaps more
effectively.



Peter.
--
Return address is invalid to help stop junk mail.
E-mail replies to (E-Mail Removed) but remove the X and the Y.
Please do NOT copy usenet posts to email - it is NOT necessary.

On Fri, 20 Aug 2004 11:15:43 +0100, (E-Mail Removed) (Peter) wrote:


>Does anyone know of any commercial software which would do this? We
>plan to run FreeBSD on the server - it will also be a www/ftp server,
>later running online shopping...

Start grey listing incoming mail using spamd.


greg

--
Es ist mein Teil - nein
Mein Teil - nein
Denn das ist mein Teil - nein
Mein Teil - nein

On Fri, 20 Aug 2004 11:15:43 +0100, (E-Mail Removed) (Peter) wrote:

>Hi,
>
>I have a small business. Every day we get 10-20 legit emails, and
>(especially over a weekend) about 50x as many in spam.

Hello,

An ongoing problem! I don't know of anything that meets your specific
requirements - collecting all the spam in the morning, automated
adding to whitelists for example - but here is a suggestion.

I'd suggest using an artificial intelligence system for identifying
spam / not spam. I use K9 which is free http://www.keir.net/k9.html .
This acts as an email "proxy server" which adds a message to each
email it thinks is spam, either in the form of an extra header or in
the subject of the email, so that you can then filter it using your
mailreader.

You could set K9 running on one machine and then everybody in your
company uses their standard mail clients (with some simple settings
changes) to retrieve email via that machine. Email is filtered "on the
fly" (doesn't slow it down signficantly) and over time K9 slowly
learns what your company considers as spam - and the advantage of
everybody using K9 on the same computer means everybody benefits from
everybody else's spam recognition.

Once a day for the first couple of weeks you would have to visit the
server running K9 and identify any spam that has got through the
filters - this won't be necessary for long as K9 learns what is or
isn't spam.

My copy of K9 now correctly identifies 96% of all spam, and has never
had a "false positive"

Just a suggestion...
--
King Queen - Remove .lartsspammers to reply. http://www.kingqueen.org.uk
"Advertising is the rattling of a stick in a swill bucket" George Orwell

On 20 Aug 2004 in uk.telecom.broadband, Peter wrote:

>I would appreciate any suggestions as to how to do this
>perhaps more effectively.

Please let us all know what solution you end up with. I can see
the benefit of your whitelist idea, and know of problems with any
mail addresses being (historically) on websites.

98% does seem a major problem - I was getting a lot of junk mail
at one time (44000 items one day alone) but seem to have cut it
back a lot (K9 shows 98% accuracy for detection of spam over an
11 month period for which this data has been collected, so I've
certainly got it down a lot...

By coincidence total incoming items processed by K9 is around
45000, of which ~33% is valid. The rest does gets marked as
spam but is not deleted. With your situation on ISDN I can
see deletion being near essential to limit the quantity of
mail. I've made a big reduction in junk mail for one of my
clients (which had some mail addresses on their site back in
'96) and will ask them if they'll let you contact them direct,
if you're interested. I will be visiting them this afternoon,
by chance, so will mention it to their IT guy - they're happy
with their mail service now, after replacing MS Exchange with
MDaemon (Exchange was on an NT server which died a year ago,
corrupting their mail database and was costly to restore).

In article <(E-Mail Removed)>,
Peter <(E-Mail Removed)> wrote:

>I gather sendmail can have plug-ins so this sort of thing could be
>written. I know someone who could do it but he thinks what I am
>proposing is an overkill.

Sendmail is fine, and no need to write anything - connected
with Mimedefang and Spamassasian and an anti-virus program it's
great. Spamassasian (& Mimedefang) have lots of configurability to do
just about everything you want. It's a bit of a fiddle to get going,
but the HowTos are very well written and easy to follow.

In these spam & virus laden days nothing is overkill...

>Presently we have 64k ISDN dial-up access but will be getting BB soon,
>so I am trying to get something sorted.

I probably would advise you to not run your web site from inside the
broadband, but look for an external web host - unless you get something
other than a standard package your outgoing bandwidth will likely only
be 256Kbps which is adequate for a very low volume site, if you place
large images on it, someone with a 512Kbps line will saturate all your
outgoing bandwidth when they are sucking stuff off it. (and unless you
are clever with the various bandwidth management and QoS witchcraft
it'll then kill incoming traffic at the same time)

>I would appreciate any suggestions as to how to do this perhaps more
>effectively.

Drop me an email for more details - I run Mimedefang+SA+Sendmail on
several Linux setups, but they run just as well under FreeBSD too.

Gordon

On Fri, 20 Aug 2004 11:15:43 +0100, (E-Mail Removed) (Peter) wrote:

<snip ideas about anti-spam>

Since you're going with a FreeBSD server, try running up a copy of
Spamassassin. Very flexible software. Can take a while to configure to
your liking though.

Peter wrote:
> Hi,
>
> I have a small business. Every day we get 10-20 legit emails, and
> (especially over a weekend) about 50x as many in spam.
>
> The spam probably comes from email addresses ripped off our website
> over the years, plus a load sent to admin@ and webmaster@ etc. I
> realise one can stop some of this by having java code in place of
> mailto: links, or use www enquiry forms (which a lot of people hate)
> but it's too late to do that now.
>
> This very high spam ratio means that no simple rules can be used
> effectively. Blacklisting (DNS and addresses) works only partly; we
> still get loads of emails clogging up our email clients. So I am
> looking for a way to get rid of it more effectively, while not dumping
> any legit incoming emails.
>
> We've been using Mailwasher, which is pretty good but if set up to
> work usefully, it is not 100% safe. It is also not a good solution for
> multiple users receiving email - we have to run it first thing in the
> morning, on one specific PC where it maintains its database, before
> anybody reads their email. It also has no automatic way of adding to
> the whitelist because it doesn't see outgoing emails.
>
> It occured to me that if we could maintain a whitelist (by
> automatically adding the To: header from our OUTgoing email to it) we
> could have a very good working system which would never drop an email
> from an existing contact. Moreover, we could get it started by
> processing all our existing emails (going back to 1995) and extracting
> the To: headers from them.
>
> One can get services like Messagelabs but they cost a fair bit of
> money. Also they can blacklist some addresses without us knowing about
> it. I know a man who does "legit" commercial mailings and he has a
> team of people working for him who spend most of their time working
> out how to get around these message processing services!
>
> So I have decided to set up an in house mail server on which we can
> run antivirus software, do spam DNS checking, bouncing, etc, and
> through which outgoing email will pass so it can be added to the
> whitelist.

This is a very sensible approach, I would suggest that you choose
something like Exim (http://www.exim.org) together with SpamAssassin
(http://www.spamassassin.org). SpamAssassin (just on the point of a
major updated release, in -rc status now) is able to filter on a
combination of built-in rules, user specified rules, and Bayesian word
probability analysis that learns your spam and ham mail characteristics
and scores appropriately.

Exim (or another MTA if you prefer it) can be integrated with
SpamAssassin in several ways, either used to reject mail at smtp time if
it seems spammy enough, or simply to mark it up and then allow mail
client filters to check on the status of the mail and decide whether to
bin it or bung it into a spam folder for review.

Some information on this can be found here:

http://wiki.apache.org/spamassassin/IntegratedInMta

>
> To complicate matters somewhat, we want to enable all emails from a
> particular company, so if we send an email reply to e.g.
> joe-(E-Mail Removed) we want to add *@flowsensors.co.uk to the
> whitelist. But if we get an email from joe-(E-Mail Removed) we don't
> want to do that! So there would be some rules; e.g. major domains like
> aol, yahoo, btinternet etc would never acquire wildcards.

It should be possible to script this so that on sent mail Exim calls a
script that adds whitelist entries to the SpamAssassin local
configuration file to allow return mail.

Generic whitelisting is also very simple using SpamAssassin.

But you may find that you don't need it if you use Exim, especially if
you look at:

http://slett.net/spam-filtering-for-mx/index.html

which has some good sugegstions on how to set up smtp transactions to
avoid a fair amount of spam.

>
> Does anyone know of any commercial software which would do this? We
> plan to run FreeBSD on the server - it will also be a www/ftp server,
> later running online shopping...

Both Exim and SpamAssassin will run under FreeBSD (or indeed under
Linux). And they are Free software so do not directly cost money, just
time to learn how to configure.

>
> I gather sendmail can have plug-ins so this sort of thing could be
> written. I know someone who could do it but he thinks what I am
> proposing is an overkill.

Nothing is overkill where spam is concerned.

>
> Presently we have 64k ISDN dial-up access but will be getting BB soon,
> so I am trying to get something sorted.
>
> I would appreciate any suggestions as to how to do this perhaps more
> effectively.

Hope the above is useful.

--

Brian Morrison

please observe reply-to address

On Fri, 20 Aug 2004 12:23:54 +0100, King Queen wrote:

> You could set K9 running on one machine and then everybody in your
> company uses their standard mail clients (with some simple settings
> changes) to retrieve email via that machine.

Note that K9 only runs on Windows though. One cross-platform alternative
is Popfile (popfile.sourceforge.net)

--
TimH
Pull tooth to reply by email

On Fri, 20 Aug 2004 11:15:43 +0100, (E-Mail Removed) (Peter) wrote:

>Hi,
>
>I have a small business. Every day we get 10-20 legit emails, and
>(especially over a weekend) about 50x as many in spam.
>
>The spam probably comes from email addresses ripped off our website
>over the years, plus a load sent to admin@ and webmaster@ etc. I
>realise one can stop some of this by having java code in place of
>mailto: links, or use www enquiry forms (which a lot of people hate)
>but it's too late to do that now.
>
[snip]

Solutions you have considered in the rest of your postings may provide
an answer to your problem but the effort involved is quite extensive.
Perhaps you need to turn the problem on its head and consider if you
are doing all you can to protect your email address. Here are some
ideas which others might decry, but they work for me.

* Why use admin@ and webmaster@ as valid email addresses? Other
choices might be better such as officemanager@ and designer@ could be
used instead.

* Protect mailto: entries behind java code. I've done this for over a
year now and the protected email addresses have yet to be compromised.

* Avoid the use of CC in your emails; use BCC instead.

* Use a secondary email address for use on the WEB. And in
newsgroups, no email address at all.

* Divert any mail address to non valid email address into a separate
account where from time to time only the headers are read using, say,
POP3SCAN. Easy enough to spot the odd typo error in the email address
against the deluge of obvious SPAM messages.

* Only give your company's WEB address on your business card; anyone
wanting to send you an email can discover your email address there.

It does seem to me that once you get onto a spammers list, the mail
never stops arriving. Re-activating an old email address recently
[that was last years over three years ago] the SPAM was still arriving
each day!

David Bradley

The entity currently known as David Bradley wrote:

> * Only give your company's WEB address on your business card; anyone
> wanting to send you an email can discover your email address there.

That's just unhelpful paranoia. If spammers are willing to transcribe
your email from a business card (which I doubt, and ho did they get it
in the first place?), then they'll be just as willing to transcribe it
from the javascript-procted version on the web page.

--
Iain A F Fleming