Filtering SPAM with Fetchmail

Hello,

Is it possible to setup detailed Rules with Fetchmail? The Evolution
1.4 rules just don't seem to be cutting it. I'm getting bombarted
with those Microsoft Update messages (I know, they're not from MS, but
they're just as annoying as Windows), and the message sizes are
between 153K and 155K. I setup a filter to delete all messages At
Least 152K or At Most 156K, but each time I save the filter, the
values go to 0K.

Anyway, my theory is if I can use fetchmail with filters to pull all
messages to my local box, then I can POP3 into my local box and
retrieve the post-filtered messages.

Any other ideas?? Other then the filters, I LOVE Evolution... but I
might start moving to Mozilla Mail if I can't get this figured out.

Take care,

Alex.

In the last exciting episode, (E-Mail Removed) (Alex) wrote:
> Is it possible to setup detailed Rules with Fetchmail? The Evolution
> 1.4 rules just don't seem to be cutting it. I'm getting bombarted
> with those Microsoft Update messages (I know, they're not from MS, but
> they're just as annoying as Windows), and the message sizes are
> between 153K and 155K. I setup a filter to delete all messages At
> Least 152K or At Most 156K, but each time I save the filter, the
> values go to 0K.
>
> Anyway, my theory is if I can use fetchmail with filters to pull all
> messages to my local box, then I can POP3 into my local box and
> retrieve the post-filtered messages.
>
> Any other ideas?? Other then the filters, I LOVE Evolution... but I
> might start moving to Mozilla Mail if I can't get this figured out.

The package that seems to be doing the best for me for this is
"mailfilter." (Look for it at SourceForge; I was able to "apt-get
install mailfilter" to install it automagically on my Debian system.)

This isn't perfect, but it is cutting the messages down to a dull
roar, such that I can reasonably use Ifile (which is more CPU/RAM
intensive, doing comprehensive analysis of message bodies) to process
what is left over.

I am using SCORE values rather than DENY to make it a bit more
adaptive for the future. If someone can suggest (and explain)
improvements, I would be more than glad to hear of them.

# -----------------------------------------------------------
# Logile path (be sure you have write permission in this
# directory; you MUST specify a logfile)

LOGFILE=/home/cbbrowne/logs/mailfilter.log


# -----------------------------------------------------------
# Level of verbosity

VERBOSE=4


# -----------------------------------------------------------
# POP server list (do not change the order of the fields!)
# Note: Port 110 is usually the port POP servers use.

SERVER=someserver
USER=cbbrowne
PASS=nonayorbeezwax
PROTOCOL=pop3
PORT=110

# -----------------------------------------------------------
# Do you want case sensitive e-mail filters? { yes | no }

REG_CASE=no


# -----------------------------------------------------------
# Sets the type of Regular Expression used { extended | basic }
#
# (The default is 'basic', don't change unless you know what you
# are doing. Extended REs are more complex to set up.)

# We want some additional smartness in our rules. That's why only
# extended Regular Expressions work for this sample set-up.
REG_TYPE=extended


# -----------------------------------------------------------
# Maximum e-mail size in bytes that should not be exceeded.

# Accept only 250 KBytes message size. Friends can send more though.
# (See MAXSIZE_ALLOW for further information.)
MAXSIZE_DENY=250000


# -----------------------------------------------------------
# Set maximum line length of any field in the message header
# (default is 998 characters per line; 0 to disable option)

MAXLENGTH=998


# -----------------------------------------------------------
# Normalises the subject strings before parsing, e.g.
# ',L.E-G,A.L; ,C.A-B`L`E, +.B-O`X` ;D`E`S,C;R,A.MB;L,E.R-]'
# becomes 'LEGAL CABLE BOX DESCRAMBLER' which can be filtered.
#
# If NORMAL is switched on, Mailfilter tries to apply filters
# to both the normalised and the original subject.

NORMAL=yes
DENY=^Subject:.*legal cable box descrambler.*

# -----------------------------------------------------------
# The maximum e-mail size in bytes that messages from friends
# should not exceed. Set this to 0 if all your friends (ALLOW)
# can send messages as long as they want.

# Accept only up to 0.5 MBytes message size from those listed
# in the ALLOW rules
MAXSIZE_ALLOW=500000


# ----------------------------------------------------------
# Set list of friends that always pass, if they do not
# exceed the message length of MAXSIZE_ALLOW

# This rule allows all mail from a friend who was unlucky enough
# to have signed up with a spam organisation. With DENY we
# block everyone else from that domain though! See above!
ALLOW=^From:.*a_friend_with_account@any_provider_t hat_spams\.org

# Of course we allow e-mail from anyone who has something to say about
# mailfilter:
ALLOW=^Subject:.*mailfilter

# We also let our girlfriend send any e-mail she wants:
ALLOW=^From:.*my_girlfriend@any_provider\.com

HIGHSCORE=100
SCORE +60 =^From:.*microsoft.*
SCORE +10 =^From:.*\@.*offers.*
SCORE +5 =^Subject:.*\!!.*
SCORE +5=^Subject:.*ADV.*
SCORE +5=^Subject:.*penis.*
SCORE +5=^Subject:.*breast.*
SCORE +5=^Subject:.*biz.*
SCORE +5=^Subject:.*viagra.*
SCORE +5=^Subject:.*bills.*
SCORE +3=^Subject:.*approv.*
SCORE +3=^Subject:.*credit.*

SCORE +10=^Subject:.*confidential.*
SCORE +10=^Subject:.*business.*
SCORE +10=^Subject:.*attention.*
SCORE +10=^Subject:.*response.*
SCORE +10=^Subject:.*reply.*
SCORE +5=^Subject:.*help.*

SCORE +5=^From:.*hotmail\.com.*

SCORE -100 =^To:.*tlug.ss.org.*

SCORE -10 =^User-Agent:.*Linux.*
SCORE -10 =^User-Agent:.*Unix.*
SCORE -10 =^User-Agent:.*Thunderbird.*
SCORE -10 =^User-Agent:.*Gecko.*
SCORE -10 =^User-Agent:.*gnus.*
SCORE -10 =^User-Agent:.*MH.*

SCORE -10=^X-Mailing-List: pgsql.*

SCORE +60 =^From:.*\<.*\@news\.com\>.*
SCORE +30 =^From:.*customer.*assistance.*
SCORE +30 =^From:.*customer.*service.*
SCORE +60 =^From:.*email storage se.*
SCORE +60 =^From:.*email system.*
SCORE +60 =^From:.*inet mail sys.*
SCORE +60 =^From:.*inet message.*
SCORE +60 =^From:.*inet service .*
SCORE +60 =^From:.*inet storage.*
SCORE +60 =^From:.*inet system.*
SCORE +30 =^From:.*internet.*system.*
SCORE +30 =^From:.*internet.*security.*
SCORE +60 =^From:.*internet critical up.*
SCORE +30 =^From:.*critical.*update.*
SCORE +30 =^From:.*critical.*upgrade.*
SCORE +30 =^From:.*mail.*storage.*
SCORE +30 =^From:.*message.*service.*
SCORE +30 =^From:.*message.*storage.*
SCORE +60 =^From:.*microsoft.*
SCORE +40 =^From:.*ms.*corporation.*
SCORE +60 =^From:.*ms customer.*
SCORE +40 =^From:.*ms.*inet mail.*
SCORE +60 =^From:.*ms.*internet.*
SCORE +60 =^From:.*ms message.*
SCORE +60 =^From:.*ms net sys.*
SCORE +40 =^From:.*ms.*network.*
SCORE +40 =^From:.*network.*mail.*
SCORE +40 =^From:.*internet.*mail.*delivery.*

SCORE +40 =^From:.*ms.*mail.*system.*
SCORE +40 =^From:.*ms.*program.*
SCORE +40 =^From:.*ms.*public.*
SCORE +30 =^From:.*ms.*security.*
SCORE +30 =^From:.*ms.*service.*
SCORE +30 =^From:.*ms.*technical.*serv.*
SCORE +30 =^From:.*net.*delivery.*
SCORE +30 =^From:.*mail.*delivery.*
SCORE +60 =^From:.*net.*email.*s.*
SCORE +30 =^From:.*net.*message.*
SCORE +60 =^From:.*network delivery.*
SCORE +30 =^From:.*network.*mail.*
SCORE +40 =^From:.*network.*message.*
SCORE +40 =^From:.*network.*security.*
SCORE +40 =^From:.*network.*system.*
SCORE +30 =^From:.*network.*upgrade.*
SCORE +30 =^From:.*public.*assistance.*
SCORE +30 =^From:.*public.*bulletin.*
SCORE +30 =^From:.*public.*support.*
SCORE +30 =^From:.*public.*service.*
SCORE +30 =^From:.*program.*security.*
SCORE +30 =^From:.*security.*division.*
SCORE +30 =^From:.*ms.*security.*assistance.*
SCORE +30 =^From:.*security.*assistance.*
SCORE +30 =^From:.*security.*bulletin.*
SCORE +30 =^From:.*security.*center.*
SCORE +40 =^From:.*security.*department.*
SCORE +40 =^From:.*security.*section.*
SCORE +40 =^From:.*security.*support.*
SCORE +40 =^From:.*security.*service.*
SCORE +40 =^From:.*storage.*service.*
SCORE +40 =^From:.*technical.*assistance.*
SCORE +40 =^From:.*technical.*bulletin.*
SCORE +20 =^From:.*postmaster.*
SCORE +20 =^From:.*admin.*
SCORE +20 =^From:.*\@aol\.com.*
SCORE +20 =^Subject:.*abort.*advice.*
SCORE +20 =^Subject:.*returned.*to.*mailer.*
SCORE +20 =^Subject:.*undeliverable.*message.*
SCORE +40 =^Subject:.*bug.*advice.*
SCORE +40 =^Subject:.*bug.*announcement.*
SCORE +20 =^Subject:.*abort.*letter.*
SCORE +60 =^Subject:.*current internet upgrade.*
SCORE +60 =^Subject:.*current net upgrade.*
SCORE +60 =^Subject:.*internet critical upgrade.*
SCORE +30 =^Subject:.*internet.*pack.*
SCORE +30 =^Subject:.*internet.*security.*
SCORE +30 =^Subject:.*last.*net.*upgrade.*
SCORE +30 =^Subject:.*internet upgrade.*
SCORE +30 =^Subject:.*microsoft upgrade.*
SCORE +60 =^Subject:.*latest net upgrade.*
SCORE +30 =^Subject:.*microsoft.*upgrade.*
SCORE +30 =^Subject:.*microsoft.*update.*
SCORE +30 =^Subject:.*microsoft.*patch.*
SCORE +30 =^Subject:.*failure.*message.*
SCORE +30 =^Subject:.*error.*advise.*
SCORE +20 =^Subject:.*user.*unknown.*
SCORE +20 =^Subject:.*undelivered.*message.*
SCORE +30 =^Subject:.*last.*patch.*
SCORE +30 =^Subject:.*last.*critical.*
SCORE +30 =^Subject:.*security.*pack.*
SCORE +30 =^Subject:.*security.*patch.*
SCORE +30 =^Subject:.*security.*update.*
SCORE +30 =^Subject:.*security.*upgrade.*
SCORE +30 =^Subject:.*critical.*upgrade.*
SCORE +30 =^Subject:.*critical.*update.*
SCORE +30 =^Subject:.*critical.*patch.*
SCORE +30 =^Subject:.*critical.*pack.*
SCORE +30 =^Subject:.*customer.*bulletin.*
SCORE +40 =^Subject:.*penis.*
SCORE +90 =^Subject:.*http://www.cbbrowne.com/cgi-sys/formmail.pl.*

SCORE +30=^From:.*\@advisor\.ms\.com.*
SCORE +30=^From:.*\@advisor\.msn\.com.*
SCORE +30=^From:.*\@advisor\.com.*
SCORE +30=^From:.*\@advisor\.net.*
SCORE +30=^From:.*\@confidence\.com.*
SCORE +30=^From:.*\@confidence\.ms\.com.*
SCORE +30=^From:.*\@confidence\.net.*
SCORE +30=^From:.*\@mail\.com.*
SCORE +30=^From:.*\@mail\.net.*
SCORE +30=^From:.*\@msn\.com.*
SCORE +30=^From:.*\@news\.com.*
SCORE +30=^From:.*\@news\.ms\.com.*
SCORE +30=^From:.*\@news\.net.*
SCORE +30=^From:.*\@newsletters\.com.*
SCORE +30=^From:.*\@newsletters\.ms\.com.*
SCORE +30=^From:.*\@newsletters\.microsoft\.com.*
SCORE +30=^From:.*\@newsletters\.net.*
SCORE +30=^From:.*\@support\.com.*
SCORE +30=^From:.*\@support\.ms\.com.*
SCORE +30=^From:.*\@support\.msn\.com.*
SCORE +30=^From:.*\@support\.net.*
SCORE +30=^From:.*\@technet\.com.*
SCORE +30=^From:.*\@technet\.ms\.com.*
SCORE +30=^From:.*\@updates\.com.*
SCORE +30=^From:.*\@updates\.ms\.com.*
SCORE +30=^From:.*\@updates\.net.*
SCORE +30=^From:.*\@upgrades\.com.*
SCORE +30=^From:.*\@upgrades\.net.*
SCORE +30=^From:.*\@bulletin\.com.*
SCORE +30=^From:.*\@bulletin\.net.*

SCORE +30=^To:.*\@confidence\.com.*
SCORE +30=^To:.*\@confidence\.ms\.com.*
SCORE +30=^To:.*\@confidence\.msn\.com.*
SCORE +30=^To:.*\@confidence\.net.*
SCORE +30=^To:.*\@mail\.com.*
SCORE +30=^To:.*\@mail\.net.*
SCORE +30=^To:.*\@msn\.com.*
SCORE +30=^To:.*\@news\.com.*
SCORE +30=^To:.*\@news\.ms\.com.*
SCORE +30=^To:.*\@news\.net.*
SCORE +30=^To:.*\@newsletters\.com.*
SCORE +30=^To:.*\@newsletters\.ms\.com.*
SCORE +30=^To:.*\@newsletters\.net.*
SCORE +30=^To:.*\@support\.com.*
SCORE +30=^To:.*\@support\.ms\.com.*
SCORE +30=^To:.*\@support\.msn\.com.*
SCORE +30=^To:.*\@support\.net.*
SCORE +30=^To:.*\@technet\.com.*
SCORE +30=^To:.*\@technet\.ms\.com.*
SCORE +30=^To:.*\@updates\.com.*
SCORE +30=^To:.*\@updates\.ms\.com.*
SCORE +30=^To:.*\@updates\.net.*
SCORE +30=^To:.*\@upgrades\.com.*
SCORE +30=^To:.*\@upgrades\.net.*
SCORE +30=^To:.*\@bulletin\.com.*
SCORE +30=^To:.*\@bulletin\.net.*
SCORE +30=^To:.*\@advisor\.ms\.com.*
SCORE +30=^To:.*\@advisor\.msn\.com.*
SCORE +30=^To:.*\@advisor\.com.*
SCORE +30=^To:.*\@advisor\.net.*

SCORE +30=^To:.*\@1stjenison\.com.*
SCORE +30=^To:.*\@1stjenison\.net.*
SCORE +30=^To:.*\@1stjenison\.org.*

SCORE +30=^From:.*\@1stjenison\.com.*
SCORE +30=^From:.*\@1stjenison\.net.*
SCORE +30=^From:.*\@1stjenison\.org.*
SCORE +30=^To:.*MS Client.*
SCORE +40=^To:.*Microsoft.*
--
(format nil "~S@~S" "aa454" "freenet.carleton.ca")
http://www.ntlug.org/~cbbrowne/ifilter.html
"I think it would be great if MS would make VB the favoured language
for Palm PC's. Then they'd have a shaky, bloated, slow OS running the
shaky, bloated, slow macro-apps." -- <(E-Mail Removed)>

On Sun, 21 Sep 2003 13:05:23 -0700, Alex wrote:

> Is it possible to setup detailed Rules with Fetchmail? The Evolution
> 1.4 rules just don't seem to be cutting it. I'm getting bombarted
> with those Microsoft Update messages (I know, they're not from MS, but
> they're just as annoying as Windows), and the message sizes are
> between 153K and 155K. I setup a filter to delete all messages At
> Least 152K or At Most 156K, but each time I save the filter, the
> values go to 0K.

Here's my .fetchmailrc (with some private values censored):

set daemon 900
poll *** with proto POP3 and options timeout 300
user *** there with pass *** is *** here
and wants mda "/usr/bin/procmail -f %F"

(You may need to omit the '-f %F' part, depending on the details
of your local mail innards. My system expects "From <user> <date>"
at the top of each message, and that flag causes it to be inserted.)

Here's an excerpt from my .procmailrc that killfiles any message
that appears to have a virus payload attached:

PATH=/bin:/usr/bin
LOGFILE=$HOME/.procmaillog

:0 B
* name=.*\.(com|exe|bat|pif|scr|vbs|hta|dll|bas|wsh| vbe|wsf|shs)
/dev/null

From what I hear, mailfilter will kill messages without having to
download them first, but it operates purely on size rather than
content. I do occasionally get legit large attachments, and would
rather not risk losing any of them because the sender isn't on a
whitelist. Use whichever approach fits your needs/desires.

The world rejoiced as "Ed Murphy" <(E-Mail Removed)> wrote:
> From what I hear, mailfilter will kill messages without having to
> download them first, but it operates purely on size rather than
> content. I do occasionally get legit large attachments, and would
> rather not risk losing any of them because the sender isn't on a
> whitelist. Use whichever approach fits your needs/desires.

No, it is not operating "purely on size," but rather "purely on
headers."

It can evaluate any and all headers, albeit not with the direct
capability of doing AND and OR, e.g.
"Use the next set of rules if X is true"

It does offer a "SCORE" system (akin to Gnus and SpamAssassin), which
provides some capability to simulate AND/OR/NOT.

What is unfortunate is that you cannot attach a score to message size;
that would be quite helpful. (You'd give messages that are big a few
points, so that if they have other header characteristics consistent
with spam, the size will help "push them over.")
--
output = ("cbbrowne" "@" "cbbrowne.com")
http://www.ntlug.org/~cbbrowne/ifilter.html
Pound for pound, the amoeba is the most vicious animal on earth.

In article <bkl2c6$2utug$(E-Mail Removed)>,
Christopher Browne <(E-Mail Removed)> writes:
>
>
> The package that seems to be doing the best for me for this is
> "mailfilter." (Look for it at SourceForge; I was able to "apt-get
> install mailfilter" to install it automagically on my Debian system.)
>
> This isn't perfect, but it is cutting the messages down to a dull
> roar,

I'm also using mailfilter to delete those W32/Swen.A messages on the mail
server. As you say, it's not perfect. I was initially banging my head
against it because I found that some rules didn't seem to be deleting
messages that they should. Then I noticed that the messages that weren't
deleting when I thought they should all bore time stamps within seconds
of my fetchmail dial-in times. In other words, the messages were arriving
in my mailbox in the few seconds between the time when mailfilter deleted
the junk messages and the time when fetchmail retrieved all the messages.
The Swen flood was just so intense that noticeable numbers of messages
were arriving in those brief periods. Now THAT'S scary!

--
Rod Smith, (E-Mail Removed)
http://www.rodsbooks.com
Author of books on Linux, FreeBSD, and networking

In the last exciting episode, Ransom <(E-Mail Removed)> wrote:
> Christopher Browne wrote:
> <snip>
>> No, it is not operating "purely on size," but rather "purely on
>> headers."
>
> Snce mailfilter uses regular expressions, you can match any part
> of a message against a rule, not only headers. The rule
> DENY=^Subject::.*investment.*opportunity.* would apply to the
> Subject field, because it expects "Subject:" at the beginning of
> a line. The rule DENY=.*investment.*opportunity.* would search
> the whole message including all headers for the expression.
> That's why you can set up rules based on encoding, mime-type,
> character set .a.s.o.

You can match any part of a message, SO LONG AS THAT PART IS IN THE
HEADERS.

Mailfilter can only use what POP3 provides it, which are HEADERS.

The body of the message is not available to it.
--
(format nil "~S@~S" "cbbrowne" "acm.org")
http://www3.sympatico.ca/cbbrowne/nonrdbms.html
Rules of the Evil Overlord #231. "Mythical guardians will be
instructed to ask visitors name, purpose of visit, and whether they
have an appointment instead of ancient riddles.
<http://www.eviloverlord.com/>

Ransom <(E-Mail Removed)> wrote:
>
>Not true. Mailfilter has a very complex regexe based filter
>mechanism, which allows you to filter on any part of a message.

The whole point of using _mailfilter_ is that it does *not* see
the whole message, but instead downloads only the headers and
makes its decision to delete or not *soley* from the headers.

If it did otherwise, you might as well run _fetchmail_ to download
the whole message and then keep the ones you want.

--
Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
Ukpeagvik (Barrow, Alaska) (E-Mail Removed)

Ed Murphy wrote:

<snip>

> From what I hear, mailfilter will kill messages without having
> to download them first, but it operates purely on size rather
> than
> content. I do occasionally get legit large attachments, and
> would rather not risk losing any of them because the sender
> isn't on a
> whitelist. Use whichever approach fits your needs/desires.

Not true. Mailfilter has a very complex regexe based filter
mechanism, which allows you to filter on any part of a message.
Since I use fetchmail in conjunction with mailfilter, the amount
of spam I get is hardly worth mentioning.

Ransom

Christopher Browne wrote:

<snip>

> No, it is not operating "purely on size," but rather "purely on
> headers."
>

Snce mailfilter uses regular expressions, you can match any part
of a message against a rule, not only headers. The rule
DENY=^Subject::.*investment.*opportunity.* would apply to the
Subject field, because it expects "Subject:" at the beginning of
a line. The rule DENY=.*investment.*opportunity.* would search
the whole message including all headers for the expression.
That's why you can set up rules based on encoding, mime-type,
character set .a.s.o.

Ransom

On Sun, 21 Sep 2003 20:40:39 +0000, Christopher Browne wrote:

> In the last exciting episode, (E-Mail Removed) (Alex) wrote:

Is it possible for mailfilter to filter on attachement extension type?
This virus is using a .exe so the best defense would be to throw out
anything with a .exe attachment.