sendmail wrapper to find spammer-magnet scripts

Hi folks,
I've managed to bring server loads and local/preprocess
queues down by implementing valid-user checking in C so
qmail can block mail at the front door. So now we are not
backscattering other people's spam, but now I find that
our remote queue is now filled with locally generated spam.

The problem is.. once the mail is just one of many in the
queue, I have only (invoked by uid: nobody) to tell me that
it's from a cgi-script and only the time/date to try to
track the offending script down in the access_log.

I'd like to know if there is any wrapper for the mail
command which simply logs the calling script's name
then carries on with the job of injecting into the
queue.

I'd try and script one myself but I'm a little frightened
off by qmail using fd1 and fd2 for sending the message
and headers (message first) etc. I'm still learning and
using fd2 stderr as an input just freaks me out.

I also think the odds are rather high that this problem
has been met and solved many times over. Unfortunately
my thinking has been along the lines of:
mail logging or
apache userid or
sendmail wrappers
and nothing so far seems to address the problem. I'm
sure I'm searching for the wrong thing, so time for
some humans to have a look at it.

Thanks
--
Andy Richardson

Never criticize a man 'til you've walked a mile in his shoes.
After that, you can say what you like..
'cos you're a mile away and you've got his shoes.

On Thu, 28 Dec 2006 18:55:58 +0000, Andy wrote:
> Hi folks,
> I've managed to bring server loads and local/preprocess
> queues down by implementing valid-user checking in C so
> qmail can block mail at the front door.

> So now we are not
> backscattering other people's spam,

thank you! Thank You!! THANK YOU!!! *THANK YOU!!!!*

HNY
Jonesy

In comp.os.linux.networking Andy <andy-(E-Mail Removed)>:
> Hi folks,
> I've managed to bring server loads and local/preprocess
> queues down by implementing valid-user checking in C so
> qmail can block mail at the front door. So now we are not
> backscattering other people's spam, but now I find that
> our remote queue is now filled with locally generated spam.

[..]

> I'd like to know if there is any wrapper for the mail
> command which simply logs the calling script's name

How would you know it is using the mail command? There are
several ways to inject mail. One could even use telnet.

> then carries on with the job of injecting into the
> queue.

I'd start with the obvious:

- which scripts send mail at all?
- how often is each invoked?
- how often is local mail generated send from the user apache
runs as?

These figures should help to drill down the problem, as usual
'grep' is your friend and should be all you need. I'd hurry up
before your system is getting blacklisted for being abused as
spambot. In general I wouldn't recommend to run a public
webserver on some MTA.

Good luck

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 439: Hot Java has gone cold

Hello again Michael,
it was your suggestions in the thread called "multifunction
production servers" that encouraged me to get on and sort
this spam thing out.

>> I'd like to know if there is any wrapper for the mail
>> command which simply logs the calling script's name
>
> How would you know it is using the mail command? There are
> several ways to inject mail. One could even use telnet.

I don't, but since our servers are making it so easy for
spammers to use scripts we host, they don't need to bother
with anything clever. Also, if I were to set aside just one
working day to look at only one server, I could find 20+
scripts easily which are actively being exploited by
spammers. So I thought I'd tackle those first.

> I'd start with the obvious:
>
> - which scripts send mail at all?
> - how often is each invoked?
> - how often is local mail generated send from the user apache
> runs as?

I think our hard drives are mounted noatime (since they are
in PIO mode remember) So that is one tool gone for
invokation frequency.
Also some scripts are used once, but with 100-200 cc:/Bcc:
and others are called hundreds of times with one mail
address each time.

If we average and say 50 servers each with 1000 websites,
(some have fewer some have many more) each site COULD
contain .cgi/.pl/.php files which use the mail function.
Nobody knows how many, but a
grep -ri 'mail(' /home/*/public_html/* would give a huge list.

I know -c would count,or |sort|uniq would list them for me,
but we are talking hundreds (on each server).

Currently the drill is something like this:
1) Woah look at the mail queues on server31 !
2) Ah, they're outgoing. List them
3) pick out the hundred or so which were invoked by 99
(nobody/apache)
4) which ones were injected since the last time access_log
was processed and wiped? Oh only 3.. view them for spam like
content..pick one and look at the time it got injected.
5) e.g. grep '13:27:06' access_log
6) buggar! a couple of pages-full, scroll through
7) Ah only 8 are .cgi/.pl/.php the rest are jpeg/gif/html
8) grep these 8 for any mention of the mail function
9) whew! down to 3. View the email and look for clues about
the script itself like:
"Thank you for your interest in our armpit-hair-comb-overs
weekly website. (E-Mail Removed)".
10) If we find a spam-email-to-script match, we chmod out
the script and try to contact the webmaster. If we can't
find a match, we just have to give up and move onto the
other emails.


My idea of a sendmail wrapper just to log the scripts which
call the mail function would speed up steps 3-8 above.

> I'd hurry up
> before your system is getting blacklisted for being abused as
> spambot.
Too late. many of the outgoing messages stuck in the queue
are perfectly good, genuine emails to real people from real
people, but our IPs have been blacklisted by many agencies
and ISPs and so the next server will not take our mail.

The boss is pushing for an IP migration as soon as we start
in earnest in the new year. I'd like to do what I can to
make sure that the new IP's don't immediately get blacklisted.

> In general I wouldn't recommend to run a public
> webserver on some MTA.

Unfortunately I don't have that kind of freedom.
I've only been a sys-admin for 3 months and my practical
experience with email amounted to configuring Thunderbird.

So far,
i)I have introduced the notion that HELOHOST really ought to
match the servername - That unclogged a lot of the outgoing
queues
ii)introduced magic-smtpd and written the valid user
checking program. So now each server I've implemented it on
deals with much less than 5% of the incoming mail that it
used to.

iii) stopping scripts on our servers being abused by
spammers is just my next project.

None of this is on my official list of priorities so this is
all done at home/evenings/weekends. I just love all this
stuff and am glad to get out of teaching school-kids.


--
Andy Richardson

Never criticize a man 'til you've walked a mile in his shoes.
After that, you can say what you like..
'cos you're a mile away and you've got his shoes.

In comp.os.linux.misc Andy <andy-(E-Mail Removed)>:
> Hello again Michael,
> it was your suggestions in the thread called "multifunction
> production servers" that encouraged me to get on and sort
> this spam thing out.

Sounds like a good idea, I can hardly remember the thread, so
don't bother. ;-)

>>> I'd like to know if there is any wrapper for the mail
>>> command which simply logs the calling script's name

>> How would you know it is using the mail command? There are
>> several ways to inject mail. One could even use telnet.

> I don't, but since our servers are making it so easy for
> spammers to use scripts we host, they don't need to bother
> with anything clever. Also, if I were to set aside just one
> working day to look at only one server, I could find 20+
> scripts easily which are actively being exploited by
> spammers. So I thought I'd tackle those first.

You could write a short wrapper shell script and replace the mail
command with it, which records "$@" to some file ">>" and then
hands it over to the real mail.bin which has been renamed.

However my guess is more those are pre made scripts used quite
often, (Matt's php scripts) or so, with well known
vulnerabilities which are targeted by automated attacks. Odd
thing they are likely to be much harder to defeat then dump
ratware you might get rid of in the thousands hourly on valid MTA
lately.

>> I'd start with the obvious:

>> - which scripts send mail at all?
>> - how often is each invoked?
>> - how often is local mail generated send from the user apache
>> runs as?

> I think our hard drives are mounted noatime (since they are
> in PIO mode remember) So that is one tool gone for
> invokation frequency.

Using mostly SCSI on server, dunno about PIO mode and how it
prevents you from remounting with atime enabled?

> Also some scripts are used once, but with 100-200 cc:/Bcc:
> and others are called hundreds of times with one mail
> address each time.

Doesn't matter any MTA will only use RCPT TO for tasnport, you
want to look into restricting it with sendmail to something
reasonable, you would expect for valid mail, as hot fix. Dunno
offhand how to do it with sendmail, though it is trivial with
exim.

> If we average and say 50 servers each with 1000 websites,
> (some have fewer some have many more) each site COULD
> contain .cgi/.pl/.php files which use the mail function.
> Nobody knows how many, but a
> grep -ri 'mail(' /home/*/public_html/* would give a huge list.

There is some apache module that can run scripts under each users
uid, this should ease up things.

[..]

> > I'd hurry up
>> before your system is getting blacklisted for being abused as
>> spambot.

> Too late. many of the outgoing messages stuck in the queue
> are perfectly good, genuine emails to real people from real
> people, but our IPs have been blacklisted by many agencies
> and ISPs and so the next server will not take our mail.

Glad to know.

> The boss is pushing for an IP migration as soon as we start
> in earnest in the new year. I'd like to do what I can to
> make sure that the new IP's don't immediately get blacklisted.

>> In general I wouldn't recommend to run a public
>> webserver on some MTA.

> Unfortunately I don't have that kind of freedom.
> I've only been a sys-admin for 3 months and my practical
> experience with email amounted to configuring Thunderbird.

It is certainly a network design flaw to let any machine send out
mails directly over the internet.

> So far,
> i)I have introduced the notion that HELOHOST really ought to
> match the servername - That unclogged a lot of the outgoing
> queues

And helped more spam getting through?

> ii)introduced magic-smtpd and written the valid user
> checking program. So now each server I've implemented it on
> deals with much less than 5% of the incoming mail that it
> used to.

Sounds good!

> iii) stopping scripts on our servers being abused by
> spammers is just my next project.

> None of this is on my official list of priorities so this is
> all done at home/evenings/weekends. I just love all this
> stuff and am glad to get out of teaching school-kids.

Don't get me wrong, you haven't made the situation and I welcome
your tries to get out of it.

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 137: User was distributing pornography on server;
system seized by FBI.

> There is some apache module that can run scripts under each users
> uid, this should ease up things.

Yep we are looking at suexec module at the moment

>>> I'd hurry up
>>> before your system is getting blacklisted for being abused as
>>> spambot.
>
>> Too late. many of the outgoing messages stuck in the queue
>> are perfectly good, genuine emails to real people from real
>> people, but our IPs have been blacklisted by many agencies
>> and ISPs and so the next server will not take our mail.
>
> Glad to know

So am I, if I'm honest. I just find it embarrassing that our
servers are adding to the spam problem and if customers'
valid emails aren't getting delivered maybe I'll be allowed
to dedicate some working hours to the problem.

> It is certainly a network design flaw to let any machine send out
> mails directly over the internet.

The network doesn't have a design. It appears to have
started small scale with self-contained servers with each
one dealing with its own pop3/ftp/apache/smtp/mysql/dns...
then grown very quickly by replication of live servers.

>> So far,
>> i)I have introduced the notion that HELOHOST really ought to
>> match the servername - That unclogged a lot of the outgoing
>> queues
>
> And helped more spam getting through?

Damn! How did I miss that? Customers complained that email
from our servers wasn't being accepted thru failing the
reverse DNS test. I sorted that one and completely forgot
that for every customer email that now gets through, 20 or
so spam emails now get thorough too. Sorry- I'm doing my
best :-(

> Don't get me wrong, you haven't made the situation and I welcome
> your tries to get out of it.

Thanks.. I am enjoying this a lot.. and learning loads.

--
Andy Richardson

Never criticize a man 'til you've walked a mile in his shoes.
After that, you can say what you like..
'cos you're a mile away and you've got his shoes.

On Sun, 31 Dec 2006 14:02:14 +0000, Andy wrote:

>> There is some apache module that can run scripts under each users
>> uid, this should ease up things.
>
> Yep we are looking at suexec module at the moment

You'll probably want to look at suPHP for .php scripts (especially if
you'll be moving from mod_php to a 'per user' configuration). As suEXEC
expects to handle "normal" CGI scripts (with a shebang line) and SSI. If
this all hapens to cause performance problems, have a look at FastCGI with
like mod_fcgid too.

Rather then wrap the (send)mail commands you could then just run an
ident service (like pidentd or authd) in order to log any offending users.

Since lusers aren't going to be happy should you disable their scripts
(however buggy they may be); something you might want to consider aswell
is to write some filter rules for mod_security, or something similar.

BTW: Cfengine or Puppet can automate rolling this stuff out.

(Better still, however even more effort on your part, would indeed be to
redesign the whole thing dedicating machines to specific tasks, as Michael
pointed out).

-Menno.

In comp.os.linux.misc Andy <andy-(E-Mail Removed)>:

>> There is some apache module that can run scripts under each users
>> uid, this should ease up things.

> Yep we are looking at suexec module at the moment

That is the name, it escaped me while typing and thought you'd
easily find it out.

[..]

> So am I, if I'm honest. I just find it embarrassing that our
> servers are adding to the spam problem and if customers'
> valid emails aren't getting delivered maybe I'll be allowed
> to dedicate some working hours to the problem.

You should, once your IPs are blocked from any rbl, you'll
encounter problems sooner or later.

>> It is certainly a network design flaw to let any machine send out
>> mails directly over the internet.

> The network doesn't have a design. It appears to have

A non existing design is although a "design"...

> started small scale with self-contained servers with each
> one dealing with its own pop3/ftp/apache/smtp/mysql/dns...
> then grown very quickly by replication of live servers.

Oh dear, and none had ever heard about virtual hosting...;( Guess
most systems are just idling along, wasting rack space and
power. This would be a great challenge for you, combining all
those on a very few systems.

>>> So far,
>>> i)I have introduced the notion that HELOHOST really ought to
>>> match the servername - That unclogged a lot of the outgoing
>>> queues

>> And helped more spam getting through?

> Damn! How did I miss that? Customers complained that email
> from our servers wasn't being accepted thru failing the
> reverse DNS test. I sorted that one and completely forgot
> that for every customer email that now gets through, 20 or
> so spam emails now get thorough too. Sorry- I'm doing my
> best :-(

OK, you could drop me your IPs via mail, so I could block them at
any MTA. ;-)

Good luck

[..]

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 224: Jan 9 16:41:27 huber su: 'su root' succeeded
for .... on /dev/pts/1

>> started small scale with self-contained servers with each
>> one dealing with its own pop3/ftp/apache/smtp/mysql/dns...
>> then grown very quickly by replication of live servers.
>
> Oh dear, and none had ever heard about virtual hosting...;( Guess
> most systems are just idling along, wasting rack space and
> power. This would be a great challenge for you, combining all
> those on a very few systems.

Oops, I really ought to say "self-contained servers with
each one dealing with its own pop3/ftp/apache/smtp/mysql/dns
for well over 1000 domains". Yep, we certainly have virtual
hosting sorted.


As for Menno's comments.. I think I'm going to have to read
that through very slowly.

> You'll probably want to look at suPHP for .php scripts
> (especially if you'll be moving from mod_php to a
> 'per user' configuration). As suEXEC expects to handle
> "normal" CGI scripts (with a shebang line) and SSI.
> If this all hapens to cause performance problems, have
> a look at FastCGI with like mod_fcgid too.

I hadn't realised that suEXEC is purely for tradistional CGI
scripts. Ah.. I'll carry on reading... slowly.


> Rather then wrap the (send)mail commands you could then
> just run an ident service (like pidentd or authd) in
> order to log any offending users.

I'll have to find out about these two services. I'm finding
that this wrapper idea is more hassle than I thought.
There is the
/bin/mail (binary)
/usr/sbin/sendmail -> /var/qmail/bin/sendmail (binary)
/var/qmail/bin/mailsubj (binary)
/var/qmail/bin/qmail-inject (binary)

Each method of injecting mail into the queue needs its own
wrapper since since I need the calling process info.

cat /proc/$PPID/cmdline | tr '\0' '\n'
ls -l /proc/$PPID/cwd
and maybe even
proc/$PPID/environ | tr '\0' '\n' (since some qmail methods
pick up environment variables too)

> Since lusers aren't going to be happy should you disable
> their scripts (however buggy they may be);

It seems that this has become company policy as a quick fix
to stop one cause of spam.

> something you might want to consider aswell is to write
> some filter rules for mod_security, or something similar.

mod_security?.. blank stare...
Again.. some more very slow reading involved since there is
nobody at the company I could ask, it would be up to me to
learn, convince, implement then educate. I may as well get
started then.

> BTW: Cfengine or Puppet can automate rolling this stuff
> out.
I shall google for these.

> (Better still, however even more effort on your part,
> would indeed be to redesign the whole thing dedicating
> machines to specific tasks, as Michael pointed out).

Oh, I'd love to, and that was a top priority for about 2
weeks, then it looks like it's being shelved so that we can
do an IP migration. (Hence my push for squashing the spam
generated by scripts on our servers.) Ireally don't want to
end up on SpamBlackLists withing days of the migration.

Oh well.. off to do some Googling:-
suPHP
FastCGI
mod_fcgid
pidentd
authd
mod_security
Puppet
Cfengine

Thanks all

--
Andy Richardson

Never criticize a man 'til you've walked a mile in his shoes.
After that, you can say what you like..
'cos you're a mile away and you've got his shoes.