sendmail compromised - Somebody help me!

I have a redhat 9 machine on a 24/7 ADSL connection that runs apache and
sendmail. I have an FQDN on this machine. Sendmail has always worked for
the domain but not anymore as my ISP now blocks all port 25 traffic,
other than directly to their mail server. I now use sendmail to send me
system email. I use pop3 to connect to the machine locally and get my
system mail. The system can send mail via smarthost to my ISP's mail
server, mail.bellsouth.net, but cannot receive mail directly anymore,
because of the ISP blocking port 25 now.

Recently I got a "disturbing email" as follows:

---------------------------------------------------------------------

Return-Path: <(E-Mail Removed)>
Received: from ohmster.com (localhost.localdomain [127.0.0.1])
by ohmster.com (8.12.8/8.12.8) with ESMTP id j3J0bADa030038
for <(E-Mail Removed)>; Mon, 18 Apr 2005 20:37:17 -0400
Received: (from apache@localhost)
by ohmster.com (8.12.8/8.12.8/Submit) id j3J0b8wP030036
for root; Mon, 18 Apr 2005 20:37:08 -0400
Date: Mon, 18 Apr 2005 20:37:08 -0400
From: Apache <(E-Mail Removed)>
Message-Id: <(E-Mail Removed)>
To: (E-Mail Removed)
Subject: Account compromised
Status: O
X-SpamSubtract-Analysis: other: not to or cc me
X-SpamSubtract-Analysis: user moved message to inbox.

This account has been compromised, please clean it

---------------------------------------------------------------------

Uh oh...

Who would send such an email and how could they access my sendmail to do
it? apache sends mail now? How can apache actually send mail? This rather
disturbing email seems to be beneficial and is meant to help me, but how
can I "clean it" as suggested? One cannot reach the sendmail server at
port 25. I noticed this last Wednesday on the 20th of this month. I use
ssh to connect to the system but telnet is closed. I can also use webmin
but only from the local machine or from 1 NAT address on the LAN.

I looked through the apache logs to see if I could find anything, by
grepping for the word "CONNECT". I got some interesting results, here is
a sample:

---------------------------------------------------------------------
[root@ohmster httpd]# grep CONNECT *
access_log.2:218.32.241.55 - - [24/May/2004:03:03:20 -0400] "CONNECT
1.3.3.7:1337 HTTP/1.0" 405 993 "-" "-"
ohmster.com_access_logroxypool-43.undernet.org - -
[17/Apr/2005:05:39:45 -0400] "CONNECT 193.109.122.67:6668 HTTP/1.0" 405
995 "-" "pxyscand/2.0"
ohmster.com_access_logroxypool-59.undernet.org - -
[18/Apr/2005:14:06:19 -0400] "CONNECT 193.109.122.67:6668 HTTP/1.0" 405
995 "-" "pxyscand/2.0"
ohmster.com_access_logur-customers-got-spam-with-your-ip.brightmail.com
- - [19/Apr/2005:11:02:56 -0400] "CONNECT 216.250.16.96:25 HTTP/1.0" 405
995 "-" "-"
ohmster.com_access_log.2roxypool-17.undernet.org - -
[03/Apr/2005:15:01:16 -0400] "CONNECT 193.109.122.67:6668 HTTP/1.0" 405
995 "-" "pxyscand/2.0"
---------------------------------------------------------------------

Oh crud, now I have messages where "our-customers-got-spam-with-your-
ip.brightmail.com" people got spammed from my domain!

Today I got back a bunch of emails that could not be delivered for 5 days
that I had not sent, where did these emails come from? I peeked into the
maillog and found one of the cannot deliver mails to yahoo.com. I never
send any mail to yahoo.com. Peeking into maillog.1 shows tons of email
sent to yahoo.com and emails to tons of other people at other domains,
all appearing to be spam. Sob! Here is some of it (I used the word
"munge" to mangle some legitimate email addresses.):

---------------------------------------------------------------------
Apr 18 09:49:47 ohmster sendmail[28668]: j3IDnlSK028668: from=apache,
size=627, class=0, nrcpts=2, msgid=<200504181349.j3IDnlSK028668
@ohmster.com>, relay=apache@localhost
Apr 18 09:49:50 ohmster sendmail[28670]: j3IDnlDa028670: from=
<(E-Mail Removed)>, size=849, class=0, nrcpts=2, msgid=
<(E-Mail Removed)>, proto=ESMTP, daemon=MTA,
relay=localhost.localdomain [127.0.0.1]
Apr 18 09:49:50 ohmster sendmail[28668]: j3IDnlSK028668:
to=(E-Mail Removed),(E-Mail Removed), ctladdr=apache (48/48),
delay=00:00:03, xdelay=00:00:03, mailer=relay, pri=60168, relay=
[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (j3IDnlDa028670 Message
accepted for delivery)
Apr 18 09:50:04 ohmster sendmail[28672]: j3IDnlDa028670: to=
<(E-Mail Removed)>,<(E-Mail Removed) >, ctladdr=
<(E-Mail Removed)> (48/48), delay=00:00:16, xdelay=00:00:14,
mailer=relay, pri=60390, relay=mail.bellsouth.net. [205.152.59.16], dsn=
2.0.0, stat=Sent (Message received:
20050418134957.SAEV1994.imf16aec.mai... (E-Mail Removed))
Apr 18 17:03:57 ohmster sendmail[29592]: j3IL3uHY029592: from=apache,
size=40, class=0, nrcpts=1, msgid=<200504182103.j3IL3uHY029592
@ohmster.com>, relay=apache@localhost
Apr 18 17:03:58 ohmster sendmail[29597]: j3IL3vAI029597: from=apache,
size=42, class=0, nrcpts=1, msgid=<200504182103.j3IL3vAI029597
@ohmster.com>, relay=apache@localhost
Apr 18 17:04:01 ohmster sendmail[29613]: j3IL40Da029613: from=
<(E-Mail Removed)>, size=320, class=0, nrcpts=1, msgid=
<(E-Mail Removed)>, proto=ESMTP, daemon=MTA,
relay=localhost.localdomain [127.0.0.1]
Apr 18 17:04:01 ohmster sendmail[29602]: j3IL3v9b029602: from=apache,
size=40, class=0, nrcpts=1, msgid=<200504182103.j3IL3v9b029602
@ohmster.com>, relay=apache@localhost
Apr 18 17:04:02 ohmster sendmail[29597]: j3IL3vAI029597:
to=(E-Mail Removed), ctladdr=apache (48/48), delay=00:00:05,
xdelay=00:00:02, mailer=relay, pri=30042, relay=[127.0.0.1] [127.0.0.1],
dsn=2.0.0, stat=Sent (j3IL40Da029613 Message accepted for delivery)
Apr 18 17:04:03 ohmster sendmail[29606]: j3IL3xDa029606: from=
<(E-Mail Removed)>, size=316, class=0, nrcpts=1, msgid=
<(E-Mail Removed)>, proto=ESMTP, daemon=MTA,
relay=localhost.localdomain [127.0.0.1]
Apr 18 17:04:04 ohmster sendmail[29592]: j3IL3uHY029592:
to=(E-Mail Removed), ctladdr=apache (48/48), delay=00:00:08, xdelay=
00:00:05, mailer=relay, pri=30040, relay=[127.0.0.1] [127.0.0.1], dsn=
2.0.0, stat=Sent (j3IL3xDa029606 Message accepted for delivery)
Apr 18 17:04:07 ohmster sendmail[29616]: j3IL42Da029616: from=
<(E-Mail Removed)>, size=316, class=0, nrcpts=1, msgid=
<(E-Mail Removed)>, proto=ESMTP, daemon=MTA,
relay=localhost.localdomain [127.0.0.1]
Apr 18 17:04:08 ohmster sendmail[29602]: j3IL3v9b029602:
to=(E-Mail Removed), ctladdr=apache (48/48), delay=00:00:11, xdelay=
00:00:06, mailer=relay, pri=30040, relay=[127.0.0.1] [127.0.0.1], dsn=
2.0.0, stat=Sent (j3IL42Da029616 Message accepted for delivery)
Apr 18 17:04:13 ohmster sendmail[29618]: j3IL3xDa029606: to=
<(E-Mail Removed)>, ctladdr=<(E-Mail Removed)> (48/48), delay=
00:00:10, xdelay=00:00:09, mailer=relay, pri=30316,
relay=mail.bellsouth.net. [205.152.59.17], dsn=2.0.0, stat=Sent (Message
received:
20050418210405.XEPN2054.imf19aec.mai... (E-Mail Removed))
Apr 18 17:04:17 ohmster sendmail[29615]: j3IL40Da029613: to=
<(E-Mail Removed)>, ctladdr=<(E-Mail Removed)> (48/48), delay=
00:00:17, xdelay=00:00:15, mailer=relay, pri=30320,
relay=mail.bellsouth.net. [205.152.59.16], dsn=2.0.0, stat=Sent (Message
received:
20050418210405.XEPO2054.imf19aec.mai... (E-Mail Removed))
Apr 18 17:04:20 ohmster sendmail[29620]: j3IL42Da029616: to=
<(E-Mail Removed)>, ctladdr=<(E-Mail Removed)> (48/48), delay=
00:00:13, xdelay=00:00:12, mailer=relay, pri=30316,
relay=mail.bellsouth.net. [205.152.59.17], dsn=2.0.0, stat=Sent (Message
received:
20050418210411.XESB2054.imf19aec.mai... (E-Mail Removed))

---------------------------------------------------------------------

Now the (E-Mail Removed) and bellsouth.net email is legitimate,
that is one of the webmasters getting feedback from a guestbook on one of
the apache virtual hosts and myself. But this one:

---------------------------------------------------------------------
Apr 18 17:04:20 ohmster sendmail[29620]: j3IL42Da029616: to=
<(E-Mail Removed)>, ctladdr=<(E-Mail Removed)> (48/48), delay=
00:00:13, xdelay=00:00:12, mailer=relay, pri=30316,
relay=mail.bellsouth.net. [205.152.59.17], dsn=2.0.0, stat=Sent (Message
received:
20050418210411.XESB2054.imf19aec.mai... (E-Mail Removed))
---------------------------------------------------------------------

Is totally bogus, as are much of the other ones. The maillog.1 file is
7.3Mb in size! Somebody is using my sendmail server via my smarthost ISP
mail server, to spam the world at large, or at least this is what I think
is happening. How can I address this problem, I need someplace to start.
I want sendmail to send local mail for me to pickup via pop3 and not send
mail to the outside world anymore if it is spam, I do use sendmail to
mass mail the family when the family website sends mass emails alerting
of family events. I can remove the smarthost sendmail.mc line if I have
to, but I don't want the system to accept spam email at all and would
rather leave the system capable of sending world email. This system has
been very good, very stable for years, and I have never had a problem
with it.

Here is some version information:

[root@ohmster mail]# rpm -q sendmail
sendmail-8.12.8-9.90
[root@ohmster mail]# rpm -q httpd
httpd-2.0.40-21.17.legacy
[root@ohmster mail]# uname -a
Linux ohmster.com 2.4.20-31.9 #1 Tue Apr 13 18:04:23 EDT 2004 i686 i686
i386 GNU/Linux
[root@ohmster mail]#

I desperately need helpful advice, please don't yell at me and tell me
that redhat 9 is too far out of date and it is my fault for not upgrading
to a newer distro. I have years of configs, installs, tweaks and setups
on this machine and it would be difficult or impossible to bring it all
over to a new distro such as FC without losing everything for a long time
until a new distro could be brought up with everything that is on the
system now.

How can I track down these spams and close the doors to spammers at large
from using my system? Please help me, I need a logical order of things to
check and seal up. Please help and give me a place to start and how to
follow up. I need your help now. Thank you.

--
~Ohmster
ohmster at newsguy dot com

Ohmster <(E-Mail Removed)> wrote in
news:Xns96429ED4C65F0MyBigKitty@216.77.188.18:

> Subject: sendmail compromised - Somebody help me!

Please follow up on this and read:
Subject: sendmail compromised - Somebody help me!
in the newsgroup: comp.mail.sendmail

I am making progress with help in the followups. It seems to be an apache
perl exploit but I need more help. If anyone can read the followups in
comp.mail.sendmail and offer more help, this would be greatly appreciated.
Maybe this was not the right newsgroup to post in but the thread has
started and some details are coming to light. Thank you for reading, for
your input, and for your help. I don't mean to run anyone around from
newsgroup to newsgroup but comp.mail.sendmail is where the followups are
being posted and that is where the meat of this discussion is.

I hope that I did not offend anyone but I really need your help. Thank you
and bless you all.

--
~Ohmster
ohmster at newsguy dot com

In article <Xns96429ED4C65F0MyBigKitty@216.77.188.18>
Ohmster <(E-Mail Removed)> wrote:

>I have a redhat 9 machine on a 24/7 ADSL connection that runs apache and
>sendmail. I have an FQDN on this machine. Sendmail has always worked for
>the domain but not anymore as my ISP now blocks all port 25 traffic,
>other than directly to their mail server. I now use sendmail to send me
>system email. I use pop3 to connect to the machine locally and get my
>system mail. The system can send mail via smarthost to my ISP's mail
>server, mail.bellsouth.net, but cannot receive mail directly anymore,
>because of the ISP blocking port 25 now.
>
>Recently I got a "disturbing email" as follows:
>
>---------------------------------------------------------------------
>
>Return-Path: <(E-Mail Removed)>
>Received: from ohmster.com (localhost.localdomain [127.0.0.1])
> by ohmster.com (8.12.8/8.12.8) with ESMTP id j3J0bADa030038
> for <(E-Mail Removed)>; Mon, 18 Apr 2005 20:37:17 -0400
>Received: (from apache@localhost)
> by ohmster.com (8.12.8/8.12.8/Submit) id j3J0b8wP030036
> for root; Mon, 18 Apr 2005 20:37:08 -0400
>Date: Mon, 18 Apr 2005 20:37:08 -0400
>From: Apache <(E-Mail Removed)>
>Message-Id: <(E-Mail Removed)>
>To: (E-Mail Removed)
>Subject: Account compromised
>Status: O
>X-SpamSubtract-Analysis: other: not to or cc me
>X-SpamSubtract-Analysis: user moved message to inbox.
>
>This account has been compromised, please clean it
>
>---------------------------------------------------------------------
>
>Uh oh...
>
>Who would send such an email and how could they access my sendmail to do
>it? apache sends mail now? How can apache actually send mail? This rather
>disturbing email seems to be beneficial and is meant to help me, but how
>can I "clean it" as suggested? One cannot reach the sendmail server at
>port 25. I noticed this last Wednesday on the 20th of this month. I use
>ssh to connect to the system but telnet is closed. I can also use webmin
>but only from the local machine or from 1 NAT address on the LAN.

<Snip>

>How can I track down these spams and close the doors to spammers at large
>from using my system? Please help me, I need a logical order of things to
>check and seal up. Please help and give me a place to start and how to
>follow up. I need your help now. Thank you.


I would recommend, before doing anything that you do the following:

1. Download, install and run a rootkit check, or root kit hunter.

http://www.chkrootkit.org/
http://freshmeat.net/projects/rkhunter/

See if the system is compromised, if it is, unfortunately, your only way
out of it is a clean fresh install, unless you have the time to play
around with the server and install packages and libraries one by one.

2. Download and install APF.

http://www.rfxnetworks.com/apf.php

It will be able to detect brute force attacks on your server and email
you when it happens.

3. Make sure that no root or member of the root group is able to log in
through ssh, instead log in as a member of a restricted group, and su to
your super user account.

Good luck.

Adam <(E-Mail Removed)> wrote in
news:(E-Mail Removed):

> I would recommend, before doing anything that you do the following:
>
> 1. Download, install and run a rootkit check, or root kit hunter.
>
> http://www.chkrootkit.org/
> http://freshmeat.net/projects/rkhunter/

Excellent advice, yes I really need to do this and needed a good URL for
it. Thanks.

>
> See if the system is compromised, if it is, unfortunately, your only
way
> out of it is a clean fresh install, unless you have the time to play
> around with the server and install packages and libraries one by one.
>
> 2. Download and install APF.
>
> http://www.rfxnetworks.com/apf.php

This is interesing, will check into this further.

>
> It will be able to detect brute force attacks on your server and email
> you when it happens.
>
> 3. Make sure that no root or member of the root group is able to log in
> through ssh, instead log in as a member of a restricted group, and su
to
> your super user account.
>
> Good luck.

Got it. Thanks for your help, Adam.

--
~Ohmster
ohmster at newsguy dot com

In article <Xns96435BDC7B40CMyBigKitty@216.77.188.18>
Ohmster <(E-Mail Removed)> wrote:

>>
>> It will be able to detect brute force attacks on your server and email
>> you when it happens.
>>
>> 3. Make sure that no root or member of the root group is able to log in
>> through ssh, instead log in as a member of a restricted group, and su
>to
>> your super user account.
>>
>> Good luck.
>
>Got it. Thanks for your help, Adam.

Glad I could be of help.

One thing I forgot to say.

If you can afford getting a router with VPN features, I would recommend
doing that, putting your server behind it, allow only the bare minimum
ports for SMPT, POP, HTTP, and any other basic service that you need to
have connecting to the world, and make sure that ssh, FTP (unless it is
absolutely necessary) are not accessible unless you are connected
through VPN.

Also make sure that most server software, Apache for example are not
running as a privileged users, and not using the standard nobody:nobody
user and group, and for all other users, use strong, hard to guess
usernames and passwords.

It is scary how you would find usernames like test with passwords like
test, password, nothing, secret, new, ...etc.

Adam <(E-Mail Removed)> wrote in
news:(E-Mail Removed):

> Glad I could be of help.

It is very much appreciated.

>
> One thing I forgot to say.
>
> If you can afford getting a router with VPN features, I would recommend
> doing that, putting your server behind it, allow only the bare minimum
> ports for SMPT, POP, HTTP, and any other basic service that you need to
> have connecting to the world, and make sure that ssh, FTP (unless it is
> absolutely necessary) are not accessible unless you are connected
> through VPN.

I do use an FTP server. It has to be accessible for simple, family
members. I run vsftp as both standard FTP for myself with login and
password, and also as anon so that I can give my family members an FTP
URL, directly to the write only incoming directory, and they can click
it. This brings up a blank, white Explorer window on their desktop. Then
the family member will highlight several photos in Explorer and they can
then drag them on top of the empty (Cannot see anything in there, write
only.) Internet Explorer window and let go of the mouse button. Then the
photos will copy over to the anon FTP server and I get pictures to post
on the family web site that way. My mom is over 80 years old and running
a standard FTP client, let alone VPN is pretty much out of the question.
For her and for most of the family. Click on URL, blank Internet Explorer
windows opens, drag the pictures on top of the blank Internet Explorer
window, and let go of the mouse button. Poof, the pictures are here.

>
> Also make sure that most server software, Apache for example are not
> running as a privileged users, and not using the standard nobody:nobody
> user and group, and for all other users, use strong, hard to guess
> usernames and passwords.

Very careful about that, would not want apache or anything else running
privileged. Web server runs as user "apache" and group "apache, neither
of which are privileged.

>
> It is scary how you would find usernames like test with passwords like
> test, password, nothing, secret, new, ...etc.

Yeah really. Thanks for helping out!

--
~Ohmster
ohmster at newsguy dot com