System intrusion detection, primarily on linux servers with a handfulof others

Okay, we've got some serious issues going on here at work. We've lost
the dude that knew everything about how our network was set up; myself
and my cow orker are working full time on recovering all of the
information that we can and now we've got the added task of hardening
our systems from outside attack.

Right now I need to know if any of you *NIX networking ppl and/or
sysadmins can possibly recommend an intrusion detection system/suite.
I've heard that 'snort' is the de facto standard. We're administering
a handful of windows machines, a load of distributed Linux servers
over a WAN with 3 gateways to the outside world + a T1, 2 Linux
asterisk PBX servers, and some laptops that occasionally VPN in. I'd
at the very least like to be able to monitor all of our linux systems
in a centralized manner.

To this point I've been utilizing nagios, mrtg, and ntop. As our
potential cracker realizes how we've identified him in bandwidth
spikes, identifying compromised binaries by modification dates, and
top on any potentially compromised machines to watch for processes
spawned.

I'm adding syslogging of login attempts, etc, but the dude that got
into our WAN is sophisticated. He'll be covering his tracks better in
no time. And honestly, keeping a tail -f /var/log/messages open on
each server and my eyeballs glued to bandwidth graphs updating at 5
minute intervals will not be an option much longer.

Anybody got any ideas and/or need more information to be able to give
some helpful hints?

TIA

On Wed, 14 Jan 2009 20:05:08 -0800, Damon Getsman wrote:

> Okay, we've got some serious issues going on here at work. We've lost
> the dude that knew everything about how our network was set up; myself
> and my cow orker are working full time on recovering all of the
> information that we can and now we've got the added task of hardening
> our systems from outside attack.

I would look firstly at your firewall. Remove EVERYTHING that is not
strickly necessary and fire the first arsehole that tries to change it.

Unless, it was done from inside, they must have come in through the
firewall. Most likely they know a password, or through known
vulnerabilities through all that junk that IT like to reward
themselves with (ICQ, skype, etc, etc).

Then you put a monitoring application on the firewall to see what probes
it is receiving from outside.

Which one depends on your skill and support contracts.




> I'd at the very
> least like to be able to monitor all of our linux systems in a
> centralized manner.

The monitoring you would most likely do is uptime/services related
related.

You have a major security inadequacy if your external cracker has been
able to reach individual systems inside your network.



>
> To this point I've been utilizing nagios, mrtg, and ntop. As our
> potential cracker realizes how we've identified him in bandwidth spikes,
> identifying compromised binaries by modification dates, and top on any
> potentially compromised machines to watch for processes spawned.

Do not recover the old system from backup tapes unless you adsolutely have
to. The real problem is you will not know when they first compromised your
system.

It may take longer, but you are far better to reinstall the system from
installation media, and then re-install/reconfigure the applications. You
are going to have to learn it anyway if you are going to support it.


>
> I'm adding syslogging of login attempts, etc, but the dude that got into
> our WAN is sophisticated.

how is he getting in?
That is where you should be lookng first and fixing it.

> He'll be covering his tracks better in no
> time. And honestly, keeping a tail -f /var/log/messages open on each
> server and my eyeballs glued to bandwidth graphs updating at 5 minute
> intervals will not be an option much longer.

Totally useless.
I'm curious as to what extra bandwidth indicated?
Are they trying to steal your data?

On Jan 14, 11:29*pm, terryc <newssevenspam-s...@woa.com.au> wrote:
> I would look firstly at your firewall. Remove EVERYTHING that is not
> strickly necessary and fire the first arsehole that tries to change it.

That's precisely what we're starting with at the moment. It's been a
little bit difficult to convince everyone of that as we've [very]
recently had a change of management around here, and we're working
with another company providing a service, but we're starting to
implement these very changes at the moment.

> Unless, it was done from inside, they must have come in through the
> firewall. Most likely they know a password, or through known
> vulnerabilities through all that junk that IT like to reward
> themselves with (ICQ, skype, etc, etc).

I'm pretty sure that the vulnerability was a 'webmin' package that
shouldn't have even been installed on a system that myself and my
colleage were denied access to until this point. We're now whittling
down the services on the new version of this machine that we're
installing to make sure that any services like this are gone. It took
a lot of convincing with the other people associated with the machine
that provided the gateway in this instance to get them to trim down
their distributions, but they're complying fully with our security
recommendations and, in fact, having us implement them, from this
point forward.

> Then you put a monitoring application on the firewall to see what probes
> it is receiving from outside.
>
> Which one depends on your skill and support contracts.

Looks like we're going to be going with snort right now, and seeing
what that gives us at first.

> > I'd at the very
> > least like to be able to monitor all of our linux systems in a
> > centralized manner.
>
> The monitoring you would most likely do is uptime/services related
> related.
>
> You have a major security inadequacy if your external cracker has been
> able to reach individual systems inside your network.

I was worried that this happened at first, but it's beginning to look
like maybe we got lucky about that. However our DMZ that the
'gateway' machine should have been in was not correctly implemented
and we're locking that down as I type.

> how is he getting in?
> That is where you should be lookng first and fixing it.

Yep, that's what we're concentrating on at this point.

> Totally useless.
> I'm curious as to what extra bandwidth indicated?

Our 'gateway' machine that was definitely compromised showed at the
point of 'recruitment' of this server that our bandwidth spiked up to
around 3-4 times our normal PEAK bandwidth usage on that machine. It
is normally used for a high bandwidth service that only consumes a
small fraction, even at highest consumption, of what it began to use
at the minute that we detected his actions. We have logs of bandwidth
going back almost a year so we were able to identify this as
suspicious activity immediately. Obviously relying on logs and
bandwidth is not enough, as you said, and that's why (now that the
person that was previously holding us back is no longer a problem)
we're now implementing much more comprehensive strategies.

Not sure if any of our data was compromised. It looks a lot more like
whoever was trying to get in was looking for a machine with an assload
of bandwidth and minimal CPU/RAM requirements. It was a linux
machine, though, and they were using modified system binaries to
establish back doors, so I'm thinking it was something probably more
malicious than just a spambot. Multiple connections were established
(presumably generating the bandwidth spikes) to different IRC
servers. Botnet is my guess.

-Damon

On 01/14/2009 08:05 PM, Damon Getsman sent:

* * * Cross Posted * * *

Snip, snip...

Hello Damon:

Please consider crossposting all future replies to this thread so we
aren't burdened with duplicitous posts at cross purposes.

<http://en.wikipedia.org/wiki/Crossposting>

I know that once you send a post, another newsgroup comes to mind. All
of us have done it.

Looking forward to your reply,

Pete
--
1PW @?6A62?FEH9E=6o2@=]4@> [r4o7t]

> NOTE: Posting from groups.google.com (or some web-forums) dramatically
> reduces the chance of your post being seen. *Find a real news server.

Thank you for this information; however, the ISPs that we are using
for the gateway that I'm forced to use at this point prevent me from
having access to a decent raw nntp server AFAIK. Once I'm not swamped
with other things I'll check this out.

> NOTE: Please don't multipost the same message to multiple groups. If
> you MUST post to multiple groups, include them all in the Newsgroups:
> header, and set a Followup-To: header as I have done here.

I normally do. It was a mistake on my part.

> Are you indicating that someone already _HAS_ gotten in? *If so, you
> are screwed - as you don't know what that person has done. You mention
> identifying compromised binaries - that implies the person has root -
> and think you can identify things by modification date. *Boy, are you
> ever wrong:

I am aware of that, and yes, I'm pretty sure that he did get root.
The modification stamps on /bin/bash and /bin/grep didn't get there by
anyone else and I didn't do it personally. It does tell a little bit
about the sophistication level of the person that I'm dealing with,
though.

I am fully aware of the ability to modify datestamps by 'touch' and
hex filesystem access on a low level. Obviously it either indicates
that the person thought we were incompetent (or did not have enough
resources available as a target) enough to not bother with the time
for that, or they do not posses the level of sophistication to
implement that.

> And the bad guy is getting in exactly how? * You're posting from an
> address of a company in the medical business. If there is even the
> remote possibility that this is subject to HIPAA, disconnect the network
> from the Internet RIGHT NOW.

Almost certainly looks to be from an apache or webmin hole. I
attempted to close these things a long time ago and was hamstrung by
someone over my head that has been dealt with for this already. I'm
making sure that I don't make the same mistakes again.

> as it's a waste of time if the bad guy has gotten in once already. How
> do you know that the tools you are using haven't been 0wn3d already?

I know, that's why I'm looking for something better on the next way
around.

> You're posting from what is indicating Ubunto 8.04. *If that is
> representative of your Linux, have you been keeping things up to date?
> If so, the bad guy may be getting in through a compromised account.
> Which one?

We haven't been (AGAINST my advice, which is why the aforementioned
people are being 'dealt with'), and if he's got a brain he's snagged
our password and shadow files, so we're considering everything
compromised. I've dealt with network security from the other side of
the spectrum when I was an angsty teenager and learned how
exploitations occur. So I may not have the knowledge to deal with
this on the administration side just yet, but I do know what I'm
looking for, what he's capable of, and have an idea of what we're
playing with.

> In the Usenet newsgroup comp.os.linux.security, Bit Twister has
> suggested aide and samhain (and ossec which I have no experience with).
> A problem with these tools is that they are best used starting with a
> "clean" system, and allowing them to report on changes. *In both Usenet
> newsgroups, others have recommended a reinstall. I TOTALLY agree with
> that recommendation. If you believe you've been compromised, the _only_
> solution is a wipe and reinstall, followed by a full update before
> allowing access from the world.

Thank you for the advice. I'll look into the tools. I'm doing what I
can to advocate what you've mentioned, but I might be hamstrung from
higher up in being able to do a full cleaning.

> Hmmm, I wonder if your SSH setup is one of the compromised ones. From
> the US-CERT Technical Cyber Security Alert TA08-137A from last May:

On the old machine it certainly was. I was aware of this ssl
vulnerability but didn't think of it because I didn't know about
Asterisk's usage of SSL. I'll look into it more now. As soon as that
issue was mentioned I made sure to update all of our servers to non-
vulnerable implementation and keyset.

Thank you much, I appreciate the time and consideration that you put
into your message a lot. I don't have a whole lot of resources to
work with right now and a pointing in the right direction saves me a
lot of time and enables me to do what I can over the weekend when all
of our users are off of the main servers, so quick action is a
necessity.

-D

Gotcha, and I thank you much for bringing it to my attention. That's
exactly what happened, I'll try to think a little bit more before I
smack send next time.

On Jan 15, 2:49*pm, 1PW <barcrnahgjuvf...@nby.pbz> wrote:

> Please consider crossposting all future replies to this thread so we
> aren't burdened with duplicitous posts at cross purposes.

Hello Damon:

Exactly which Linux OSs are you supporting on these servers?
Distro, version...


--
1PW @?6A62?FEH9E=6o2@=]4@> [r4o7t]

Ubuntu 8.04, OpenSuSE 10.4, and then I'm working on getting us back onto
a few Solaris systems right now.

Ubuntu 8.04 is definitely our primary; the only different version of
Ubuntu that we have is on a few of our IT personnel personal
workstations, in which case we're using 8.10.

In comp.os.linux.security 1PW <(E-Mail Removed)> wrote:
>
> Hello Damon:
>
> Exactly which Linux OSs are you supporting on these servers?
> Distro, version...
>
>

In comp.os.linux.security Moe Trin <(E-Mail Removed)> wrote:
> NOTE: Posting from groups.google.com (or some web-forums) dramatically
> reduces the chance of your post being seen. Find a real news server.

Obviously I've gotten to that step now, something I had been neglecting
previously. Thanks.

>>> Are you indicating that someone already _HAS_ gotten in? If so, you
>>> are screwed - as you don't know what that person has done.
>
>>I am aware of that, and yes, I'm pretty sure that he did get root.

At this stage I know that he got root on one of our externally facing
machines. This machine has been wiped and reinstalled but is still not
up to the standards that I would like it to be at. It also had an
internal interface which was not securely firewalled or DMZed. Yes, I
understand the horror of that situation, but it was out of my control.
Thanks to that issue I now have the ability to make suggestions that
will be taken much more seriously regarding our handling of network
security in the future. This does not help the fact that I'm still
learning this as I go, however, at least at this level of
sophistication.

> The only safe solution is to wipe and reinstall from known clean media.
> But you know that too.

As a starting point I have wiped and reinstalled my own workstation; I
am not totally sure that I had it locked down before I reconnected it to
the WAN, however, so I may just end up doing that again. I am also
using a startpoint of my own personal laptop which, at this point, is
disconnected from the WAN and I am fairly certain is not compromised at
the moment. It is running a significantly different kernel from the
other machines, as well.

>>Almost certainly looks to be from an apache or webmin hole.
>
> Yeah, there are plenty of those. Two points: PHP is generally a walking
> disaster area - looking at Bugtraq shows more PHP exploits than
> anything else, followed fairly closely by SQL.

Thank you. As we get downtime where our users are not needing vital
network services I will make sure that we concentrate on these areas and
not providing them where they aren't needed, and maintaining the best I
can find to lock them down where they absolutely are needed.

> Second point, if one of your users has been 0wn3d _elsewhere_ and they
> had their SSH authentication tokens on "that" system, there is an
> exploit out there that uses stolen authentication to get "in" to your
> system, uses any number of local exploits to escalate privilege to
> root, and then installs the 'Phalanx2' root kit that hides itself
> pretty well, and searches for authentication data on your system (which
> it mails out to a drop-box). CERT reported this about five months ago.

Yep. I remember similar .rhost exploits and disasters surrounding them
over a decade ago.

> The only safe solution is the wipe/reinstall/update. There are a
> number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
> (http://www.chkrootkit.org), 'rkhunter' (http://www.rootkit.nl), and
> from a cursory look, 'ossec' (http://www.ossec.net/) that can be used
> to look for many rootkits. By in large, these are trivial to defeat or
> bypass. As one example, all look for the existence of a file named
> '/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
> sniffer worm from 2003). That's all well and good, but what if the
> mal-ware author does something terribly complicated, such as changing
> the filename to '/tmp/.../A' or '/tmp/.../b' or something... yup, the
> anti-mal-ware tools won't find it.

I've actually been able to start going through some OSSEC logs at this
point that are showing things of this sort. Of course, I do not know
enough about the specifics of OSSEC administration just yet to be able
to eliminate all of the false positives, however there are a few entries
in them that are leading me to believe that we have significant
intrusion on some if not all of our primary internal servers within the
WAN. IE:

OSSEC HIDS Notification.
2009 Jan 20 15:29:47

Received From: (agentname) 192.168.1.NOU->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event
(rootcheck)."
Portion of the log(s):

Port '41861'(tcp) hidden. Kernel-level rootkit or trojaned version of
netstat.



--END OF NOTIFICATION


> Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
> strong point, but I'm much less thrilled by it now.

I'm also working on getting these alternate tools installed on a
completely clean system right now to get some decent monitoring in
place.

> Many of us suffer through the same interference. Best advice remains
> a wipe, reinstall from trusted media, and updates. In this case, I'd
> also strongly recommend new passwords for ALL accounts. Then, take a
> snapshot of the system using 'aide' (or the fore-runner 'tripwire').
> This is what the system looks like "now" - and you can then compare
> this snapshot to "later" and look for changes. Big caution: the
> aide binary and database go on removable media that is kept in a safe
> place when not being used. Also, you have to retake the snapshot each
> time you intentionally change something (such as security updates). It
> is a pain in the a$$, but less of a pain than what you are going
> through now.

Gotcha. Your recommendations have given me an excellent starting point
and I thank you much. Any further feedback is greatly appreciated.

-Damon Getsman

On 01/21/2009 07:11 AM, (E-Mail Removed) sent:
> In comp.os.linux.security Moe Trin <(E-Mail Removed)> wrote:
>> NOTE: Posting from groups.google.com (or some web-forums) dramatically
>> reduces the chance of your post being seen. Find a real news server.
>
> Obviously I've gotten to that step now, something I had been neglecting
> previously. Thanks.
>
>>>> Are you indicating that someone already _HAS_ gotten in? If so, you
>>>> are screwed - as you don't know what that person has done.
>>> I am aware of that, and yes, I'm pretty sure that he did get root.
>
> At this stage I know that he got root on one of our externally facing
> machines. This machine has been wiped and reinstalled but is still not
> up to the standards that I would like it to be at. It also had an
> internal interface which was not securely firewalled or DMZed. Yes, I
> understand the horror of that situation, but it was out of my control.
> Thanks to that issue I now have the ability to make suggestions that
> will be taken much more seriously regarding our handling of network
> security in the future. This does not help the fact that I'm still
> learning this as I go, however, at least at this level of
> sophistication.
>
>> The only safe solution is to wipe and reinstall from known clean media.
>> But you know that too.
>
> As a starting point I have wiped and reinstalled my own workstation; I
> am not totally sure that I had it locked down before I reconnected it to
> the WAN, however, so I may just end up doing that again. I am also
> using a startpoint of my own personal laptop which, at this point, is
> disconnected from the WAN and I am fairly certain is not compromised at
> the moment. It is running a significantly different kernel from the
> other machines, as well.
>
>>> Almost certainly looks to be from an apache or webmin hole.
>> Yeah, there are plenty of those. Two points: PHP is generally a walking
>> disaster area - looking at Bugtraq shows more PHP exploits than
>> anything else, followed fairly closely by SQL.
>
> Thank you. As we get downtime where our users are not needing vital
> network services I will make sure that we concentrate on these areas and
> not providing them where they aren't needed, and maintaining the best I
> can find to lock them down where they absolutely are needed.
>
>> Second point, if one of your users has been 0wn3d _elsewhere_ and they
>> had their SSH authentication tokens on "that" system, there is an
>> exploit out there that uses stolen authentication to get "in" to your
>> system, uses any number of local exploits to escalate privilege to
>> root, and then installs the 'Phalanx2' root kit that hides itself
>> pretty well, and searches for authentication data on your system (which
>> it mails out to a drop-box). CERT reported this about five months ago.
>
> Yep. I remember similar .rhost exploits and disasters surrounding them
> over a decade ago.
>
>> The only safe solution is the wipe/reinstall/update. There are a
>> number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
>> (http://www.chkrootkit.org), 'rkhunter' (http://www.rootkit.nl), and
>> from a cursory look, 'ossec' (http://www.ossec.net/) that can be used
>> to look for many rootkits. By in large, these are trivial to defeat or
>> bypass. As one example, all look for the existence of a file named
>> '/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
>> sniffer worm from 2003). That's all well and good, but what if the
>> mal-ware author does something terribly complicated, such as changing
>> the filename to '/tmp/.../A' or '/tmp/.../b' or something... yup, the
>> anti-mal-ware tools won't find it.
>
> I've actually been able to start going through some OSSEC logs at this
> point that are showing things of this sort. Of course, I do not know
> enough about the specifics of OSSEC administration just yet to be able
> to eliminate all of the false positives, however there are a few entries
> in them that are leading me to believe that we have significant
> intrusion on some if not all of our primary internal servers within the
> WAN. IE:
>
> OSSEC HIDS Notification.
> 2009 Jan 20 15:29:47
>
> Received From: (agentname) 192.168.1.NOU->rootcheck
> Rule: 510 fired (level 7) -> "Host-based anomaly detection event
> (rootcheck)."
> Portion of the log(s):
>
> Port '41861'(tcp) hidden. Kernel-level rootkit or trojaned version of
> netstat.
>
>
>
> --END OF NOTIFICATION
>
>
>> Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
>> strong point, but I'm much less thrilled by it now.
>
> I'm also working on getting these alternate tools installed on a
> completely clean system right now to get some decent monitoring in
> place.
>
>> Many of us suffer through the same interference. Best advice remains
>> a wipe, reinstall from trusted media, and updates. In this case, I'd
>> also strongly recommend new passwords for ALL accounts. Then, take a
>> snapshot of the system using 'aide' (or the fore-runner 'tripwire').
>> This is what the system looks like "now" - and you can then compare
>> this snapshot to "later" and look for changes. Big caution: the
>> aide binary and database go on removable media that is kept in a safe
>> place when not being used. Also, you have to retake the snapshot each
>> time you intentionally change something (such as security updates). It
>> is a pain in the a$$, but less of a pain than what you are going
>> through now.
>
> Gotcha. Your recommendations have given me an excellent starting point
> and I thank you much. Any further feedback is greatly appreciated.
>
> -Damon Getsman

Hello Damon:

I know you are still trying to "drain the swamp" there. At some point,
I hope you, and those you work for, are able to consider using a fairly
recent enterprise solution such as RHEL5/CentOS5. The addition of
SELinux can help you harden your systems as few other schemes can.

I know that integrating SELinux with Hardy Heron has been done, but it's
a significant undertaking, whereas SELinux has been part of
RHEL/CentOS/Fedora/Debian for awhile.

Below is a URL that points to many guides to system hardening:

<http://www.nsa.gov/ia/guidance/security_configuration_guides/current_guides.shtml>

In the list, you will find hardening tips for Solaris systems.

I immediately realize that the following does not *directly* apply to
*any* of your systems. However, I believe that some of these steps may
still have value for you:

<hXXp://www.nsa.gov/ia/_files/os/redhat/rhel5-pamphlet-i731.pdf>

<hXXp://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf>

I obfuscated the URLs for your security protection.

I'm hoping that some of this will give you food for thought. Good luck.

Pete
--
1PW @?6A62?FEH9E=6o2@=]4@> [r4o7t]