ip_conntrac

Hi,

My servers' /proc/net/ip_conntrac table is full of entries originating
from a single computer (over 3500 entries). This is obviously a
virus/worm gone bezerk. Now my server is behaving erradically, ping
times are slow or fail, etc. I've increased the neighbour table but it
doesnt really help. I've cut off the offending computer but the entries
in the ip_conntrac table remain. How can I terminate all those
connections?

Thanks.
Tobias

In comp.os.linux.networking (E-Mail Removed) <(E-Mail Removed)>:
> Hi,

> My servers' /proc/net/ip_conntrac table is full of entries originating
> from a single computer (over 3500 entries). This is obviously a
> virus/worm gone bezerk. Now my server is behaving erradically, ping
> times are slow or fail, etc. I've increased the neighbour table but it
> doesnt really help. I've cut off the offending computer but the entries
> in the ip_conntrac table remain. How can I terminate all those
> connections?

I'd raise /proc/sys/net/ipv4/netfilter/ip_conntrack_max first if
you have enough memory, see if this helps and then just wait.

ip_conntrack_tcp_timeout_established seems 5 days per default,
might be some RFC recommendation, wouldn't change this without a
deeper look and I don't know if this would work for connections
already in the ip_conntrack table at all or just for new.

Sure one could just temporarily try it out and see what happens
or look deep into the ip_contract source from the running kernel
for more info.

Good luck
--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 218: The UPS doesn't have a battery backup.

Hi,

Thanks. ip_conntrac_max is set to 65528 so I don't think raising it
will help.
It seems mindboggling though that there is no easy solution for this. I
have disconnected the offending server so it would make sense to
terminate all connections originating from that IP. Surely there must
be an easy solution for this?
It seems to me like its a pretty serious DOS to me if someone can just
open a few thousand connections and cripple the server..... (it's not
crippled yet, but it has been in the past due to this same kind of
thing).

regards,
Tobias

Michael Heiming wrote:
> In comp.os.linux.networking (E-Mail Removed) <(E-Mail Removed)>:
> > Hi,
>
> > My servers' /proc/net/ip_conntrac table is full of entries originating
> > from a single computer (over 3500 entries). This is obviously a
> > virus/worm gone bezerk. Now my server is behaving erradically, ping
> > times are slow or fail, etc. I've increased the neighbour table but it
> > doesnt really help. I've cut off the offending computer but the entries
> > in the ip_conntrac table remain. How can I terminate all those
> > connections?
>
> I'd raise /proc/sys/net/ipv4/netfilter/ip_conntrack_max first if
> you have enough memory, see if this helps and then just wait.
>
> ip_conntrack_tcp_timeout_established seems 5 days per default,
> might be some RFC recommendation, wouldn't change this without a
> deeper look and I don't know if this would work for connections
> already in the ip_conntrack table at all or just for new.
>
> Sure one could just temporarily try it out and see what happens
> or look deep into the ip_contract source from the running kernel
> for more info.
>
> Good luck
> --
> Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
> mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
> #bofh excuse 218: The UPS doesn't have a battery backup.

In comp.os.linux.networking (E-Mail Removed) <(E-Mail Removed)>:
> Michael Heiming wrote:
>> In comp.os.linux.networking (E-Mail Removed) <(E-Mail Removed)>:

>> > My servers' /proc/net/ip_conntrac table is full of entries originating
>> > from a single computer (over 3500 entries). This is obviously a
[..]

>> I'd raise /proc/sys/net/ipv4/netfilter/ip_conntrack_max first if
>> you have enough memory, see if this helps and then just wait.

[..]

> Thanks. ip_conntrac_max is set to 65528 so I don't think raising it
> will help.

How will you know if you just don't try it out?

> It seems mindboggling though that there is no easy solution for this. I

It seems mind boggling though that you just don't try it out and
see what happens? This would be an easy solution...

I am out of this thread.

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 77: Typo in the code

In comp.os.linux.networking Michael Heiming <michael+(E-Mail Removed)>:
> In comp.os.linux.networking (E-Mail Removed) <(E-Mail Removed)>:
>> Michael Heiming wrote:
>>> In comp.os.linux.networking (E-Mail Removed) <(E-Mail Removed)>:

>>> > My servers' /proc/net/ip_conntrac table is full of entries originating
>>> > from a single computer (over 3500 entries). This is obviously a
> [..]

>>> I'd raise /proc/sys/net/ipv4/netfilter/ip_conntrack_max first if
>>> you have enough memory, see if this helps and then just wait.

> [..]

>> Thanks. ip_conntrac_max is set to 65528 so I don't think raising it
>> will help.

> How will you know if you just don't try it out?

In addition how many entries does it have at all?

>> It seems mindboggling though that there is no easy solution for this. I

> It seems mind boggling though that you just don't try it out and
> see what happens? This would be an easy solution...

> I am out of this thread.


--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 196: Me no internet, only janitor, me just
wax floors.

On 5 Aug 2006 02:33:25 -0700, "(E-Mail Removed)" <(E-Mail Removed)>
wrote:

>Hi,
>
>My servers' /proc/net/ip_conntrac table is full of entries originating
>from a single computer (over 3500 entries). This is obviously a
>virus/worm gone bezerk. Now my server is behaving erradically, ping
>times are slow or fail, etc. I've increased the neighbour table but it
>doesnt really help. I've cut off the offending computer but the entries
>in the ip_conntrac table remain. How can I terminate all those
>connections?
>
>Thanks.
>Tobias

If your cutoff of the offender is really working, the connections will
time out. Unfortunately, the timeout is INSANELY long in the default
setup (5 _days_ for TCP). AFAIK the only way to purge is to modprobe
-r the module -- if it is a module. Otherwise, next time you build,
set KERNELSRC/net/ipv4/netfilter/ip_conntrack_proto_tcp.c so that the
TCP timeout is reasonable. You decide what's reasonable. I normally
use 2 days but 60 minutes is probably plenty. IIRC, the UDP timeout
is 180 seconds.

Whoever set the default to 5 days should be committed to an asylum.
--
buck

In comp.os.linux.networking buck <(E-Mail Removed)>:
> On 5 Aug 2006 02:33:25 -0700, "(E-Mail Removed)" <(E-Mail Removed)>
> wrote:

>>Hi,

>>My servers' /proc/net/ip_conntrac table is full of entries originating
>>from a single computer (over 3500 entries). This is obviously a
>>virus/worm gone bezerk. Now my server is behaving erradically, ping
>>times are slow or fail, etc. I've increased the neighbour table but it
>>doesnt really help. I've cut off the offending computer but the entries
>>in the ip_conntrac table remain. How can I terminate all those
>>connections?

> If your cutoff of the offender is really working, the connections will
> time out. Unfortunately, the timeout is INSANELY long in the default
> setup (5 _days_ for TCP). AFAIK the only way to purge is to modprobe
> -r the module -- if it is a module.

Yep, I suspect the same, changes might only take affect for new
connections. That's why I asked the OP to try first raising the
limit to see what happens...

> Otherwise, next time you build, set
> KERNELSRC/net/ipv4/netfilter/ip_conntrack_proto_tcp.c so that
> the TCP timeout is reasonable. You decide what's reasonable.
> I normally

Why would you recompile the kernel:

-rw-r--r-- 1 root root 0 Aug 5 20:03
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established

You can change it while running.

> use 2 days but 60 minutes is probably plenty. IIRC, the UDP timeout
> is 180 seconds.

> Whoever set the default to 5 days should be committed to an asylum.

Just be aware you might drop idle ssh connections.

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 410: Electrical conduits in machine room are
melting.

Hi,

There are only about 4000 entries in the ip_conntrack, 3500 of them are
originating from this single IP. This is why I think it wont change
anything to raise the limit.




Michael Heiming wrote:
>
> >> Thanks. ip_conntrac_max is set to 65528 so I don't think raising it
> >> will help.
>
> > How will you know if you just don't try it out?
>
> In addition how many entries does it have at all?
>
> >> It seems mindboggling though that there is no easy solution for this. I
>
> > It seems mind boggling though that you just don't try it out and
> > see what happens? This would be an easy solution...

memory is limited on this server. also I'm looking for a tool to remove
entries from the ip_conntrac, as I would like to build a script to
purge the connections if a client has been disconencted as I think it's
somehow cleaner. If a client is no longer connected then why should my
server keep connections 'open'.? What if a virus on a client computer
opens not 3500 connections but 100.000 connections? should I just keep
raising the limit?
I'd rather cut the client off and purge the connections... this could
be done in a script automagically....

regards,
Tobias

Hi,

> Why would you recompile the kernel:
>
> -rw-r--r-- 1 root root 0 Aug 5 20:03
> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established

I don't have this file. I'm using RH 9 there is no
/proc/sys/net/ipv4/netfilter directory. the ip_conntrac file is in the
/prov/sys/net/ipv4 directory for example. I couldn't find the file
elsewhere.
Do you know if there is another way to do it on this system without
recompiling?

regards,
Tobias

On 6 Aug 2006 02:52:52 -0700, "(E-Mail Removed)" <(E-Mail Removed)> wrote:

>Hi,
>
>> Why would you recompile the kernel:
>>
>> -rw-r--r-- 1 root root 0 Aug 5 20:03
>> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
>
>I don't have this file. I'm using RH 9 there is no
>/proc/sys/net/ipv4/netfilter directory. the ip_conntrac file is in the
>/prov/sys/net/ipv4 directory for example. I couldn't find the file
>elsewhere.
>Do you know if there is another way to do it on this system without
>recompiling?

Yes, unplug rh9 box from the 'net That's old stuff, no?

grant@deltree:~$ ls /proc/sys/net/ipv4/netfilter/
ip_conntrack_buckets ip_conntrack_tcp_timeout_fin_wait
ip_conntrack_generic_timeout ip_conntrack_tcp_timeout_last_ack
ip_conntrack_icmp_timeout ip_conntrack_tcp_timeout_syn_recv
ip_conntrack_max ip_conntrack_tcp_timeout_syn_sent
ip_conntrack_tcp_timeout_close ip_conntrack_tcp_timeout_time_wait
ip_conntrack_tcp_timeout_close_wait ip_conntrack_udp_timeout
ip_conntrack_tcp_timeout_established ip_conntrack_udp_timeout_stream

grant@deltree:~$ uname -r
2.4.33-rc3a

<http://bugsplatter.mine.nu/test/boxen/deltree/> for comment stripped
..config and dmesg, info about the box.

Grant.
--
<http://bugsplatter.mine.nu/>