Lost packets - strange problem

(x-posted in linux-net mailing list)

Hi!

I'm having a very strange problem. I have already tested a *lot* of
things before asking, and I still have no clue of wha't happening.

I have 6 linux boxes acting as firewalls/routers. They are using
similar configurations and netfilter rules since 4 years ago, when I
installed the first of these. Some of them route more than 10 Mbps
between interfaces, 50000+ connections tracked with netfilter, traffic
shaping, NAT, and stuff, and they don't even blink.

BUT, two of them started giving headaches, they doesn't have the highes
usage, but they lose packets (in any interface) up to 80%, sometimes
softirqd eats all the cpu, and you cannot even connect to the boxes.
This does not happen from the very first day, and not all the time!

The NICs are mostly 3c905*(a mix of them), also some e100 and 3c940
(sk98lin). The troublesome computers have 3c905 and 3c940, but I do
not find any pattern on hardware.

Also, the error count is 0 in the internet interface of the host which
fails the most.

I tried rewriting the rules, turning off traffic shaping, changing
NICs, then changing ALL the hardware (they have some very nice and fast
hardware now). I even migrated from debian woody with 2.4.x kernels to
debian sarge with 2.6.8 kernels and the problem is still the same. I
don't really know what to do.

I suspect that this could be triggered by some internet DoS attack, but
I didn't find anything special (I have already solved the recursion
problem with DNS servers). The 6 servers receive loads of dumb attacks
all the time.

Any help would be greatly appreciated!

PS: please, CC me, as I'm not subscribed.

--
Martín Ferrari

I think you can use some tools,like tcpdump/ethereal, to grap some
packages --espesially the packages that go through the 2 servers.
Maybe you can find some info about your trouble.

Good luck!

I think you can use some tools,like tcpdump/ethereal, to grasp some
packages --espesially the packages that go through the 2 servers.
Maybe you can find some info about your trouble.

Good luck!

Uhm. I have already tried that But, when you have so much traffic
those tools aren't so helpful. I have plenty of attacks, both in the
well behaving machines and in the other two.

I'm thinking in something much more low-level. Maybe someone with
knowledge in kernel internals could give me a clue....

Do you even try using "ntop" to capture all traffic that flow throght
the firewall. It can provide many valuable information of your network
flow. Good luck