System Freeze ... looks like a new remote exploit

Hi!

I have a linux system running as a NAT box with some tc qdiscs to
control traffic for around 100 users.
OF late, the system has started freezing... even the console does not
respond, the keyboard status lights also cannot be changed once it
freezez.

The details of the system are:

Linux Kernel Ver 2.4.26
iptables 1.2.11
named (bind 9.2.1)

I had added a drop rule to top of the iptables INPUT and PREROUTING
chains using the iptables unclean module, and found a a lot of packets
getting dropped.


Still the system freezez alomost once a day.

The logs show like this:
artian destination 0.1.0.4 from 192.168.1.46, dev eth1
martian destination 0.1.0.4 from 192.168.1.46, dev eth1
martian destination 0.1.0.4 from 192.168.1.46, dev eth1
martian destination 0.1.0.4 from 192.168.1.46, dev eth1
martian destination 0.1.0.4 from 192.168.1.46, dev eth1
martian destination 0.1.0.4 from 192.168.1.46, dev eth1
ipt_unclean: TCP flags bad: 21
IN=eth0 OUT= MAC=00:08:a1:79:93:b6:00:03:e3:8f:ed:68:08:00
SRC=85.96.153.48 DST=203.122.21.130 LEN=40 TOS=0x00 PREC=0x00 TTL=52
ID=35244 PROTO=TCP SPT=5134 DPT=1751 WINDOW=0 RES=0x00 ACK RST FIN
URGP=0
ipt_unclean: TCP flags bad: 21
ipt_unclean: (embedded packet) Embedded fragment.IN=eth0 OUT=
MAC=00:08:a1:79:93:b6:00:03:e3:8f:ed:68:08:00 SRC=213.158.98.17
DST=203.122.21.130 LEN=56 TOS=0x00 PREC=0x00 TTL=53 ID=5
651 PROTO=ICMP TYPE=3 CODE=3 [SRC=203.122.21.130 DST=213.158.98.17
LEN=34 TOS=0x00 PREC=0x00 TTL=1 ID=0 MF FRAG:1809 PROTO=UDP ]
ipt_unclean: (embedded packet) Embedded fragment.

this is the last thing before the system froze.


Any clues why this is happening?

Thanks,
Padam JS.

On 2005-04-07, (E-Mail Removed) <(E-Mail Removed)> wrote:
> Linux Kernel Ver 2.4.26

Maybe is worth trying to upgrade the kernel.
Davide

--
Q: How many Microsoft Programmers does it take to screw in a lightbulb?
A: It cannot be done. You will need to upgrade your house.
-- Geoff Johnson

padam.si...@gmail.com wrote:
> Hi!
>
> I have a linux system running as a NAT box with some tc qdiscs to
> control traffic for around 100 users.
> OF late, the system has started freezing... even the console does not
> respond, the keyboard status lights also cannot be changed once it
> freezez.
>
> The details of the system are:
>
> Linux Kernel Ver 2.4.26
> iptables 1.2.11
> named (bind 9.2.1)
>
> I had added a drop rule to top of the iptables INPUT and PREROUTING
> chains using the iptables unclean module, and found a a lot of
packets
> getting dropped.
>
>
> Still the system freezez alomost once a day.
[snip]

http://www.netfilter.org/patch-o-mat...-extra-unclean

[q]
Author: Various
Status: Dangerous

The unclean match allows you to do very detailed checks on every bit of
of the packet header. It is present in the 2.4.x kernels and was
removed
before 2.6.x was released since it is considered to be a very harmful
and
potentially future-incompatible way of doing packet filtering.

So in order to prevent stuff like the ECN blackholes from happening
again,
we moved it into patch-o-matic-ng, where lots of other potentially
harmful
code resides
[eq]

Hmmm ... perhaps you should "drop" the unclean?

hth,
prg

Hi prg,

If I remove the drop the unclean rule, the server hangs almost ten
times a day.
With the unclean rule added, it freezes only once.

I even write a libnet based program to generate absoilutely junk
packets and sent them to the server but it did not freeze! I still
can't figure out what maybe the issue...

ty,
padam.

(E-Mail Removed) wrote:
> Hi prg,
>
> If I remove the drop the unclean rule, the server hangs almost ten
> times a day.
> With the unclean rule added, it freezes only once.

I presume you are using unclean to drop packets. Correct?

> I even write a libnet based program to generate absoilutely junk
> packets and sent them to the server but it did not freeze! I still
> can't figure out what maybe the issue...

The question is whether the specific packets you note are freezing up
your machine, though it is nice to know that "junk packets" are not a
problem.

If you look at /usr/src/linux-2.4/net/ipv4/netfilter/ipt_unclean.c you
can see why the log messages are showing up and what/how "unconforming"
packets are tested.

I presume that you are allowing incoming SYN (new connections) packets.
In this case, you really should filter according to your specific
needs. You can look at examples of other scripts for ideas of what you
might need. Eg.,

http://www.linuxguruz.com/iptables/
http://www.stearns.org/modwall/sample/tcpchk-sample

Some firewall projects may be suitable. Eg.,

http://www.shorewall.net/

The logs you posted are not sufficient to really pin down what's going
on and/or why it might be freezing your machine, especially as you are
running custom qdisc configs and probably several netfilter modules.
The packets _may_ be designed to freeze your machine and force a
reboot, but my guess is that it's scanning activity/probing (by someone
that does not know what they are doing?).

Run down the IP src/dst addresses and see if they "make sense" or give
evidence that they are spoofed. Eg., the martians are _patently_
someone playing around -- the dst IP 0.1.0.4 is illegal and
192.168.1.46 would indicate someone on your private network. The
unclean packets show legal (but sensible?) IPs, but the port # 1751 is
"assigned" to Swiftnet. Swiftnet is a vpn provider for institutional
financial messaging/transaction services (not clear if 1751 is their
vpn access port). The embedded fragment packet is how UDP returns a
RST for an unused port #.

I would set up more complete logging in iptables and try to focus on
the suspicious traffic -- IPs, ports, TCP flags. You really need
timestamps for the entries to be useful. Are the packets showing at
particular times? In bursts? All with the same src/dst/ports?

To get a complete picture of what may be happening, you will have to
capture the packets and examine them. This is best done on a separate
box using a hub,eg., capable of seeing all passing traffic.

The tough part is that your machine is providing an ongoing service
which means you have to be more careful and cautious about packet
captures and trying solutions. No way to tell you how best to go about
these details.

You may also want to post to the netfilter or ethereal mailing lists if
you can't figure out what's happening or for advice. There are people
on those lists that are very sharp/experienced in these detective
efforts. Also some of the other ngs may be worth a visit, like:
comp.protocols.tcp-ip
comp.security.firewalls
comp.security.unix

This may take a week or two to sort out unless you run across someone
who has seen this traffic before. I could not find any hints that this
is part of a particular or current hack attempt. There is always a
chance that it's bad luck and a misconfigured/misbehaving router. You
also have to determine that it's not originating from "inside" your
network. Tedious work...

I'm clearing my desk for a trip, so I can't be of any more help. If I
do run across something, I'll post, otherwise...

good luck,
prg

So the update is that I upgraded the kernel to 2.6.11-7 and still the
box hangs... freeze with no sign of any kernel panic/oops. It's got to
be something else as it is highly unlikely that there is exists such a
remote attack gone untraced across a few dozen kernel versions!

It's not a hardware issue as the freeze keep coming no matter what I
use... a celeron 1.7ghz pc, to a 300mhz AMD.
With nothing more than bin 9.2.2 running and that too firewalled from
the WAN side, nothing seems to be giving a plausible answer to why this
is happening!

Anyone got any thought why this issue may be arising?

Thank you,
Padam JS.