Connection get dropped on my router (arp problem perhaps?)

I have this box running as proxy server with kernel 2.4.22 keep
dropping all the NATed connections. The setup is like this

bond0 192.168.1.2 (eth0+eth4) -> firewall which connected to a leased
line and DMZ.
eth1 unused
eth2 192.168.200.1 broadband router pool (with 6 broadband modems in
the zone)
eth3 10.3.254.254 LAN with approx 300 machines

eth1 eth2 and eth4 are using 3c95x driver
eth3 is using tg3 driver
eth0 is using pcnet32 driver

I setup the default route using the following

ip route prepend default scope global equalize \
nexthop via 192.168.200.7 dev eth2 onlink \
nexthop via 192.168.200.8 dev eth2 onlink \
nexthop via 192.168.200.12 dev eth2 onlink \
nexthop via 192.168.200.17 dev eth2 onlink \
nexthop via 192.168.200.18 dev eth2 onlink \
nexthop via 192.168.200.19 dev eth2 onlink

All the workstations get the default gateway via dhcp as 10.3.254.254.
Connections seems okay at first glance, but if any client uses any
services require a permenent connection, problems arises. Some of the
clients (in random basis as I observed, but maybe not), about 70 - 100
at a time, will get switched to the next hop from their current hop,
i.e. if its on 192.168.200.7, it will suddenly start to use
192.168.200.8, and at the same time, as I've observed lately, their
arp entry are flushed, i.e. a bunch of arp entries would just
disappear and at the same time, their default gateway will switch. It
happens about every 5 - 10 mins, and disregarding the size of the arp
table, but appears very consistent. If I look at the whole thing in
the middle of the night, the arp table will grow to about 93 - 100
entries, suddenly shrinks to 20+ entries and all clients match the
disappear entries get move to next default gatway. While in a busy
afternoon, the size of the arp table will grwo to 200+ then it shrinks
to about 50+.

I've tried a lot of things, like changing gc_stale_time, increasing
gc_thresh123, increase/decrease gc_interval... to no avail.

My boss's started asking questions... so I am quite deseperated to
solve this problem. Any pointer or suggestion would be very much
appreciated.

Thanks a million in advanced
Kenneth Ho


Kenneth Ho

One more discovery. Connections get dropped (moved to next default
gateway and arp entry removed) are those arp entries in "stale" state
in arp table at that moment.