Load Balancing using 2 NICS (same network) with (iproute2) multipath route

Greetings,

I´m trying to set up load balancing on a Linux (Trustix) box, which is acting as
a firewall + HTTP/FTP proxy (Squid) + NAT gateway for an internal network.

It has 3 network interfaces: two that connect to the external world (Internet)
and one that connects to the internal network. The external connection is
currently going through two Cisco routers, each with a 1Mbps link. The external
network interfaces are on the same network, together with the two Ciscos.

I´m currently balancing outgoing traffic using iproute2´s ip util and a
multipath route with equal weights. However, I lack the confidence that this is
indeed the proper way of doing such thing, specially because since this
configuration has been adopted, the internal network users have been
experiencing some odd network behavior. Some regular web sites just became hard
to access (usually returning 500 Internal Server Errors) and, in particular,
on-line banking sites started failing miserably.

We concluded that, perhaps, since the banking sites use secure connections (SSL)
and some kind of application validation, it might get confused if some packets
arrive with a source address of the first external NIC and others with a source
address of the second external NIC. We added static routes to those sites, but
we still get trouble. Still, I thought routes were cached and thus, all packets
flowing to a specific site would (at least for some time) flow out using the
same external NIC.

Does iproute2´s multipath route balance each packet as they come or does it work
based on connections? Additionaly, is this the proper way of doing such
balancing? Are there other alternatives, perhaps channel bonding or such?

I´m including relevant config information below (IPs slightly changed to protect
the innocent, netmasks preserved). I´d appreciate any help, suggestion or report
on the subject.

--

eth0 (external 1) -> 209.X.X.19 / 255.255.255.240
eth1 (external 2) -> 209.X.X.22 / 255.255.255.240
eth2 (internal) -> 192.168.1.1 / 255.255.255.0

Cisco 1 (1Mbps) -> 209.X.X.17 / 255.255.255.240
Cisco 2 (1Mbps) -> 209.X.X.20 / 255.255.255.240

--

root@hal ~# ip route show
209.X.X.16/28 dev eth0 scope link
209.X.X.16/28 dev eth1 proto kernel scope link src 209.X.X.22
192.168.1.1/24 dev eth2 scope link
127.0.0.0/8 dev lo scope link
default
nexthop via 200.X.X.17 dev eth0 weight 1
nexthop via 200.X.X.20 dev eth1 weight 1
root@hal ~#

--

root@hal ~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
209.X.X.192 0.0.0.0 255.255.255.240 U 0 0 0 eth0
209.X.X.192 0.0.0.0 255.255.255.240 U 0 0 0 eth1
192.168.1.1 0.0.0.0 255.255.255.0 U 0 0 0 eth2
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 209.X.X.19 0.0.0.0 UG 0 0 0 eth0
root@hal ~#

--

Here´s how I´m adding the multipath route:

# to add a second route, to the same network, through a different interface.
otherwise we get two routes using eth1.
ip route replace 209.X.X.16/28 dev eth0
#
ip route add 209.X.X.16/28 dev eth0 src 209.X.X.19 table firstlink
ip route add default via 209.X.X.17 table firstlink
ip route add 209.X.X.16/28 dev eth1 src 209.X.X.22 table secondlink
ip route add default via 209.X.X.20 table secondlink
ip rule add from 209.X.X.19 table firstlink
ip rule add from 209.X.X.22 table secondlink
ip route add default scope global nexthop via 200.X.X.17 dev eth0 weight 1
nexthop via 200.X.X.20 dev eth1 weight 1

--

And here are some relevant iptables rules:

iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j SNAT --to-source
209.X.X.19
iptables -t nat -A POSTROUTING -o eth1 -s 192.168.1.0/24 -j SNAT --to-source
209.X.X.22

--

Regards,
Christopher Markov



Christopher Markov