Multiple Internet connections

I have a Linux router/gateway/firewall that has two Internet
connections. The main connection is symmetrical (fast upload and
download), while the other connection is asymmetrical (fast download,
slow upload).

I've got some ideas so far - I'm hoping for some comments or hints to
tell me if I'm on the right path. I haven't tried implementing any of
this as yet.


Up until now I've just been using the main connection, with the the
second connection being used with different network equipment. But I'd
like to try to make use of both connections.

The ideal setup I would like is this:

$IF1 is the main interface for most traffic.

Outgoing http traffic should be split between $IF1 and $IF2.

If $IF1 goes down, all outgoing traffic should go through $IF2 (and
similarly if $IF2 goes down, everything should go through $IF1).

For any incoming traffic, replies should go back through the same
interface as the incoming packet.



The idea is the main downstream-heavy web traffic will benefit from the
extra bandwidth of the secondary connection, while things like email
will continue to use the symmetrical main connection. And in the event
of a failure on the main line, we will still have access.



As far as I can see, I could get a simple fail-over by just making two
default routes, one for each interface but with a higher metric for
$IF2. However, that would not get me any sort of load balancing and
replies to anything coming in on $IF2 would go out on $IF1.


I've been looking at
<http://lartc.org/howto/lartc.rpdb.multiple-links.html>.

What I need, I think, is to set up two routing tables (in
/etc/iproute2/rt_tables) T1 and T2, and put $IF1 and its default route
into T1, and similarly for $IF2 in T2:

ip route add $P1_NET dev $IF1 src $IP1 table T1
ip route add default via $P1 table T1
ip route add $P2_NET dev $IF2 src $IP2 table T2
ip route add default via $P2 table T2

The new tables can be added into the main routing by:

ip rule add from $IP1 table T1
ip rule add to $IP1 table T1
ip rule add from $IP2 table T2
ip rule add to $IP2 table T2


However, now I'm a bit stuck. From lartc.org and the man page for "ip",
I can see how to set up the routing so that it will work for fail-over:

ip route add default via $P1 metric 0
ip route add default via $P2 metric 10


lartc.org also gives an example of load balancing:

ip route add default scope global nexthop via $P1 dev $IF1 \
weight 1 nexthop via $P2 dev $IF2 weight 1

However, I only want such load balancing for http traffic - I certainly
don't want have my outgoing smtp traffic on the low upstream connection!

As a general idea, I think I am looking to use iptables rules to mark
packets, and then using those marks to select the routing table. I
think I then need another table for the balanced http routing. For example:

# For forwarded packets
iptables -A PREROUTING -t mangle -p tcp --dport 80 -j MARK \
--set-mark 1
# For packets from the firewall machine, for completeness
iptables -A OUTPUT -t mangle -p tcp --dport 80 -j MARK \
--set-mark 1

ip route add default via $P1 metric 0
ip route add default via $P2 metric 10

ip route add default scope global nexthop via $P1 dev $IF1 \
weight 1 nexthop via $P2 dev $IF2 weight 1 table balanced

ip rule add fwmark 1 table balanced


Any comments, corrections, hints, or links?

Thanks,

David

David Brown <(E-Mail Removed)> wrote in
news:14adndRzA-(E-Mail Removed):

> I've been looking at
> <http://lartc.org/howto/lartc.rpdb.multiple-links.html>.

Did you also check out "policy routing"? The most understandable
documentation was written by a man whose last name was Brown. First
name might have been Martin.
--
buck

On 29/08/11 17:40, buck wrote:
> David Brown<(E-Mail Removed)> wrote in
> news:14adndRzA-(E-Mail Removed):
>
>> I've been looking at
>> <http://lartc.org/howto/lartc.rpdb.multiple-links.html>.
>
> Did you also check out "policy routing"? The most understandable
> documentation was written by a man whose last name was Brown. First
> name might have been Martin.
> --
> buck


Would that be <http://linux-ip.net/html/> ? It certainly looks like a
comprehensive document, and includes a lot of interesting stuff -
definitely worth a read. Thanks for the pointer.

David Brown <(E-Mail Removed)> wrote in
news:(E-Mail Removed):

| Would that be <http://linux-ip.net/html/> ?

Yes, but IIRC Brown authored more than is documented there and all of
his related "stuff" is well worth studying.

The problem with flakey connections is that dead gateway detection
("DGD"), even with Julian's patches, DOES NOT WORK. That's because
the gateway is alive and well but whatever is beyond it is not. I no
longer have multiple connections so I haven't kept current, but
believe that there are solutions now that are able to know when one of
multiple uplinks fails. Perhaps "high availability" or "BalanceNG"
might lead you to where you wish to be. You don't want the LARTC docs
as your primary reading because these solutions did not exist when it
was written.

When I had multiple (all terrible) uplinks (as many as 4 at one time),
the only way I was able to isolate the one that was down was to
constantly 'ping -I' a reliable remote machine via a shell script and
adjust the routing table based on the result. I considered that shell
script an abomination.
--
buck

On 30/08/11 18:54, buck wrote:
> David Brown<(E-Mail Removed)> wrote in
> news:(E-Mail Removed):
>
> | Would that be<http://linux-ip.net/html/> ?
>
> Yes, but IIRC Brown authored more than is documented there and all of
> his related "stuff" is well worth studying.
>

I still haven't had time to read that document, but it looks like it
goes into enough detail for me for now.

> The problem with flakey connections is that dead gateway detection
> ("DGD"), even with Julian's patches, DOES NOT WORK. That's because
> the gateway is alive and well but whatever is beyond it is not.

I've come across this sort of situation. In cases when we have lost
internet connection, it has mostly been further upstream at the backbone
provider - and the two ISP connections I have share the same backbone.
If that goes down, the connection is lost. The simple arrangement I
plan should work if the local connection fails (say, the ADSL modem
dies). Maybe it won't help if the local connection is intact but there
is a problem beyond that - okay, it won't fix every possible problem,
but it may fix /some/.

> I no
> longer have multiple connections so I haven't kept current, but
> believe that there are solutions now that are able to know when one of
> multiple uplinks fails. Perhaps "high availability" or "BalanceNG"
> might lead you to where you wish to be. You don't want the LARTC docs
> as your primary reading because these solutions did not exist when it
> was written.
>

That's always a problem with looking up information on the net - most of
it is out of date. Still, I've learned from reading LARTC - I'm not
looking for really high reliability stuff.

> When I had multiple (all terrible) uplinks (as many as 4 at one time),
> the only way I was able to isolate the one that was down was to
> constantly 'ping -I' a reliable remote machine via a shell script and
> adjust the routing table based on the result. I considered that shell
> script an abomination.

That doesn't sound like an abomination to me - it checks exactly what
you want. I'm not sure how you could do a full check of your routing
without sending packets across it on a regular basis.