Linux multihomed routing (2 ISP, 1 internal network) problem

Hi all !!

I'm having a little trouble with a multihomed setup I am running here.
I'll try to give as much information as possible.

I am using the latest Debian release with kernel 2.6.18-4-686.
I am doing the routing using iproute2. the rules are setup with
iptables.
The system has 3 NICs: eth0 has the cable ISP (called "TELENET"), eth1
has the ADSL ISP (called "SKYNET") (with external modem) and eth2 is
my internal network (called "INTERN"). The ADSL line has a fixed IP,
and is used to connect to one of our servers from remote locations
(the modem is setup to forward everything that hits it to my IP on
eth1)
The cable ISP is the preferred ISP (ie: all undefined outgoing traffic
goes thru the cable provider).
All clients on the lan have full access, no restrictions, to do
whatever they want online.
I am using ip route rules and iptables with --set-mark to tell certain
traffic which routing table to use.

When I finished configuring Saturday evening, everything was working
like a charm.
When I tested again today, suddenly it didn't work as good anymore ...
Something has changed overnight but I have no clue what it is.
After some testing, I figured out 2 things: my dns traffic suddenly
wants to go thru the DSL line and my clients can not directly connect
to the internet anymore. After setting a clients dns servers to the
DSL ISP's dns servers, they could resolve again, but they can still
not surf anymore. The Linux machine itself can still perfectly do
everything.

I hope somebody can help me out here, because I am at a loss; I've
once been able to set this up about 6 years ago, but that knowledge
has faded a bit (and isn't apparantly very useful anymore anyways). I
am also not a linux noob ;-)

Below you can find all technical stuff.

Thanks to all of you in advance for your help !!

Cheers,
Andy


------------ MAIN ROUTING TABLE -----------
# ip route show table main
192.168.0.0/24 dev eth2 proto kernel scope link src 192.168.0.254
192.168.254.0/24 dev eth1 proto kernel scope link src 192.168.254.2
81.82.0.0/19 dev eth0 proto kernel scope link src 81.82.x.x
default via 81.82.0.1 dev eth0

------------ EXTRA ROUTING TABLE-----------
# ip route show table 4
192.168.0.0/24 dev eth2 proto kernel scope link src 192.168.0.254
192.168.254.0/24 dev eth1 proto kernel scope link src 192.168.254.2
81.82.0.0/19 dev eth0 proto kernel scope link src 81.82.x.x
default via 192.168.254.1 dev eth1

----------- ROUTING RULES -----------
# ip rule show
0: from all lookup 255
32764: from 192.168.254.1 lookup 4
32765: from all fwmark 0x4 lookup 4
32766: from all lookup main
32767: from all lookup default

---------- FIREWALL (rules) SCRIPT (partial) ----------

IPTABLES=/sbin/iptables
TELENET="eth0"
SKYNET="eth1"
INTERN="eth2"
INTNET="192.168.0.0/24"
$IPTABLES -F
$IPTABLES -F -t nat
$IPTABLES -F -t mangle
$IPTABLES -A INPUT -i lo -s 127.0.0.1/8 -d 0.0.0.0/0 -j ACCEPT
$IPTABLES -A OUTPUT -o lo -s 127.0.0.1/8 -d 0.0.0.0/0 -j ACCEPT
$IPTABLES -A INPUT -i $TELENET -s 0.0.0.0/0 -d $TELENETIP -m state --
state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A INPUT -i $SKYNET -s 0.0.0.0/0 -d $SKYNETIP -m state --
state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A OUTPUT -o $TELENET -s $TELENETIP -d 0.0.0.0/0 -j ACCEPT
$IPTABLES -A OUTPUT -o $SKYNET -s $SKYNETIP -d 0.0.0.0/0 -j ACCEPT
$IPTABLES -A INPUT -i $INTERN -s $INTNET -d 0.0.0.0/0 -j ACCEPT
$IPTABLES -A OUTPUT -o $INTERN -s $INTERNIP -d $INTNET -j ACCEPT
$IPTABLES -t mangle -A PREROUTING -s $SERVER1IP -p tcp -m tcp --sport
443 -j MARK --set-mark 0x4
$IPTABLES -t mangle -A PREROUTING -s $SERVER1IP -p tcp -m tcp --sport
444 -j MARK --set-mark 0x4
$IPTABLES -t mangle -A PREROUTING -s $SERVER1IP -p tcp -m tcp --sport
1723 -j MARK --set-mark 0x4
$IPTABLES -t mangle -A PREROUTING -s $SERVER1IP -p tcp -m tcp --sport
4125 -j MARK --set-mark 0x4
$IPTABLES -t nat -A PREROUTING -d $SKYNETIP -p tcp -m tcp --dport 443 -
m state --state NEW,RELATED,ESTABLISHED -j DNAT --to-destination
$SERVER1IP:443
$IPTABLES -t nat -A PREROUTING -d $SKYNETIP -p tcp -m tcp --dport 444 -
m state --state NEW,RELATED,ESTABLISHED -j DNAT --to-destination
$SERVER1IP:444
$IPTABLES -t nat -A PREROUTING -d $SKYNETIP -p tcp -m tcp --dport 1723
-m state --state NEW,RELATED,ESTABLISHED -j DNAT --to-destination
$SERVER1IP:1723
$IPTABLES -t nat -A PREROUTING -d $SKYNETIP -p tcp -m tcp --dport 4125
-m state --state NEW,RELATED,ESTABLISHED -j DNAT --to-destination
$SERVER1IP:4125
$IPTABLES -t nat -A POSTROUTING -o $TELENET -j SNAT --to-source
$TELENETIP
$IPTABLES -t nat -A POSTROUTING -o $SKYNET -j SNAT --to-source
$SKYNETIP
$IPTABLES -A INPUT -d $SKYNETIP -i $SKYNET -p tcp -m tcp --sport
1024:65535 --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
$IPTABLES -A INPUT -d $SKYNETIP -i $SKYNET -p tcp -m tcp --sport
1024:65535 --dport 444 -m state --state NEW,ESTABLISHED -j ACCEPT
$IPTABLES -A INPUT -d $SKYNETIP -i $SKYNET -p tcp -m tcp --sport
1024:65535 --dport 1723 -m state --state NEW,ESTABLISHED -j ACCEPT
$IPTABLES -A INPUT -d $SKYNETIP -i $SKYNET -p tcp -m tcp --sport
1024:65535 --dport 4125 -m state --state NEW,ESTABLISHED -j ACCEPT
$IPTABLES -A FORWARD -d $INTNET -j ACCEPT
$IPTABLES -A FORWARD -s $INTNET -j ACCEPT
$IPTABLES -A FORWARD -i $SKYNET -o $INTERN -p tcp -m tcp --dport 443 -
m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -A FORWARD -i $SKYNET -o $INTERN -p tcp -m tcp --dport 444 -
m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -A FORWARD -i $SKYNET -o $INTERN -p tcp -m tcp --dport 1723 -
m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -A FORWARD -i $SKYNET -o $INTERN -p tcp -m tcp --dport 4125 -
m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -P FORWARD DROP
$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT DROP

(E-Mail Removed) wrote:
> Hi all !!

> I'm having a little trouble with a multihomed setup I am running here.
> I'll try to give as much information as possible.

> I am using the latest Debian release with kernel 2.6.18-4-686.
> I am doing the routing using iproute2. the rules are setup with
> iptables.
> The system has 3 NICs: eth0 has the cable ISP (called "TELENET"), eth1
> has the ADSL ISP (called "SKYNET") (with external modem) and eth2 is
> my internal network (called "INTERN"). The ADSL line has a fixed IP,
> and is used to connect to one of our servers from remote locations
> (the modem is setup to forward everything that hits it to my IP on
> eth1)
> The cable ISP is the preferred ISP (ie: all undefined outgoing traffic
> goes thru the cable provider).
> All clients on the lan have full access, no restrictions, to do
> whatever they want online.
> I am using ip route rules and iptables with --set-mark to tell certain
> traffic which routing table to use.

> When I finished configuring Saturday evening, everything was working
> like a charm. When I tested again today, suddenly it didn't work
> as good anymore ...

> Something has changed overnight but I have no clue what it is.
> After some testing, I figured out 2 things: my dns traffic suddenly
> wants to go thru the DSL line and my clients can not directly connect
> to the internet anymore. After setting a clients dns servers to the
> DSL ISP's dns servers, they could resolve again, but they can still
> not surf anymore. The Linux machine itself can still perfectly do
> everything.

All posted information considered, one guess would be that IP forwarding
is not working. Does cat /proc/sys/net/ipv4/ip_forward yield 1 or 0?
Another guess would be the LAN interface isn't working, but that would
also prevent LAN<->Linux_box traffic.

Although I'm no expert, the two iptables network forwarding rules for
LAN<->TELENET looked reasonable. However, the unfettered inbound Internet
traffic would worry me.

<snip technical stuff>

--
Clifford Kite
/* Speak softly and carry a +6 two-handed sword. */

Thank you for your response Clifford !

Forwarding is definately enabled, as the $SERVERIP server is still
able to do dns queries thru the DSL line (although, and that's the
problem, I see no reason why it suddenly wanted to go thru the DSL
connection, none of my rules are set up to do that ... like i said,
my standard random tcp/ip traffic should be routed thru the $TELENET
connection

also, all interfaces are definately working : when i was testing the
setup before, i downloaded things thru the DSL and Cable straight from
the lan...

what is this unferttered inbound Internet traffic
you are referring to ?

another thing i suddenly remember : when the setup stopped working,
and i logged on, the screen had all these "martian source" things on
them, which I am sure are part of the problem.
the thing is, i fixed those on saturday, and on monday it was broken
again

I'll keep on looking, and if anybody else has an idea, please do tell
me !

Andy

(ps: i replied to my own post instead of to Cliffords post because
Google Groups isn't showing his message yet, and my news client
program won't let me post messages

(E-Mail Removed) wrote:

> what is this unferttered inbound Internet traffic
> you are referring to ?

My bad. I was thinking (or not...) that the rules

$IPTABLES -A FORWARD -d $INTNET -j ACCEPT
$IPTABLES -A FORWARD -s $INTNET -j ACCEPT

would allow virtually unrestricted access to $INTNET from the Internet,
which is not true with SNATed RFC 1918 networks. The only non-local
network access possible should be directly from an ISP to $INTNET or to
the DNATed server and server-ports from the Internet via the $SKYNETIP
address.

--
Clifford Kite
/* I hear and I forget. I see and I remember. I do and I understand.
--Confucius, 551-479 BC */