Right Interface - Wrong IP

Hi all,

I'm having some troubles with a debian server, server's description :
- two network interfaces (eth0 - eth1)
- default gateway GATEWAY_eth0 on eth0
- a second gateway GATEWAY_eth1 on eth1, used for local web access

-----------------------
Here are my ip rules :

(i've created a new table called 'web' in /etc/iproute2/rt_tables)
ip rule add fwmark 80 lookup web

ip route add default via GATEWAY_eth0 dev eth0

ip route add table web default via GATEWAY_eth1 dev eth1
ip route add table web LOCAL_SUBNET/MASK dev eth1

-----------------------
My iptables (there's no nat, no input/output rules) :

iptables -t mangle -A OUTPUT -p tcp --dport 80 -j MARK --set-mark 80

-----------------------
What appens :
When I look for output packets on eth1 :

tcpdump -vni eth1 host SERVER_WEB

10:49:33.837398 IP (tos 0x10, ttl 64, id 18604, offset 0, flags [DF],
length: 60) IP_eth0.33040 > SERVER_WEB.80: S [tcp sum ok]
2649806980:2649806980(0) win 5840 <mss 1460,sackOK,timestamp 348491816
0,nop,wscale 0>

The host is trying to acces an external web server via the GATEWAY_eth1
with his ip address set up on eth0 !

I've never seen that... Any idea ?

Serge.

Hello,

serge a écrit :
>
> The host is trying to acces an external web server via the GATEWAY_eth1
> with his ip address set up on eth0 !
>
> I've never seen that... Any idea ?

It happens all the time. The reason is quite simple : source address
selection is done before the packet enters the OUTPUT chains and can be
marked. One workaround is to use SNAT or MASQUERADE (if kernel >=
2.6.14) on eth1.

Thanks for your answer.

I'm not sure it happens all the time.
I've setted up a similar configuration, with exactly the same rules, the
same iptables' chains, and all works fine. The only difference is around
the hardware (we have same kernel/debian version too).

Source address selection is done before, but how is it decided ?

My filling is that, when generating packets, the interaction between
netfilter and iproute2 looks like this:
Routing decision made
Netfilter OUTPUT (nat + mangle) hooks are traversed (packets are
marked with 80)
OUPTUT chains are traversed
POSTROUTING hooks are traversed
Routing decision made again, going into table web
Packets are transmetted

What do you think about it ?

Regards.

Pascal Hambourg wrote:
> Hello,
>
> serge a écrit :
>>
>> The host is trying to acces an external web server via the
>> GATEWAY_eth1 with his ip address set up on eth0 !
>>
>> I've never seen that... Any idea ?
>
> It happens all the time. The reason is quite simple : source address
> selection is done before the packet enters the OUTPUT chains and can be
> marked. One workaround is to use SNAT or MASQUERADE (if kernel >=
> 2.6.14) on eth1.

serge a écrit :
>> The host is trying to acces an external web server via the
>> GATEWAY_eth1 with his ip address set up on eth0 !
>>
>> I've never seen that... Any idea ?
>
> It happens all the time. The reason is quite simple : source address
> selection is done before the packet enters the OUTPUT chains and can
> be marked. One workaround is to use SNAT or MASQUERADE (if kernel >=
> 2.6.14) on eth1.
>
> I'm not sure it happens all the time.
> I've setted up a similar configuration, with exactly the same rules, the
> same iptables' chains, and all works fine. The only difference is around
> the hardware (we have same kernel/debian version too).

This is a purely IP layer software issue, the hardware cannot make any
difference.

> Source address selection is done before, but how is it decided ?

As usual, by looking up the routing table. In short :
- if the route to the destination contains a source address, that
address is selected ;
- else the best suitable address of the output interface (know at that
time) is selected ;
- if the output interface is unnumbered, another suitable local address
is selected.

> My feeling is that, when generating packets, the interaction between
> netfilter and iproute2 looks like this:
> Routing decision made
> Netfilter OUTPUT (nat + mangle) hooks are traversed (packets are
> marked with 80)
> OUTPUT chains are traversed

Note : OUTPUT chains are a part of Netfilter OUTPUT (actually
NF_IP_LOCAL_OUT) hooks.

> POSTROUTING hooks are traversed
> Routing decision made again, going into table web
> Packets are transmetted
>
> What do you think about it ?

This is correct except that the re-routing decision takes place between
OUTPUT and POSTROUTING, not after POSTROUTING. You need that so the new
output interface resulting from the re-routing decision can be taken
into account in POSTROUTING rules.

OK. Thanks very much for your explanation !

Something I can't understand...
Source IP is decided when first routing decision. Why source is not
changed to the right one (while precised in ip route src) ? Is it a bug ?

I can't SNAT the output packets since the source is not correct (packets
go out, but conntrack does not consider it since the packet is not right ).

Do you know a solution ? Does a kernel upgrade could correct it ? (got a
2.6.8)

I've verified on my other server, and the route is not exactly the same.
A rule specify that to join the second proxy, it must use the second
ethernet interface, so the source is right.

Pascal Hambourg wrote:
> serge a écrit :
>>> The host is trying to acces an external web server via the
>>> GATEWAY_eth1 with his ip address set up on eth0 !
>>>
>>> I've never seen that... Any idea ?
>>
>> It happens all the time. The reason is quite simple : source address
>> selection is done before the packet enters the OUTPUT chains and can
>> be marked. One workaround is to use SNAT or MASQUERADE (if kernel >=
>> 2.6.14) on eth1.
>>
>> I'm not sure it happens all the time.
>> I've setted up a similar configuration, with exactly the same rules,
>> the same iptables' chains, and all works fine. The only difference is
>> around the hardware (we have same kernel/debian version too).
>
> This is a purely IP layer software issue, the hardware cannot make any
> difference.
>
>> Source address selection is done before, but how is it decided ?
>
> As usual, by looking up the routing table. In short :
> - if the route to the destination contains a source address, that
> address is selected ;
> - else the best suitable address of the output interface (know at that
> time) is selected ;
> - if the output interface is unnumbered, another suitable local address
> is selected.
>
>> My feeling is that, when generating packets, the interaction between
>> netfilter and iproute2 looks like this:
>> Routing decision made
>> Netfilter OUTPUT (nat + mangle) hooks are traversed (packets are
>> marked with 80)
>> OUTPUT chains are traversed
>
> Note : OUTPUT chains are a part of Netfilter OUTPUT (actually
> NF_IP_LOCAL_OUT) hooks.
>
>> POSTROUTING hooks are traversed
>> Routing decision made again, going into table web
>> Packets are transmetted
>>
>> What do you think about it ?
>
> This is correct except that the re-routing decision takes place between
> OUTPUT and POSTROUTING, not after POSTROUTING. You need that so the new
> output interface resulting from the re-routing decision can be taken
> into account in POSTROUTING rules.

serge a écrit :
>
> Something I can't understand...
> Source IP is decided when first routing decision. Why source is not
> changed to the right one (while precised in ip route src) ? Is it a bug ?

It's not a bug, it's a feature. (c)(tm)(r) ;-)
Changing the source address at the re-routing may break things such as
TCP connections. And what if the sending process selected an explicit
source address instead of letting the IP stack select the default source
address ?

> I can't SNAT the output packets since the source is not correct (packets
> go out, but conntrack does not consider it since the packet is not right ).

What do you mean by "the source is not correct" ? I do not understant
why you could not use SNAT. AFAIK conntrack does not care about routing.
With SNAT, do packets go out with the correct source address ? If yes,
then conntrack and SNAT work. If your problem is that the replies from
the remote server are dropped, you must disable rp_filter on eth1 ; this
has nothing to do with SNAT, it would happen even though packets had the
correct source address without SNAT.

> Do you know a solution ? Does a kernel upgrade could correct it ? (got a
> 2.6.8)

This is a very old kernel anyway. Consider upgrading. The current stable
Debian includes the kernel 2.6.18.

Many thanks for your help.

I've modified the process (squid) to force it to send packet from the
wanted interface, and it works.

Regards.

Pascal Hambourg wrote:
> serge a écrit :
>>
>> Something I can't understand...
>> Source IP is decided when first routing decision. Why source is not
>> changed to the right one (while precised in ip route src) ? Is it a bug ?
>
> It's not a bug, it's a feature. (c)(tm)(r) ;-)
> Changing the source address at the re-routing may break things such as
> TCP connections. And what if the sending process selected an explicit
> source address instead of letting the IP stack select the default source
> address ?
>
>> I can't SNAT the output packets since the source is not correct
>> (packets go out, but conntrack does not consider it since the packet
>> is not right ).
>
> What do you mean by "the source is not correct" ? I do not understant
> why you could not use SNAT. AFAIK conntrack does not care about routing.
> With SNAT, do packets go out with the correct source address ? If yes,
> then conntrack and SNAT work. If your problem is that the replies from
> the remote server are dropped, you must disable rp_filter on eth1 ; this
> has nothing to do with SNAT, it would happen even though packets had the
> correct source address without SNAT.
>
>> Do you know a solution ? Does a kernel upgrade could correct it ? (got
>> a 2.6.8)
>
> This is a very old kernel anyway. Consider upgrading. The current stable
> Debian includes the kernel 2.6.18.

Am Tue, 02 Oct 2007 14:40:25 +0200 schrieb serge:

> Many thanks for your help.
>
> I've modified the process (squid) to force it to send packet from the
> wanted interface, and it works.
>
> Regards.
>
> Pascal Hambourg wrote:
>> serge a écrit :
>>>
>>> Something I can't understand...
>>> Source IP is decided when first routing decision. Why source is not
>>> changed to the right one (while precised in ip route src) ? Is it a bug ?
>>
>> It's not a bug, it's a feature. (c)(tm)(r) ;-)
>> Changing the source address at the re-routing may break things such as
>> TCP connections. And what if the sending process selected an explicit
>> source address instead of letting the IP stack select the default source
>> address ?
>>
>>> I can't SNAT the output packets since the source is not correct
>>> (packets go out, but conntrack does not consider it since the packet
>>> is not right ).
>>
>> What do you mean by "the source is not correct" ? I do not understant
>> why you could not use SNAT. AFAIK conntrack does not care about routing.
>> With SNAT, do packets go out with the correct source address ? If yes,
>> then conntrack and SNAT work. If your problem is that the replies from
>> the remote server are dropped, you must disable rp_filter on eth1 ; this
>> has nothing to do with SNAT, it would happen even though packets had the
>> correct source address without SNAT.
>> /etc/udev/rules.d/z25_persistent-net.rules
>>> Do you know a solution ? Does a kernel upgrade could correct it ? (got
>>> a 2.6.8)
>>
>> This is a very old kernel anyway. Consider upgrading. The current stable
>> Debian includes the kernel 2.6.18.

maybe udevd has the wrong nic for eth1 and eth0, take a look in
/etc/udev/rules.d/z25_persistent-net.rules

cheers