Traffic going through wrong interface

Hello,

I have a server with 3 Interfaces, in this scenario:

(Carrier 1[eth1]) -|
|-firewall-(Internal [eth0])
(Carrier 2[eth2]) -|

I've set up the NAT and masq rules, so the outgoing general traffic is
to be directed to the interface eth2 (4 Mbps).

The link with Carrier 1 (eth1) is designed to receive-only connections
(i.e.: SMTP, DNS, http, VPN).

(almost) Everything is working fine; traffic is flowing from eth0 ->
eth2, absolutelly normal.

However... When the traffic arrives to eth1, the traffic is correctly
parsed by the firewall, it's directed to the destination server, the
server does reply, sends it back to the firewall, then... It goes out
*via eth2* with the *IP address of eth1*!!!!

It's driving me nuts!

I've set up a routing table as depicted at
http://tldp.org/HOWTO/Adv-Routing-HO...ple-links.html

Any ideas?

Thanks,
Rodrigo.

Rodrigo A B Freire <(E-Mail Removed)> writes:

>Hello,

>I have a server with 3 Interfaces, in this scenario:

>(Carrier 1[eth1]) -|
> |-firewall-(Internal [eth0])
>(Carrier 2[eth2]) -|

>I've set up the NAT and masq rules, so the outgoing general traffic is
>to be directed to the interface eth2 (4 Mbps).

>The link with Carrier 1 (eth1) is designed to receive-only connections
>(i.e.: SMTP, DNS, http, VPN).

>(almost) Everything is working fine; traffic is flowing from eth0 ->
>eth2, absolutelly normal.

>However... When the traffic arrives to eth1, the traffic is correctly
>parsed by the firewall, it's directed to the destination server, the
>server does reply, sends it back to the firewall, then... It goes out
>*via eth2* with the *IP address of eth1*!!!!

Well, what does your routing table tell it to do
route -n
It is simply doing what you told it to do.
When the system sends out a packet, it has no idea that that packet came as
the response of some program to some previous incoming packet. It is just a
packet with an address on it. The routing algorithm cannot determine why
you sent out that packet. It looks up the routing table and does exactly
what it tells it to do with that packet. Now with ip it can be a bit more
sophisticated and can use the ( destination of receiving) to help route the
packet, but I do not think that helps you.



>It's driving me nuts!

>I've set up a routing table as depicted at
>http://tldp.org/HOWTO/Adv-Routing-HO...ple-links.html

>Any ideas?

>Thanks,
>Rodrigo.

On Wed, 08 Aug 2007 14:44:00 -0700 Rodrigo A B Freire <(E-Mail Removed)> wrote:

| However... When the traffic arrives to eth1, the traffic is correctly
| parsed by the firewall, it's directed to the destination server, the
| server does reply, sends it back to the firewall, then... It goes out
| *via eth2* with the *IP address of eth1*!!!!

The remote host that send the traffic won't recognize the response as
being associated with what it sent unless the source address of the
response matches the destination of the original traffic. How would
some host elsewhere sends something to your eth1 address and gets a
response back that came "from" your eth2 address know that it is from
the same place?

Asymmetric routing is actually a bad thing in almost all cases.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-08-09-(E-Mail Removed) |
|------------------------------------/-------------------------------------|

Hello,

phil-news-(E-Mail Removed) a écrit :
>
> Asymmetric routing is actually a bad thing in almost all cases.

I beg to differ. A large part of the routing on the internet is
asymmetric, so it cannot be that bad. Asymmetric routing is bad for
devices which perform stateful operations such as SPI (stateful packet
inspection) or stateful NAT (Network Address Translation), which need to
be in the path of both directions of the traffic associated to an
end-to-end communication.

On Aug 9, 9:00 am, phil-news-nos...@ipal.net wrote:

> The remote host that send the traffic won't recognize the response as
> being associated with what it sent unless the source address of the
> response matches the destination of the original traffic. How would
> some host elsewhere sends something to your eth1 address and gets a
> response back that came "from" your eth2 address know that it is from
> the same place?

I understand; that would be great *if* the remote host actually
received my traffic sent with *eth1 address* via *eth2*.
However, the carrier does filter the traffic (of course; it would be
spoof!) and the traffic never arrives the destination host.

If you pay a visit to http://tldp.org/HOWTO/Adv-Routing-HO...ple-links.html
you'll notice that the article deals with this exact issue; however,
no joy here; outgoing traffic ONLY via default gateway.

Adding some more detail:

ROO-MBTAS[~]#netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window
irtt Iface
201.81.224.0 0.0.0.0 255.255.255.255 UH 0 0
0 eth2
200.168.10.0 0.0.0.0 255.255.255.255 UH 0 0
0 eth1
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0
0 eth0
10.1.1.0 192.168.0.3 255.255.255.0 UG 0 0
0 eth0
200.168.10.0 0.0.0.0 255.255.255.0 U 0 0
0 eth1
201.81.224.0 0.0.0.0 255.255.240.0 U 0 0
0 eth2
0.0.0.0 201.81.224.1 0.0.0.0 UG 0 0
0 eth2

ip route commands:

ip route add 200.168.10.0 dev eth1 src 200.168.10.22 table 1
ip route add default via 200.168.10.1 table 1

ip route add 201.81.224.0 dev eth2 src 201.81.226.26 table 2
ip route add default via 201.81.224.1 table 2

ip route add 200.168.10.0 dev eth1 src 200.168.10.22
ip route add 201.81.224.0 dev eth2 src 201.81.226.26

ip route add default via 201.81.224.1

ip rule add from 200.168.10.22 table 1
ip rule add from 201.81.226.26 table 2

Unruh a écrit :
> Rodrigo A B Freire <(E-Mail Removed)> writes:
>
>>I have a server with 3 Interfaces, in this scenario:
>
>>(Carrier 1[eth1]) -|
>> |-firewall-(Internal [eth0])
>>(Carrier 2[eth2]) -|
>
>>I've set up the NAT and masq rules, so the outgoing general traffic is
>>to be directed to the interface eth2 (4 Mbps).
>
>>The link with Carrier 1 (eth1) is designed to receive-only connections
>>(i.e.: SMTP, DNS, http, VPN).
>
>>(almost) Everything is working fine; traffic is flowing from eth0 ->
>>eth2, absolutelly normal.
>
>>However... When the traffic arrives to eth1, the traffic is correctly
>>parsed by the firewall, it's directed to the destination server, the
>>server does reply, sends it back to the firewall, then... It goes out
>>*via eth2* with the *IP address of eth1*!!!!
>
>
> Well, what does your routing table tell it to do
> route -n

"route -n" only shows the contents of the main IPv4 routing table.
"ip -4 route show table all" shows the contents of all IPv4 routing
tables including main, local, default and the others.

> It is simply doing what you told it to do.

Indeed, at best a computer does what you told it to do, not what you
want it to do ! ;-)

> When the system sends out a packet, it has no idea that that packet came as
> the response of some program to some previous incoming packet.

The whole system can have an idea that a packet is a response thanks to
the connection tracking built in Netfilter, and tell the routing
subsystem to do something special with it. Cf. the connmark and mark
iptables matches and the CONNMARK and MARK iptables targets.

Here are some solutions. Some are simpler than others.

================================================== ============
1) Use secondary addresses ("aliases") for incoming connections.
For instance a host has the primary address 192.168.3.5 for general
purpose and the secondary adress 192.168.3.105 for external incoming
connections.

ifconfig eth0 192.168.3.5
ifconfig eth0:1 192.168.3.105

Incoming connections from the outside are translated to the secondary
address :

iptables -t nat -A PREROUTING -i eth1 [...] -j DNAT --to 192.168.3.105

Then create a routing rule so that packets from 192.168.3.105 are routed
via eth1 :

ip rule add from 192.168.3.105 lookup <alt_table_no>
ip route add default dev eth1 via <eth1_gateway> table <alt_table_no>

Of course you can put all the secondary addresse in a reserved range and
use a single routing rule for all the range :

ip rule from 192.168.3.96/28 lookup <alt_table_no>

================================================== ============
2) Use the source port to identify response packets and mark them with
an iptables rule :

iptables -t mangle -A PREROUTING -i eth0 -p tcp --sport 25 \
-m state --state ESTABLISHED -j MARK --set-mark 0x1

Then you direct the marked packets to the alternate routing table :

ip rule add fwmark 0x1 lookup <alt_table_no>
ip route add default dev eth1 via <eth1_gateway> table <alt_table_no>

================================================== ============
3) Match the original destination address of the incoming DNATed
connection in the response packets. This is done with the "--ctorigdst"
option of the "conntrack" iptables match :

iptables -t mangle -A PREROUTING -i eth0 \
-m conntrack --ctstate DNAT --ctorigdst <eth1_address> \
-j MARK --set-mark 0x1

The "ip rule" and "ip route" setup is the ame as 2).

================================================== ============
4) Mark the connections with the CONNMARK iptables target.
This requires a kernel with connection mark support, i.e. at least
version 2.6.10 or patched with patch-o-matic-ng.

iptables -t mangle -A PREROUTING -i eth1 -m state --state NEW \
-j CONNMARK --set-mark 0x1

This sets a "connection mark" on new _connections_ (not on individual
packets) incoming on eth1. Then mark return packets incoming on eth0
which belong to a marked connection :

iptables -t mangle -A PREROUTING -i eth0 -m connmark --mark 0x1 \
-j CONNMARK --restore-mark

The "ip rule" and "ip route" setup is the ame as 2).

================================================== ============
Disclaimer : I haven't tested all the above. It may contain errors and
need some tuning.

Rodrigo A B Freire a écrit :
> Adding some more detail:

Thanks, it helps.

> ROO-MBTAS[~]#netstat -rn
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> 201.81.224.0 0.0.0.0 255.255.255.255 UH 0 0 0 eth2
> 200.168.10.0 0.0.0.0 255.255.255.255 UH 0 0 0 eth1
> 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
> 10.1.1.0 192.168.0.3 255.255.255.0 UG 0 0 0 eth0
> 200.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
> 201.81.224.0 0.0.0.0 255.255.240.0 U 0 0 0 eth2
> 0.0.0.0 201.81.224.1 0.0.0.0 UG 0 0 0 eth2

What are the two single host routes (with the H flag and 255.255.255.255
mask) for ? Those destinations are supposed to be reserved network
addresses.

> ip route commands:
>
> ip route add 200.168.10.0 dev eth1 src 200.168.10.22 table 1
> ip route add default via 200.168.10.1 table 1

The destination in the first route should be the whole subnet
200.168.10.0/24, not only the network address. By the way, I forgot this
direct rule in my other post ; it may be needed to communicate with
hosts in the subnet other than the gateway. But the src address is
useless : when the table is parsed, it means that the packet source
address matched the rule below and is already set.

> ip route add 201.81.224.0 dev eth2 src 201.81.226.26 table 2
> ip route add default via 201.81.224.1 table 2

Same remark as above. Note that this table is not required because these
routes are already present in the main routing table.

> ip route add 200.168.10.0 dev eth1 src 200.168.10.22
> ip route add 201.81.224.0 dev eth2 src 201.81.226.26
>
> ip route add default via 201.81.224.1

Now I see where the two weird host routes in the main routing table come
from. Same remark as above. Besides, theses two routes are already
present in the main routing table when you configure eth1 and eth2 with
their addresses and netmasks.

> ip rule add from 200.168.10.22 table 1
> ip rule add from 201.81.226.26 table 2

These two rules have an effect only on locally-generated packets sent by
your router. And again, the last rule is useless because the routes in
table 2 are the same as in the main table.

On 9 ago, 10:39, Pascal Hambourg <boite-a-s...@plouf.fr.eu.org> wrote:
> ================================================== ============
> 2) Use the source port to identify response packets and mark
> them with an iptables rule :
>
> iptables -t mangle -A PREROUTING -i eth0 -p tcp --sport 25 \
> -m state --state ESTABLISHED -j MARK --set-mark 0x1
>
> Then you direct the marked packets to the alternate routing table:
>
> ip rule add fwmark 0x1 lookup <alt_table_no>
> ip route add default dev eth1 via <eth1_gateway> table
> <alt_table_no>

Pascal,

Les plus elegant solutions sont les plus simples! ;-)

I've sticked with the solution nr 2.

I've did some polishing on the iptables rule specifying the
destination server (because every traffic related with that port would
go through eth1; which is not desirable).

For reference, to the future generations:

iptables -t mangle -A PREROUTING -i eth0 -s 192.168.0.33 -p tcp --
sport 80 -m state --state ESTABLISHED -j MARK --set-mark 0x1
..
.. more iptables rules, relating the ports with its respective server
..
ip rule add fwmark 0x1 lookup 1
ip route add default dev eth1 via 200.168.10.1 table 1
ip route add default dev eth2 via 201.81.224.1 table 2

Thank you very much for your guiding and help! :-)

Best regards,
RF.