iptables+multi-route - i've rtfm - need expert eyes

i've posted this on LARTC as well - this is a tough one.

okay, so i've been working at this for like 2 days straight. i've
read all the howto's and bunch of emails in the archive here. still
not getting it.

here's the situation:

2 isp's
digis (207.173.210.0)
speakeasy AKA speak (66.92.224.0)
1 linux router (debian, woody, 2.4.18)
1 private LAN (192.168.0.0)

so, what am i trying to do? i want all the traffic from my private
lan to go over speakeasy, EXCEPT the web traffic (port 80) should go
over digis.
sounds simple, right?

use iptables to mark the port 80 packets, route them to a special
routing table made by ip and then use iptables to SNAT them on the
way out- right?

no worky for me.

okay, here is my ultra-simple /etc/init.d/rc.firewall script (ips have
been changed to protect the guilty):


#########START rc.firewall script#########
#!/bin/sh

iptables="/sbin/iptables"

# FLUSH
$iptables -F INPUT
$iptables -F OUTPUT
$iptables -F FORWARD
$iptables -F -t mangle
$iptables -F -t nat
$iptables -F -t filter
$iptables -P OUTPUT ACCEPT
$iptables -P INPUT ACCEPT #default rule to ALLOW everyone

# MARK WEB TRAFFIC
$iptables -t mangle -A PREROUTING -p tcp --dport 80 -s 192.168.0.0/24
-j MARK -
-set-mark 9

# THE NAT
$iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source
207.173.210.222
$iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source
66.93.224.111

echo "1" > /proc/sys/net/ipv4/tcp_syncookies
echo "1" > /proc/sys/net/ipv4/ip_forward

#######END rc.firewall script##########

okay, next step is to use /sbin/ip to make a new routing table so that
my locally marked packets (marked with a 9) can get routed correctly.

here's how i do that:

/bin/echo "9 speak" >> /etc/iproute2/rt_tables
/sbin/ip route add table speak 66.93.224.0 dev eth2 src 66.93.224.111
/sbin/ip route add table speak default via 66.93.224.1
/sbin/ip rule add fwmark 9 table speak

right? okay so now "/sbin/ip rule show" gives me this:

0: from all lookup local
32765: from all fwmark 9 lookup speak
32766: from all lookup main
32767: from all lookup default

alright, and "/sbin/ip route" gives me this:

207.173.210.0/24 dev eth1 proto kernel scope link src
207.173.210.222
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1
66.93.224.0/24 dev eth2 proto kernel scope link src 66.93.224.111
default via 207.173.210.1 dev eth1
default via 66.93.224.1 dev eth2

--notice that there are two default routes...is this good? i'm not
sure, but this is exactly what comes up when i boot up (i can show
you my /etc/network/interfaces file if you want, but its pretty
normal.)

and "/sbin/ip route show table 9" gives me this:

66.92.224.0 dev eth2 scope link src 66.93.224.111
default via 66.93.224.1 dev eth2

--so i know that example 10.1 of the LARTC docs says to copy your
whole main routing table, but it doesn't seem to make a difference.
so i didn't do it. anyway, that's where i am.

so, what is the behaviour of my system after doing all this?

from my router, i can ping and wget anything on the web, so that's
good.

but from my private network i can ping, but wget hangs. the name
resolves, but no connection can be established. tcpdump looks like
this

-- me trying to telnet to port 80 of www.google.com

tcpdump: listening on eth2
66.93.224.111.2009 > 66.102.7.99.80: S [tcp sum ok]
3415820435:3415820435(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
(ttl 127, id 32020, len 48)
0x0000 4500 0030 7d14 4000 7f06 11af 425d e0de
E..0}.@.....B]..
0x0010 4266 0763 07d9 0050 cb99 4893 0000 0000
Bf.c...P..H.....
0x0020 7002 faf0 fed3 0000 0204 05b4 0101 0402
p...............
66.102.7.99.80 > 66.93.224.111.2009: S [tcp sum ok]
342866441:342866441(0) ack 3415820436 win 8190 <mss 1460> (ttl 242,
id 7725, len 44)
0x0000 4500 002c 1e2d 0000 f206 3d9a 4266 0763
E..,.-....=.Bf.c
0x0010 425d e0de 0050 07d9 146f ba09 cb99 4894
B]...P...o....H.
0x0020 6012 1ffe 2044 0000 0204 05b4 0000 `....D........
66.93.224.111.2008 > 216.49.88.118.80: S [tcp sum ok]
3413582795:3413582795(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
(ttl 127, id 32023, len 48)
0x0000 4500 0030 7d17 4000 7f06 2acd 425d e0de
E..0}.@...*.B]..
0x0010 d831 5876 07d8 0050 cb77 23cb 0000 0000
..1Xv...P.w#.....
0x0020 7002 faf0 3ce0 0000 0204 05b4 0101 0402
p...<...........
216.49.88.118.80 > 66.93.224.111.2008: . [tcp sum ok]
3272476716:3272476716(0) ack 3413582796 win 65535 (DF) (ttl 112, id
28324, len 40)
0x0000 4500 0028 6ea4 4000 7006 4848 d831 5876
E..(n.@.p.HH.1Xv
0x0010 425d e0de 0050 07d8 c30e 082c cb77 23cc
B]...P.....,.w#.
0x0020 5010 ffff 994a 0000 0000 0000 0000 P....J........
216.49.88.118.80 > 66.93.224.111.2008: S [tcp sum ok]
3272476715:3272476715(0) ack 3413582796 win 65535 <mss
1460,nop,nop,sackOK> (DF) (ttl 112, id 28492, len 48)
0x0000 4500 0030 6f4c 4000 7006 4798 d831 5876
E..0oL@.p.G..1Xv
0x0010 425d e0de 0050 07d8 c30e 082b cb77 23cc
B]...P.....+.w#.
0x0020 7012 ffff 6c86 0000 0204 05b4 0101 0402
p...l...........
66.93.224.111.2009 > 66.102.7.99.80: S [tcp sum ok]
3415820435:3415820435(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
(ttl 127, id 32029, len 48)
0x0000 4500 0030 7d1d 4000 7f06 11a6 425d e0de
E..0}.@.....B]..
0x0010 4266 0763 07d9 0050 cb99 4893 0000 0000
Bf.c...P..H.....
0x0020 7002 faf0 fed3 0000 0204 05b4 0101 0402
p...............
66.102.7.99.80 > 66.93.224.111.2009: S [tcp sum ok]
342866441:342866441(0) ack 3415820436 win 8190 <mss 1460> (ttl 242,
id 20901, len 44)
0x0000 4500 002c 51a5 0000 f206 0a22 4266 0763
E..,Q......"Bf.c
0x0010 425d e0de 0050 07d9 146f ba09 cb99 4894
B]...P...o....H.
0x0020 6012 1ffe 2044 0000 0204 05b4 0000 `....D........

--and then it just kinda hangs out, until telnet gives up saying that
it timed out.

the good news is that the traffic IS going over the right interface
(eth2 which is hooked up to speakeasy.) the bad news is that it is
hosed. NOTE - it works fine if i skip the "add fwmark rule" step in
the process, but then of course everything gets routed over digis,
which is not good.

okay, so long email i know - if you've read this far, then i really
appreciate it. so, i think i haven't done anything obviously wrong -
so am i missing something? is there some fatal flaw in my
understanding of how this is all supposed to work?

like i said, i've rtfm and i'm really struggling here - any GOOD
ideas?

nerdb0t wrote:

> i've posted this on LARTC as well - this is a tough one.
>
> okay, so i've been working at this for like 2 days straight. i've
> read all the howto's and bunch of emails in the archive here. still
> not getting it.
>
> here's the situation:
>
> 2 isp's
> digis (207.173.210.0)
> speakeasy AKA speak (66.92.224.0)
> 1 linux router (debian, woody, 2.4.18)
> 1 private LAN (192.168.0.0)
>
> so, what am i trying to do? i want all the traffic from my private
> lan to go over speakeasy, EXCEPT the web traffic (port 80) should go
> over digis.
> sounds simple, right?
>
> use iptables to mark the port 80 packets, route them to a special
> routing table made by ip and then use iptables to SNAT them on the
> way out- right?
>
> no worky for me.
>
> okay, here is my ultra-simple /etc/init.d/rc.firewall script (ips have
> been changed to protect the guilty):
>
>
> #########START rc.firewall script#########
> #!/bin/sh
>
> iptables="/sbin/iptables"
>
> # FLUSH
> $iptables -F INPUT
> $iptables -F OUTPUT
> $iptables -F FORWARD
> $iptables -F -t mangle
> $iptables -F -t nat
> $iptables -F -t filter
> $iptables -P OUTPUT ACCEPT
> $iptables -P INPUT ACCEPT #default rule to ALLOW everyone
>
> # MARK WEB TRAFFIC
> $iptables -t mangle -A PREROUTING -p tcp --dport 80 -s 192.168.0.0/24
> -j MARK -
> -set-mark 9
>
> # THE NAT
> $iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source
> 207.173.210.222
> $iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source
> 66.93.224.111
>
> echo "1" > /proc/sys/net/ipv4/tcp_syncookies
> echo "1" > /proc/sys/net/ipv4/ip_forward
>
> #######END rc.firewall script##########
>
> okay, next step is to use /sbin/ip to make a new routing table so that
> my locally marked packets (marked with a 9) can get routed correctly.
>
> here's how i do that:
>
> /bin/echo "9 speak" >> /etc/iproute2/rt_tables
> /sbin/ip route add table speak 66.93.224.0 dev eth2 src 66.93.224.111
> /sbin/ip route add table speak default via 66.93.224.1
> /sbin/ip rule add fwmark 9 table speak
>
> right? okay so now "/sbin/ip rule show" gives me this:
>
> 0: from all lookup local
> 32765: from all fwmark 9 lookup speak
> 32766: from all lookup main
> 32767: from all lookup default
>
> alright, and "/sbin/ip route" gives me this:
>
> 207.173.210.0/24 dev eth1 proto kernel scope link src
> 207.173.210.222
> 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1
> 66.93.224.0/24 dev eth2 proto kernel scope link src 66.93.224.111
> default via 207.173.210.1 dev eth1
> default via 66.93.224.1 dev eth2
>
> --notice that there are two default routes...is this good? i'm not
> sure, but this is exactly what comes up when i boot up (i can show
> you my /etc/network/interfaces file if you want, but its pretty
> normal.)
>
> and "/sbin/ip route show table 9" gives me this:
>
> 66.92.224.0 dev eth2 scope link src 66.93.224.111
> default via 66.93.224.1 dev eth2
>
> --so i know that example 10.1 of the LARTC docs says to copy your
> whole main routing table, but it doesn't seem to make a difference.
> so i didn't do it. anyway, that's where i am.
>
> so, what is the behaviour of my system after doing all this?
>
> from my router, i can ping and wget anything on the web, so that's
> good.
>
> but from my private network i can ping, but wget hangs. the name
> resolves, but no connection can be established. tcpdump looks like
> this
>
> -- me trying to telnet to port 80 of www.google.com
>
> tcpdump: listening on eth2
> 66.93.224.111.2009 > 66.102.7.99.80: S [tcp sum ok]
> 3415820435:3415820435(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
> (ttl 127, id 32020, len 48)
> 0x0000 4500 0030 7d14 4000 7f06 11af 425d e0de
> E..0}.@.....B]..
> 0x0010 4266 0763 07d9 0050 cb99 4893 0000 0000
> Bf.c...P..H.....
> 0x0020 7002 faf0 fed3 0000 0204 05b4 0101 0402
> p...............
> 66.102.7.99.80 > 66.93.224.111.2009: S [tcp sum ok]
> 342866441:342866441(0) ack 3415820436 win 8190 <mss 1460> (ttl 242,
> id 7725, len 44)
> 0x0000 4500 002c 1e2d 0000 f206 3d9a 4266 0763
> E..,.-....=.Bf.c
> 0x0010 425d e0de 0050 07d9 146f ba09 cb99 4894
> B]...P...o....H.
> 0x0020 6012 1ffe 2044 0000 0204 05b4 0000 `....D........
> 66.93.224.111.2008 > 216.49.88.118.80: S [tcp sum ok]
> 3413582795:3413582795(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
> (ttl 127, id 32023, len 48)
> 0x0000 4500 0030 7d17 4000 7f06 2acd 425d e0de
> E..0}.@...*.B]..
> 0x0010 d831 5876 07d8 0050 cb77 23cb 0000 0000
> .1Xv...P.w#.....
> 0x0020 7002 faf0 3ce0 0000 0204 05b4 0101 0402
> p...<...........
> 216.49.88.118.80 > 66.93.224.111.2008: . [tcp sum ok]
> 3272476716:3272476716(0) ack 3413582796 win 65535 (DF) (ttl 112, id
> 28324, len 40)
> 0x0000 4500 0028 6ea4 4000 7006 4848 d831 5876
> E..(n.@.p.HH.1Xv
> 0x0010 425d e0de 0050 07d8 c30e 082c cb77 23cc
> B]...P.....,.w#.
> 0x0020 5010 ffff 994a 0000 0000 0000 0000 P....J........
> 216.49.88.118.80 > 66.93.224.111.2008: S [tcp sum ok]
> 3272476715:3272476715(0) ack 3413582796 win 65535 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 112, id 28492, len 48)
> 0x0000 4500 0030 6f4c 4000 7006 4798 d831 5876
> E..0oL@.p.G..1Xv
> 0x0010 425d e0de 0050 07d8 c30e 082b cb77 23cc
> B]...P.....+.w#.
> 0x0020 7012 ffff 6c86 0000 0204 05b4 0101 0402
> p...l...........
> 66.93.224.111.2009 > 66.102.7.99.80: S [tcp sum ok]
> 3415820435:3415820435(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
> (ttl 127, id 32029, len 48)
> 0x0000 4500 0030 7d1d 4000 7f06 11a6 425d e0de
> E..0}.@.....B]..
> 0x0010 4266 0763 07d9 0050 cb99 4893 0000 0000
> Bf.c...P..H.....
> 0x0020 7002 faf0 fed3 0000 0204 05b4 0101 0402
> p...............
> 66.102.7.99.80 > 66.93.224.111.2009: S [tcp sum ok]
> 342866441:342866441(0) ack 3415820436 win 8190 <mss 1460> (ttl 242,
> id 20901, len 44)
> 0x0000 4500 002c 51a5 0000 f206 0a22 4266 0763
> E..,Q......"Bf.c
> 0x0010 425d e0de 0050 07d9 146f ba09 cb99 4894
> B]...P...o....H.
> 0x0020 6012 1ffe 2044 0000 0204 05b4 0000 `....D........
>
> --and then it just kinda hangs out, until telnet gives up saying that
> it timed out.
>
> the good news is that the traffic IS going over the right interface
> (eth2 which is hooked up to speakeasy.) the bad news is that it is
> hosed. NOTE - it works fine if i skip the "add fwmark rule" step in
> the process, but then of course everything gets routed over digis,
> which is not good.
>
> okay, so long email i know - if you've read this far, then i really
> appreciate it. so, i think i haven't done anything obviously wrong -
> so am i missing something? is there some fatal flaw in my
> understanding of how this is all supposed to work?
>
> like i said, i've rtfm and i'm really struggling here - any GOOD
> ideas?
Hi,

why not use a proxy (Squid) on your debian box. Tell the Proxy to use the
digis interface->faster, safer, easier to configure, less traffic

Alex

On Thu, 13 May 2004 20:41:01 -0700, nerdb0t wrote:


[CUT]
> so, what is the behaviour of my system after doing all this?
>
> from my router, i can ping and wget anything on the web, so that's
> good.
>
> but from my private network i can ping, but wget hangs. the name
> resolves, but no connection can be established. tcpdump looks like
> this
>
> -- me trying to telnet to port 80 of www.google.com
>
> tcpdump: listening on eth2

what is happening on eth0 instead? or on the very client?
do those packets reach the client on the way back?


matteo

>
> why not use a proxy (Squid) on your debian box. Tell the Proxy to use the
> digis interface->faster, safer, easier to configure, less traffic
>
> Alex

okay, well - that's a good idea. that should work just fine. it
solves the high-level problem, but it sure doesn't fix whats broken
with my iptables/iproute2 setup.

what i didnt mention is that i also want to eventually route p2p, ftp
and other bulk data over the digis line.

so basically, i really just want to get this iptable/iproute2 to work
so that i can generalize it to solve lots of things. besides, i
really just want to understand all this better, and i'd like to know
why this setup does not work. it seems to me its "by the book".

did you see any glaring errors in the configs i posted? thanks for
you help.

matteo <matteo_@_themanmachine.net> wrote in message
>
> what is happening on eth0 instead? or on the very client?
> do those packets reach the client on the way back?
>
> matteo

good question - i've looked into it further, and it looks like i'm not
completing my 3-way handshake. the client side, which is 192.168.0.12
and is connected to the router via eth0 is not seeing any return
traffic from the TCP connection attempt.

it seems like somehow the final handshake gets lost in the router box
and never makes it back to the NAT'd client.

crap. here is how i do my NAT:

# THE NAT
$iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source
207.173.210.222
$iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source
66.93.224.111

so what is wrong here? for some reason the NAT session is not working
-

STEP ONE - 192.168.0.12 attempts a TCP session with 66.102.7.99
through eth0

STEP TWO - router accepts eth0 packet, NATs it from to 192.168.0.12 to
66.93.224.111

STEP THREE - router sends packet through eth2 out to the internet

STEP FOUR - 66.102.7.99 replies to 66.93.224.111 and this comes in
from the internet to eth2 (observed via tcpdump)

STEP FIVE - nothing happens. this is where the session "gets lost"

i watch via tcpdump on eth0 and see nothing coming in to 192.168.0.12

okay - we're making progress...any ideas??

okay dudes - even though bailed on me, i got tthis thing solved.

here was the problem:

/proc/sys/net/ipv4/conf/eth0 was set to 1.

this is bad and made it drop my SNAT'd packets.

so, set it to 0 and it works. w00t!

this should totally be in some docs somewhere, but its not. that sux0rs.

(E-Mail Removed) (nerdb0t) wrote in message news:<(E-Mail Removed). com>...
> okay dudes - even though bailed on me, i got tthis thing solved.
>
> here was the problem:
>
> /proc/sys/net/ipv4/conf/eth0 was set to 1.

Had seen your struggles and thought of replying to look around here --
glad I didn't now. You solved it your own self (with some well
intentioned help;-) -- good go! It pays to hang in there.

> this is bad and made it drop my SNAT'd packets.
>
> so, set it to 0 and it works. w00t!
>
> this should totally be in some docs somewhere, but its not. that sux0rs.

Well, it sorta is -- ie., many values are set in
/proc/sys/net/ipv4/xxx

BTW, for those who may look here for solution, _which_ value did you
set to 0?

$ cat /proc/sys/net/ipv4/conf/eth0
cat: /proc/sys/net/ipv4/conf/eth0: Is a directory
$ ls /proc/sys/net/ipv4/conf/eth0
accept_redirects forwarding proxy_arp shared_media
accept_source_route log_martians rp_filter tag
arp_filter mc_forwarding secure_redirects
bootp_relay medium_id send_redirects

rp_filter?

See these for some documentation:
http://lartc.org/howto/lartc.kernel.html
http://lartc.org/howto/lartc.kernel.obscure.html
/usr/src/linux-2.X.X/Documentation/networking/ip-sysctl.txt

Note that the sysctl facility never caught on -- most everyone just
writes to /proc/sys/net/ipv4/x?/

to the next mountain,
prg
email above disabled

whoops - yeah, i forgot to put the path in:

echo "0" > /proc/sys/net/ipv4/conf/all/rp_filter

it may be documented somewhere - but i dont recall ever seeing this in
the iptables docs or in /sbin/ip docs.

hope someone finds this info useful.

(E-Mail Removed) (P Gentry) wrote in message news:<(E-Mail Removed). com>...
> (E-Mail Removed) (nerdb0t) wrote in message news:<(E-Mail Removed). com>...
> > okay dudes - even though bailed on me, i got tthis thing solved.
> >
> > here was the problem:
> >
> > /proc/sys/net/ipv4/conf/eth0 was set to 1.
>
> Had seen your struggles and thought of replying to look around here --
> glad I didn't now. You solved it your own self (with some well
> intentioned help;-) -- good go! It pays to hang in there.
>
> > this is bad and made it drop my SNAT'd packets.
> >
> > so, set it to 0 and it works. w00t!
> >
> > this should totally be in some docs somewhere, but its not. that sux0rs.
>
> Well, it sorta is -- ie., many values are set in
> /proc/sys/net/ipv4/xxx
>
> BTW, for those who may look here for solution, _which_ value did you
> set to 0?
>
> $ cat /proc/sys/net/ipv4/conf/eth0
> cat: /proc/sys/net/ipv4/conf/eth0: Is a directory
> $ ls /proc/sys/net/ipv4/conf/eth0
> accept_redirects forwarding proxy_arp shared_media
> accept_source_route log_martians rp_filter tag
> arp_filter mc_forwarding secure_redirects
> bootp_relay medium_id send_redirects
>
> rp_filter?
>
> See these for some documentation:
> http://lartc.org/howto/lartc.kernel.html
> http://lartc.org/howto/lartc.kernel.obscure.html
> /usr/src/linux-2.X.X/Documentation/networking/ip-sysctl.txt
>
> Note that the sysctl facility never caught on -- most everyone just
> writes to /proc/sys/net/ipv4/x?/
>
> to the next mountain,
> prg
> email above disabled