HELP: NAT/Masquerading broken with 2.6.11 + pppoe (long)

Hi,

first, please excuse me if this is either a dumb question or the wrong
forum. Any pointers would be greatly appreciated in this case, though...

I have a small home network looking as follows:

192.168.42.3
------------ ------
| PMac G4 | | |---DSL Modem (ppp0)
ISDN---|ippp0 eth0|---|Switch|---two computers, printer (192.168.42.x)
------------ ------

The machine marked as "PMac G4" is a PowerMac G4/800 "Silver", running
Linux 2.6.11.4 on the Yellowdog 4.01 disto, which includes iptables
v1.2.9.

I had an "old" setup with the G4 working as router via the isdn adaptor,
which worked flawlessly. I now switched to ADSL, so I removed the ISDN
connection and just changed to ppp0 using the kernel-based pppoe driver.

Now the machines in my "local" net can still connect the G4 via eth0
(e.g. ssh or dns; the dns runs a caching nameserver), and the G4 has full
access to the internet, too. However, nat/masquerading from my local
net via the G4 to the outside world fails. I stripped down my ipfilter
config to a completely open one (see script below), but still had no
success:

<snip>
#!/bin/bash
for n in INPUT OUTPUT FORWARD; do
iptables -F $n
iptables -P $n ACCEPT
done
for n in PREROUTING POSTROUTING OUTPUT; do
iptables -t nat -F $n
iptables -t nat -P $n ACCEPT
done
iptables -t nat -A POSTROUTING -s 192.168.42.0/24 -o ppp0 -j MASQUERADE
iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS \
--clamp-mss-to-pmtu
echo "1" > /proc/sys/net/ipv4/ip_forward
for n in $(find /proc/sys/net/ipv4 -name rp_filter) ; do
echo "0" > $n
done
</snip>

Running tcpdump on both eth0 and ppp0, I saw that e.g. a http request from
one of the local machines is actually passed via ppp0 to the remote host.
However, all reply packets from that box are never passed back to eth0, so
this looks to me as if masquerading somehow fails. I got one comment from
the iptables forum that tzhe third message on the ppp0 side means the a
tcp reset has been sent back by G4 upon the reception of the packet from
the outside world:

[root@antares root]# tcpdump -nn -i eth0 tcp port 80
18:16:21.012143 IP 192.168.42.4.49223 > 213.95.27.115.80: S 2685214081:2685214081(0) win 65535 <mss 1460,nop,wscale 0,nop,nop,timestamp 2148180757 0>
18:16:23.779283 IP 192.168.42.4.49223 > 213.95.27.115.80: S 2685214081:2685214081(0) win 65535 <mss 1460,nop,wscale 0,nop,nop,timestamp 2148180762 0>
18:16:26.626863 IP 192.168.42.4.49224 > 213.95.27.115.80: S 2390183934:2390183934(0) win 65535 <mss 1460,nop,wscale 0,nop,nop,timestamp 2148180768 0>
(more messages snipped)

root@antares root]# tcpdump -nn -i ppp0 tcp port 80
18:16:21.012206 IP 84.44.131.113.49223 > 213.95.27.115.80: S 2685214081:2685214081(0) win 65535 <mss 1452,nop,wscale 0,nop,nop,timestamp 2148180757 0>
18:16:21.085651 IP 213.95.27.115.80 > 84.44.131.113.49223: S 2677460604:2677460604(0) ack 2685214082 win 5792 <mss 1460,nop,nop,timestamp 1472713132 2148180757,nop,wscale 2>
18:16:21.085748 IP 84.44.131.113.49223 > 213.95.27.115.80: R 2685214082:2685214082(0) win 0
18:16:23.779332 IP 84.44.131.113.49223 > 213.95.27.115.80: S 2685214081:2685214081(0) win 65535 <mss 1452,nop,wscale 0,nop,nop,timestamp 2148180762 0>
18:16:23.841268 IP 213.95.27.115.80 > 84.44.131.113.49223: S 2680216981:2680216981(0) ack 2685214082 win 5792 <mss 1460,nop,nop,timestamp 1472715888 2148180762,nop,wscale 2>
18:16:23.841326 IP 84.44.131.113.49223 > 213.95.27.115.80: R 2685214082:2685214082(0) win 0
(more messages snipped)

Running the same http access from the G4 *does* work without problems!

Does anyone know what I missed here? The same iptables setup (actually a
lot stricter, i.e. a "real" firewall) worked fine with ISDN/ippp0. I tried
various combinations of (below /proc/sys/net/ipv4/) rp_filter set to 0 and
1, as well as setting ip_dynaddr to 0, 1 or 2 - all without success.

I also verified that this setup is at least technically working; running
the G4 under MacOS 10.3.9 client, with a little ipfw and natd fiddling the
machine is doing full nat and firewalling as expected. However, as I
usually use Linux, a running is really important for me.

HELP - I am really lost here, so any help/pointer would be really welcome!
As I am no kernel/networking guru, I also don't know where/how I could add
more debugging to track down the problem. Maybe this is a PowerPC (i.e.
endianess) specific problem? I meanwhile asked this question in a number
of lists (inter alia netfilter, linux-net and the german ADSL forum), but
did not get any helpful reply...

Thanks in advance,
Albrecht.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
Albrecht Dreß - Johanna-Kirchner-Straße 13 - D-53123 Bonn (Germany)
Phone (+49) 228 6199571 - private.php?do=newpm&u=
GnuPG public key: http://home.arcor.de/dralbrecht.dress/pubkey.asc
__________________________________________________ _______________________



=?iso-8859-1?q?Albrecht_Dre=DF?=

Albrecht Dreß <(E-Mail Removed)> wrote:

> I have a small home network looking as follows:
>
> 192.168.42.3
> ------------ ------
> | PMac G4 | | |---DSL Modem (ppp0)
> ISDN---|ippp0 eth0|---|Switch|---two computers, printer (192.168.42.x)
> ------------ ------
....
> Now the machines in my "local" net can still connect the G4 via eth0
> (e.g. ssh or dns; the dns runs a caching nameserver), and the G4 has full
> access to the internet, too. However, nat/masquerading from my local
> net via the G4 to the outside world fails. I stripped down my ipfilter
> config to a completely open one (see script below), but still had no
> success:

I would try to use for the pppoe connection a dedicated
eth-NIC to which the DSL modem is directly connected.

IIRC NICs with an IP address assigned are supposed to
cause problems when running PPPoE over it. I can't
remember the exact reasons, though and I had even
such a setup successfully running a long time ago.

However I would give it a try with a dedicated NIC.
They are very cheap nowadays and such a setup is
smarter anyway, since the broadcasts of your private
LAN are not bridged by DSL modem towards your ISP.


Ciao, Horst
--
»When pings go wrong (It hurts me too)« E.Clapton/E.James/P.Tscharn

Am Mon, 16 May 2005 00:15:45 +0200 schrieb Horst Knobloch:
> I would try to use for the pppoe connection a dedicated
> eth-NIC to which the DSL modem is directly connected.
>
> IIRC NICs with an IP address assigned are supposed to
> cause problems when running PPPoE over it. I can't
> remember the exact reasons, though and I had even
> such a setup successfully running a long time ago.

Well, this would of course be a possibility... However, IMHO Linux ought
to be able to handle this setup - as I said before, it *does* work
perfectly with MacOS 10.3 (which is built on top of a BSD system, but that
shouldn't make a big difference).

> However I would give it a try with a dedicated NIC. They are very cheap
> nowadays and such a setup is smarter anyway, since the broadcasts of
> your private LAN are not bridged by DSL modem towards your ISP.

Hmmm, if I understand the setup correctly, the DSL modem is supposed to
catch only the packets directed to it, right (i.e the ppp0 interface)? So
it should be possible to block all broadcasts (and other unwanted packets)
coming from and sent to the isp using a proper firewall (iptables, ipfw)
setup?

Cheers, Albrecht.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
Albrecht Dreß - Johanna-Kirchner-Straße 13 - D-53123 Bonn (Germany)
Phone (+49) 228 6199571 - private.php?do=newpm&u=
GnuPG public key: http://home.arcor.de/dralbrecht.dress/pubkey.asc
__________________________________________________ _______________________

Albrecht Dreß <(E-Mail Removed)> wrote:

> Am Mon, 16 May 2005 00:15:45 +0200 schrieb Horst Knobloch:
>> I would try to use for the pppoe connection a dedicated
>> eth-NIC to which the DSL modem is directly connected.
>>
>> IIRC NICs with an IP address assigned are supposed to

"are supposed" is a little bit strong "is said" is
more appropriate.

>> cause problems when running PPPoE over it. I can't
>> remember the exact reasons, though and I had even
>> such a setup successfully running a long time ago.
>
> Well, this would of course be a possibility... However, IMHO Linux ought
> to be able to handle this setup - as I said before, it *does* work
> perfectly with MacOS 10.3 (which is built on top of a BSD system, but
> that shouldn't make a big difference).

I think I also never a read an explanation for it and
this was the reason why I tried it and it worked (at
least with rp-pppoe).


>> However I would give it a try with a dedicated NIC. They are very cheap
>> nowadays and such a setup is smarter anyway, since the broadcasts of
>> your private LAN are not bridged by DSL modem towards your ISP.
>
> Hmmm, if I understand the setup correctly, the DSL modem is supposed to
> catch only the packets directed to it, right (i.e the ppp0 interface)?
> So it should be possible to block all broadcasts (and other unwanted
> packets) coming from and sent to the isp

In your setup (pppoe is running on Linux) the ADSL modem is
a bridge, bridging traffic between the physical ethernet
(your private LAN) and the ATM interface towards your provider.
The bridge must transmit broadcasts in your LAN towards the
provider. That's how bridges work and you can't prevent this
by any filter rules deployed on the Linux box.


> using a proper firewall
> (iptables, ipfw) setup?

No (unless it is a very samrt bridge which supports filtering).
Installing filter rules on the linux box doesn't help when
the ADSL modem bridge is connected to the same LAN segment
where also the other hosts reside.

However in most cases you deploy an ADSL router or a router
to which the ADSL modem is connected on its WAN port for
doing this filtering, or connect the ADSL modem on a
dedicate ethernet of your Linux box.


PS. It wouldn't hurt to also check the rules actually in
place via iptables-save to make sure that no other rules
are installed e.g. when the ppp interfaces comes up which
causes you problems.


Ciao, Horst
--
»When pings go wrong (It hurts me too)« E.Clapton/E.James/P.Tscharn

Horst Knobloch schrieb:
> "are supposed" is a little bit strong "is said" is
> more appropriate.

Anyway, this sounds like a rather fundamental flaw (you might even want
to call it a bug) in Linux if this is a no-go here but a trivial option
in other unix'es (like OS X/Darwin).

> I think I also never a read an explanation for it and
> this was the reason why I tried it and it worked (at
> least with rp-pppoe).

See above...

>>>However I would give it a try with a dedicated NIC. They are very cheap

Thinking a little more over it, the problem is actually less trivial, as
I still want to have Internet with OS X... So I have to find a nic which
is supported by Darwin/OS X (my ISDN Fritz card wasn't, which was an
other reason to switch over to adsl - all Mac's come with a built-in
network card).

In the end, it might actually be easier to move completely to OS X
instead of using Linux anymore at all. Not a nice idea, as I am a Linux
user (and supporter!) since the days of kernel 1.2.8...

> In your setup (pppoe is running on Linux) the ADSL modem is
[snipped great explanation about bridges]
> dedicate ethernet of your Linux box.

Thanks a lot for that explanation - I didn know that ;-)

> PS. It wouldn't hurt to also check the rules actually in
> place via iptables-save to make sure that no other rules
> are installed e.g. when the ppp interfaces comes up which
> causes you problems.

They are - I already check that after a hint from the iptables list.
This really seems to be something buried deeper somewhere in the kernel.
Send a packet from the box itself - works fine, send (almost) the same
packet from the local net, using the box as router - results in a reset.

Cheers, Albrecht.

Albrecht Dreß <(E-Mail Removed)> wrote:

> Horst Knobloch schrieb:

[dedicate NIC for PPPoE]

> Thinking a little more over it, the problem is actually less trivial, as
> I still want to have Internet with OS X... So I have to find a nic which
> is supported by Darwin/OS X (my ISDN Fritz card wasn't, which was an
> other reason to switch over to adsl - all Mac's come with a built-in
> network card).

I don't know MACs but I would be surprised if they don't
support 10/100Base-T NICs with RTL chipsets. Another
solution would be to use the built in NIC for the ADSL
modem so you can still connect under OS X the Internet,
and the other to connect to the internal LAN. Of course
this has as drawback that there would be no routing for
the LAN clients if you dial-in via OS X. So, this might
be also no feasible solution.


> In the end, it might actually be easier to move completely to OS X
> instead of using Linux anymore at all. Not a nice idea, as I am a Linux
> user (and supporter!) since the days of kernel 1.2.8...

This would be bad, so we need to try a little bit harder
to solve the problem.


[iptables rules are supposed to be correct]

> This really seems to be something buried deeper somewhere in the kernel.
> Send a packet from the box itself - works fine, send (almost) the same
> packet from the local net, using the box as router - results in a reset.

I guess iptables didn't recognise that the returned packet
belongs to an "existing" client connection and thus the
[SYN,ACK] hits the Linux box TCP stack (INPUT chain). The
TCP stack doesn't know the connection due to this [RST] is
send back in reply to the [SYN,ACK].

Check whether the outgoing connection from the client is
inserted into the connection tracking table by invoking
cat /proc/net/ip_conntrack.

Put into the FORWARD and INPUT chain a "-j LOG" target to
check which chain the [SYN,ACK] segment hits.

Have you compiled the kernel yourself? Is your iptables
version suitable for the kernel version (kernel documentation
should give a hint). If you haven't compiled kernel and iptables
yourself, give it a try (if you feel comfortable with it).
Pay attention that the kernel loads modules matching its
version.

Try a different kernel version.


Note that all this is only a way to pin-point the problem
furhter. I don't know whether anything reveals useful
information or even solves your problem.


Ciao, Horst
--
»When pings go wrong (It hurts me too)« E.Clapton/E.James/P.Tscharn

Am Mon, 16 May 2005 20:17:33 +0200 schrieb Horst Knobloch:
> I don't know MACs but I would be surprised if they don't
> support 10/100Base-T NICs with RTL chipsets. Another

I heared rumors that Realtek has actually a driver for their '8139 based
cards - not sure though. The newer Mac's come with a built-in SUN Gem
Gigabit chip.

> solution would be to use the built in NIC for the ADSL modem so you can
> still connect under OS X the Internet, and the other to connect to the
> internal LAN. Of course this has as drawback that there would be no
> routing for the LAN clients if you dial-in via OS X. So, this might be
> also no feasible solution.

No, that's no solution. Internet should be accessible all time when the
router box is up.

> I guess iptables didn't recognise that the returned packet belongs to an
> "existing" client connection and thus the [SYN,ACK] hits the Linux box
> TCP stack (INPUT chain). The TCP stack doesn't know the connection due
> to this [RST] is send back in reply to the [SYN,ACK].
>
> Check whether the outgoing connection from the client is inserted into
> the connection tracking table by invoking cat /proc/net/ip_conntrack.
>
> Put into the FORWARD and INPUT chain a "-j LOG" target to check which
> chain the [SYN,ACK] segment hits.

Repeating the same test (trying to connect tcp/80 somewhere from the
internal net), here is the relevant (?) line from cat
/proc/net/ip_conntrack:

tcp 6 119 SYN_SENT src=192.168.42.4 dst=151.189.20.30 sport=49244 \
dport=80 [UNREPLIED] src=151.189.20.30 dst=84.44.211.168 sport=80 \
dport=49244 use=1

and "grep PT=80 /var/log/messages"

May 17 19:40:30 antares kernel: Forward IN=eth0 OUT=ppp0 SRC=192.168.42.4 \
DST=151.189.20.30 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=3199 DF PROTO=TCP \
SPT=49244 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
May 17 19:40:30 antares kernel: Input IN=ppp0 OUT= MAC= SRC=151.189.20.30 \
DST=84.44.211.168 LEN=44 TOS=0x00 PREC=0x00 TTL=56 ID=36346 PROTO=TCP \
SPT=80 DPT=49244 WINDOW=8192 RES=0x00 ACK SYN URGP=0

The corresponding iptables-save output is

[root@antares root]# iptables-save
# Generated by iptables-save v1.2.9 on Tue May 17 19:50:11 2005
*filter
:INPUT ACCEPT [494:86864]
:FORWARD ACCEPT [1:60]
:OUTPUT ACCEPT [545:41719]
-A INPUT -j LOG --log-prefix "Input "
-A FORWARD -j LOG --log-prefix "Forward "
-A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Tue May 17 19:50:11 2005
# Generated by iptables-save v1.2.9 on Tue May 17 19:50:11 2005
*nat
:PREROUTING ACCEPT [3:178]
:POSTROUTING ACCEPT [19:1237]
:OUTPUT ACCEPT [19:1237]
-A POSTROUTING -s 192.168.42.0/255.255.255.0 -o ppp0 -j MASQUERADE
COMMIT
# Completed on Tue May 17 19:50:11 2005

> Have you compiled the kernel yourself?

Yes, it's a self-compiled 2.6.11.4 from kernel org - not the bleeding edge
one, but afaict the more recent changelog's don't contain related
changes.

> Is your iptables version suitable
> for the kernel version (kernel documentation should give a hint). If you

Should be - it's 1.2.9 (coming as RPM with YellowDog Linux which is more
or less the same as FC2), and the latest is 1.3.1.

> haven't compiled kernel and iptables yourself, give it a try (if you
> feel comfortable with it). Pay attention that the kernel loads modules
> matching its version.
>
> Try a different kernel version.

Well, I'll try to check if any of my friends has an old Realtek card -
that sounds like the most promising attempt. I can't interpret the log
output above, but it doesn't look wrong to me...

> Note that all this is only a way to pin-point the problem furhter. I
> don't know whether anything reveals useful information or even solves
> your problem.

It is really great that you try to help me! As I said before, I am really
lost here, and it *should* actually be a trivial task...

Thanks again,

Albrecht.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
Albrecht Dreß - Johanna-Kirchner-Straße 13 - D-53123 Bonn (Germany)
Phone (+49) 228 6199571 - private.php?do=newpm&u=
GnuPG public key: http://home.arcor.de/dralbrecht.dress/pubkey.asc
__________________________________________________ _______________________

Albrecht Dreß <(E-Mail Removed)> wrote:

> Am Mon, 16 May 2005 20:17:33 +0200 schrieb Horst Knobloch:
....
>> I guess iptables didn't recognise that the returned packet belongs to an
>> "existing" client connection and thus the [SYN,ACK] hits the Linux box
>> TCP stack (INPUT chain). The TCP stack doesn't know the connection due
>> to this [RST] is send back in reply to the [SYN,ACK].
>>
>> Check whether the outgoing connection from the client is inserted into
>> the connection tracking table by invoking cat /proc/net/ip_conntrack.
>>
>> Put into the FORWARD and INPUT chain a "-j LOG" target to check which
>> chain the [SYN,ACK] segment hits.
>
> Repeating the same test (trying to connect tcp/80 somewhere from the
> internal net), here is the relevant (?) line from cat
> /proc/net/ip_conntrack:
>
> tcp 6 119 SYN_SENT src=192.168.42.4 dst=151.189.20.30 sport=49244 \
> dport=80 [UNREPLIED] src=151.189.20.30 dst=84.44.211.168 sport=80 \
> dport=49244 use=1
>
> and "grep PT=80 /var/log/messages"
>
> May 17 19:40:30 antares kernel: Forward IN=eth0 OUT=ppp0 SRC=192.168.42.4
> \ DST=151.189.20.30 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=3199 DF PROTO=TCP
> \ SPT=49244 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
> May 17 19:40:30 antares kernel: Input IN=ppp0 OUT= MAC= SRC=151.189.20.30
> \ DST=84.44.211.168 LEN=44 TOS=0x00 PREC=0x00 TTL=56 ID=36346 PROTO=TCP \
> SPT=80 DPT=49244 WINDOW=8192 RES=0x00 ACK SYN URGP=0

As expected. The outgoing request travels the FORWARD chain
and creates an entry in the connection track table. However
the incoming answer is not recognized to belong to the existing
connection and thus hits the INPUT chain and is given the local
TCP stack for processing (instead of undoing the masquerading
and traveling the FORWARD chain). The local TCP stack does not
know about the connection and sends [RST] back.

To me it's clear *what* happens but not *why* it happens.

....
>> Have you compiled the kernel yourself?
>
> Yes, it's a self-compiled 2.6.11.4 from kernel org - not the bleeding
> edge one, but afaict the more recent changelog's don't contain related
> changes.

I would test with a different version (may be 2.4.x kernel) to
make sure it's not a problem with the 2.6.11.4 kernel. To me
it looks like that something with the connection tracking is
broken.


....
> It is really great that you try to help me! As I said before, I am really
> lost here, and it *should* actually be a trivial task...

You are welcome, but this wasn't helpful yet.


Ciao, Horst
--
»When pings go wrong (It hurts me too)« E.Clapton/E.James/P.Tscharn

Horst Knobloch schrieb:
[snipped conntrack/fw debug output]
> As expected. The outgoing request travels the FORWARD chain
> and creates an entry in the connection track table. However
> the incoming answer is not recognized to belong to the existing
> connection and thus hits the INPUT chain and is given the local
> TCP stack for processing (instead of undoing the masquerading
> and traveling the FORWARD chain). The local TCP stack does not
> know about the connection and sends [RST] back.
>
> To me it's clear *what* happens but not *why* it happens.

Well, do you have an idea *where* it occurrs? If the decision whether
the packet should go into which chain is taken at one specific place,
maybe a few printk()'s would be helpful to get more insight...

> I would test with a different version (may be 2.4.x kernel) to
> make sure it's not a problem with the 2.6.11.4 kernel. To me
> it looks like that something with the connection tracking is
> broken.

Yes, I will try that... however, what do you think about some more
extensive debugging in the kernel itself? I just don't know where I
should start :-(

> You are welcome, but this wasn't helpful yet.

To me it sounds as if you're wearing the "black belt" of Linux
networking ;-)

Cheers, Albrecht.

Albrecht Dreß <(E-Mail Removed)> wrote:

> Horst Knobloch schrieb:
[Problems with connection tracking]
> Well, do you have an idea *where* it occurrs? If the decision whether
> the packet should go into which chain is taken at one specific place,
> maybe a few printk()'s would be helpful to get more insight...

At a first glance I would say, enable debugging in
net/ipv4/netfilter for files ip_conntrack_core.c,
ip_conntrack_proto_tcp.c and ip_fw_compat_masq.c by
changing

#if 0
#define DEBUGP printk

to

#if 1
#define DEBUGP printk

May be you want to add some of your own printk's in the
procedure ip_conntrack_in() in ip_conntrack_core.c,
check_for_demasq() and do_masquerade() in ip_fw_compat_masq.c


>> I would test with a different version (may be 2.4.x kernel) to
>> make sure it's not a problem with the 2.6.11.4 kernel. To me
>> it looks like that something with the connection tracking is
>> broken.
>
> Yes, I will try that... however, what do you think about some more
> extensive debugging in the kernel itself? I just don't know where I
> should start :-(

See above. However also don't forget the dedicated NIC
for pppoe to rule out another probability.


HTH

Bye, Horst
--
»When pings go wrong (It hurts me too)« E.Clapton/E.James/P.Tscharn