Problem with IP Masquerade and MTUs

A month or so ago I started having a problem with TCP connections that just
seemed to hang, when connecting to hosts on the other side of my VPN
connection. I think I've tracked the problem down, but I don't know what
to do about it. First let me describe the network:

I have a Linux box (Debian woody, running a 2.4.18 kernel) connected to my
DSL, acting as a masquerading router/firewall (iptables 1.2.6) for my home
network (which is a 10.x.x.x net). This box has two real network
interfaces (eth0, which connects to my LAN, and eth1, which connects to my
DSL) and a virtual interface (ipsec0) that is connected to my company's
network via IPSEC (Free S/WAN 1.96, patched to support AT&T's heartbeat
packets). There are a variety of systems (mostly Linux 2.4.x, one Win2K)
on the network. The router is configured to SNAT/MASQUERADE the local
hosts to both eth1 and ipsec0.

What happens when I make connections across the VPN from the router is that
everything works beautifully. When I connect from any of the other hosts,
however, large packets don't go through. For example, if I request a small
web page across the VPN from a host on my network, it works fine. When I
request a large page, the SYN and ACK packets get through, and the request
goes out (and the ACK comes back), but the packet containing the response
never shows up.

By looking at the packets on both the external interface (eth1) and the
IPSEC interface, I can verify (by looking at timestamps) that the encrypted
packet containing the response data never arrives at the external
interface, so it's not being dropped by Free S/WAN on my side.

Watching from the sending side, it appears there are some clear indications
that it's a path MTU discovery issue. When I make a request from the
router to a server on the other side of the VPN, and run tcpdump on the
server side, I see that the server sends no more than 1443 bytes in each
packet, which is precisely the MTU for my IPSEC interface. OTOH, when I
send the request from one of the NATed machines the server sends 1500 bytes
per packet.

Even clearer, if I manually set the MTU on the NATed host to 1443, responses
come back just fine.

Any ideas why MTU discovery isn't working? What's really baffling is that
this setup has worked for a year and then stopped working without any
change AFAIK. The change may have happened on the other side, I suppose.

More to the point, does anyone have any suggestions as to what I can do,
other than manually lowering my MTUs to match the IPSEC MTU?

Thanks,

--
Shawn

Procrastination: Hard work sometimes pays off later,
but laziness always pays off now.

Shawn Willden <(E-Mail Removed)> wrote:
>
>More to the point, does anyone have any suggestions as to what I can do,
>other than manually lowering my MTUs to match the IPSEC MTU?

You already know what the remedy is, why don't you want to
swallow the pill and cure the disease. It isn't as if there is
some horrible problem with lowering the interface MTU.

In fact, if you want to be really pedantic, lower the MTU to
576. It probably will never bite you, but the specs say that
the TCP buffers do not have to handle more than a 576 byte
payload, hence it is possible that any Internet router could
just up and trash any packet sent with an original MTU larger
than 576. (I haven't heard of anyone implementing a router with
buffers that small, but it would not be outside the specs if
someone did.)

Basically, while the default ethernet frame makes a 1500 byte
MTU reasonable for ethernet traffic, it is also true that if any
IP packets are tunneled through that ethernet, the MTU should be
adjusted downward as is appropriate. PPPoE and apparently your
VPN implementation are two examples.

--
Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
Ukpeagvik (Barrow, Alaska) (E-Mail Removed)

Floyd Davidson wrote:

> Shawn Willden <(E-Mail Removed)> wrote:
>>
>>More to the point, does anyone have any suggestions as to what I can do,
>>other than manually lowering my MTUs to match the IPSEC MTU?
>
> You already know what the remedy is, why don't you want to
> swallow the pill and cure the disease. It isn't as if there is
> some horrible problem with lowering the interface MTU.

No, other than having to do it on all of my machines. However, shouldn't
path MTU discovery take care of this for me? The bit I find really
confusing is that the behavior changed, i.e. it used to work with the
default MTU sizes.

In practice, I lowered the MTU on all of my machines two weeks ago.

In any case, thanks for your input.

Shawn.