Understanding path MTU discovery

Can someone confirm or correct my understanding of path MTU discovery
regarding packets passing through a linux router/firewall to a server on
a DMZ? As far as I can tell from my reading, if a computer on the
internet accesses our web server, but the reply from the server is too
big (for example, the client computer is using a PPPoE link with an MTU
of 1492), the client's ISP's gateway router will send an ICMP package
back to our router. The router will see this as a "related" packet, and
pass it on to the web server in the DMZ, and the web server will use the
smaller packet size for the rest of the connection.

Will the web server machine remember that lower MTU as being connected
to the client's IP address, so that future connections by the client
will avoid the overhead of the path MTU discovery, or will the discovery
be needed for each new connection? I presume the lower MTU will not be
needed for connections to/from other IP addresses.

What will happen if our router has more than one route back to the
client (i.e., two DSL links)? I understand that I could mark incoming
packets from clients so that replies are sent out through the same
interface they came in, but I would prefer to balance the output packets
(downstream ADSL has more than enough bandwidth for web server requests,
but it would be best to take advantage of two upstream links for
replies). Suppose the client's original request comes in via eth0, gets
passed on to the server on eth2, and the reply happens to go out on eth1
via the other ADSL line. The ICMP packet complaining about the MTU size
is going to be sent back to the eth1 address - will that still count as
"related" and therefore be passed on to the web server?

Thanks,

David

David Brown <(E-Mail Removed)> wrote:
> Can someone confirm or correct my understanding of path MTU
> discovery regarding packets passing through a linux router/firewall
> to a server on a DMZ? As far as I can tell from my reading, if a
> computer on the internet accesses our web server, but the reply from
> the server is too big (for example, the client computer is using a
> PPPoE link with an MTU of 1492), the client's ISP's gateway router
> will send an ICMP package back to our router. The router will see
> this as a "related" packet, and pass it on to the web server in the
> DMZ, and the web server will use the smaller packet size for the
> rest of the connection.

Assuming that there wasn't anything filtering the requisite ICMP
messages in the name of security.

> Will the web server machine remember that lower MTU as being
> connected to the client's IP address, so that future connections by
> the client will avoid the overhead of the path MTU discovery, or
> will the discovery be needed for each new connection? I presume the
> lower MTU will not be needed for connections to/from other IP
> addresses.

Typically a host route for that client's IP address is created in the
server's routing table. This route has a timer associated with it to
a) reset the PathMTU back to link-local and later b) remove the route
entirely if idle. So, it will be rememebered so long as the new
connections to the same client IP happen within the timeouts.

For the second question, indeed the lower PathMTU will not be used for
connections to/from other IP addresses.

> What will happen if our router has more than one route back to the
> client (i.e., two DSL links)? I understand that I could mark
> incoming packets from clients so that replies are sent out through
> the same interface they came in, but I would prefer to balance the
> output packets (downstream ADSL has more than enough bandwidth for
> web server requests, but it would be best to take advantage of two
> upstream links for replies). Suppose the client's original request
> comes in via eth0, gets passed on to the server on eth2, and the
> reply happens to go out on eth1 via the other ADSL line. The ICMP
> packet complaining about the MTU size is going to be sent back to
> the eth1 address - will that still count as "related" and therefore
> be passed on to the web server?

On that one I've no idea, other than to say that the ICMP messages
will be send based on the source IP address in the TCP segment which
triggered it, and the source IP (as well as destination IP and the
local/remote port numbers) will be invariant for a connection
regardless of the path it takes.

rick jones
--
No need to believe in either side, or any side. There is no cause.
There's only yourself. The belief is in your own precision. - Jobert
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones wrote:
> David Brown <(E-Mail Removed)> wrote:
>> Can someone confirm or correct my understanding of path MTU
>> discovery regarding packets passing through a linux router/firewall
>> to a server on a DMZ? As far as I can tell from my reading, if a
>> computer on the internet accesses our web server, but the reply from
>> the server is too big (for example, the client computer is using a
>> PPPoE link with an MTU of 1492), the client's ISP's gateway router
>> will send an ICMP package back to our router. The router will see
>> this as a "related" packet, and pass it on to the web server in the
>> DMZ, and the web server will use the smaller packet size for the
>> rest of the connection.
>
> Assuming that there wasn't anything filtering the requisite ICMP
> messages in the name of security.
>

I gather that this is becoming a problem, with many firewalls being set
to filter out the relevant ICMP messages "in the name of security". Our
current firewall/router (an old ZyWall 10, to be replaced by a linux or
possibly bsd box) blocks all ICMP messages, so I've had no choice but to
force the server's Ethernet port to use an MTU of 1492 for
compatibility with some clients. That works, but is not really ideal.
As far as I can see, allowing "related" ICMP packets through should
cover the required messages while blocking any other ICMP's.

>> Will the web server machine remember that lower MTU as being
>> connected to the client's IP address, so that future connections by
>> the client will avoid the overhead of the path MTU discovery, or
>> will the discovery be needed for each new connection? I presume the
>> lower MTU will not be needed for connections to/from other IP
>> addresses.
>
> Typically a host route for that client's IP address is created in the
> server's routing table. This route has a timer associated with it to
> a) reset the PathMTU back to link-local and later b) remove the route
> entirely if idle. So, it will be rememebered so long as the new
> connections to the same client IP happen within the timeouts.
>
> For the second question, indeed the lower PathMTU will not be used for
> connections to/from other IP addresses.
>

Excellent - thanks for that information.

>> What will happen if our router has more than one route back to the
>> client (i.e., two DSL links)? I understand that I could mark
>> incoming packets from clients so that replies are sent out through
>> the same interface they came in, but I would prefer to balance the
>> output packets (downstream ADSL has more than enough bandwidth for
>> web server requests, but it would be best to take advantage of two
>> upstream links for replies). Suppose the client's original request
>> comes in via eth0, gets passed on to the server on eth2, and the
>> reply happens to go out on eth1 via the other ADSL line. The ICMP
>> packet complaining about the MTU size is going to be sent back to
>> the eth1 address - will that still count as "related" and therefore
>> be passed on to the web server?
>
> On that one I've no idea, other than to say that the ICMP messages
> will be send based on the source IP address in the TCP segment which
> triggered it, and the source IP (as well as destination IP and the
> local/remote port numbers) will be invariant for a connection
> regardless of the path it takes.
>

The source IP will not be invariant for packets coming out of the router
towards the internet - such packets will have different SNAT addresses
depending on which of the two ADSL lines it passes through (each line
has its own Ethernet port on the router, and they have different public
IP addresses). I think I'm going to have to read a bit more about
exactly how SNAT is implemented and how "related" states are determined,
so that I can see what happens here.

Thanks,

David


> rick jones

David Brown <(E-Mail Removed)> wrote:
> The source IP will not be invariant for packets coming out of the
> router towards the internet - such packets will have different SNAT
> addresses depending on which of the two ADSL lines it passes through
> (each line has its own Ethernet port on the router, and they have
> different public IP addresses). I think I'm going to have to read a
> bit more about exactly how SNAT is implemented and how "related"
> states are determined, so that I can see what happens here.

NAT - ah, yes, that wonderful thing. The four-tuple of local/remote
IP, local/remote port is what uniquely "names" a TCP connection. When
there is only one NAT, there remains a 1-1 mapping of the TCP
connection name between its "intenal" and "external" persona.
Introduce the possibility that a given TCP connection may have _two_
external personas (personi?) then I suspect all Heck breaks loose and
Prince Phil will run rufshod over your connectivity.

rick jones

Longing for the day when a stake can be driven through the heart of
NAT.

NAT, routing, firewall - all different things...even if they are in
the same box...

--
portable adj, code that compiles under more than one compiler
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones wrote:
> David Brown <(E-Mail Removed)> wrote:
>> The source IP will not be invariant for packets coming out of the
>> router towards the internet - such packets will have different SNAT
>> addresses depending on which of the two ADSL lines it passes through
>> (each line has its own Ethernet port on the router, and they have
>> different public IP addresses). I think I'm going to have to read a
>> bit more about exactly how SNAT is implemented and how "related"
>> states are determined, so that I can see what happens here.
>
> NAT - ah, yes, that wonderful thing. The four-tuple of local/remote
> IP, local/remote port is what uniquely "names" a TCP connection. When
> there is only one NAT, there remains a 1-1 mapping of the TCP
> connection name between its "intenal" and "external" persona.
> Introduce the possibility that a given TCP connection may have _two_
> external personas (personi?) then I suspect all Heck breaks loose and
> Prince Phil will run rufshod over your connectivity.
>

It could be that I'll limit outgoing reply packets to match the
interface the original request came in on - it's not going to be a big
issue, and may be better than fighting MTU issues. The only case were
it could really make a difference is when using rsync to an offsite
location as part of our backup routines - doubling the outward speed
with two lines would be useful here. But in that case, there will be no
MTU telegrams to worry about, since the MTU is the same at each end.

> rick jones
>
> Longing for the day when a stake can be driven through the heart of
> NAT.
>
> NAT, routing, firewall - all different things...even if they are in
> the same box...
>

I can certainly agree that NAT is a pain for these sorts of issues,
certain other kinds of routing problems, and it negates any sort of
IP-based authentication or tracking on external servers. But I think
NAT is a useful part of a firewalling and security strategy - it makes
the machines on the inside invisible to the outside. That's an
inconvenience when you *want* the inside machines to be visible, but in
most cases you do not want them visible. If you do not have NAT, then
all machines on the inside that can connect to the net have globally
visible addresses - that means that they are targets. Sure, your
firewall should block any incoming telegrams aimed at them - but NAT
means that incoming telegrams cannot be aimed in the first place.
That's an extra layer of protection if something should go wrong (like
configuration errors in the firewall, or bugs of some sort). I think in
most cases, that makes NAT a useful security aid. The other type of
NAT, DNAT, is also useful for its flexibility - it allows traffic to be
aimed at the static router, and redirected to different servers as
needed, without the clients needing to know anything about the
configuration (DNS can do similar things in a different way, of course).

David Brown <(E-Mail Removed)> wrote:
> I can certainly agree that NAT is a pain for these sorts of issues,
> certain other kinds of routing problems, and it negates any sort of
> IP-based authentication or tracking on external servers. But I
> think NAT is a useful part of a firewalling and security strategy -
> it makes the machines on the inside invisible to the outside.
> That's an inconvenience when you *want* the inside machines to be
> visible, but in most cases you do not want them visible. If you do
> not have NAT, then all machines on the inside that can connect to
> the net have globally visible addresses - that means that they are
> targets. Sure, your firewall should block any incoming telegrams
> aimed at them - but NAT means that incoming telegrams cannot be
> aimed in the first place. That's an extra layer of protection if
> something should go wrong (like configuration errors in the
> firewall, or bugs of some sort). I think in most cases, that makes
> NAT a useful security aid. The other type of NAT, DNAT, is also
> useful for its flexibility - it allows traffic to be aimed at the
> static router, and redirected to different servers as needed,
> without the clients needing to know anything about the configuration
> (DNS can do similar things in a different way, of course).

Add little to little and soon there will be a great pile

rick jones
--
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones wrote:
> David Brown <(E-Mail Removed)> wrote:
>> I can certainly agree that NAT is a pain for these sorts of issues,
>> certain other kinds of routing problems, and it negates any sort of
>> IP-based authentication or tracking on external servers. But I
>> think NAT is a useful part of a firewalling and security strategy -
>> it makes the machines on the inside invisible to the outside.
>> That's an inconvenience when you *want* the inside machines to be
>> visible, but in most cases you do not want them visible. If you do
>> not have NAT, then all machines on the inside that can connect to
>> the net have globally visible addresses - that means that they are
>> targets. Sure, your firewall should block any incoming telegrams
>> aimed at them - but NAT means that incoming telegrams cannot be
>> aimed in the first place. That's an extra layer of protection if
>> something should go wrong (like configuration errors in the
>> firewall, or bugs of some sort). I think in most cases, that makes
>> NAT a useful security aid. The other type of NAT, DNAT, is also
>> useful for its flexibility - it allows traffic to be aimed at the
>> static router, and redirected to different servers as needed,
>> without the clients needing to know anything about the configuration
>> (DNS can do similar things in a different way, of course).
>
> Add little to little and soon there will be a great pile
>
> rick jones

There is something to be said for that. But at the moment, NAT is the
only practical solution - perhaps one day IPv6 will give us an
alternative. But even within IPv6, I am far from convinced that we
should not use NAT even then for standard clients on a LAN. For servers
in the DMZ, direct addressing would be easier. Of course, maybe my
opinions will change as my network gets more complicated!

mvh.,

David