Path Maximum Transmission Unit Discovery

I have recently been bitten by a problem with Path Maximum Transmission Unit
Discovery, and have been having major problems with email, usenet and
browsing the world wide web. I have now resolved these by disabling PMTUD
on my computers.

However, I suspect that there is a problem with the way that this works on
my LAN. I am using Ethernet, and I have two computers side by side on a
switch. I suspect that PMTUD is not working between them.

Both computers are running Debian (Squeeze/Testing).

I know that I could use tcpdump to examine the packets, and try to force
connections from one side to the other to examine this, but I was wondering
if there is any ready made test suite available.

I am thinking that maybe there is a piece of software that I can run on
both sides that tries to transmit packets of various sizes, and displays
on the screen the size of the packets sent, and the size of packets it
receives together with an interpretation of the Maximum Transmission Unit
as it stands and the Maximum Segment Size values being transmitted and
received by the host.

Could Linux be trying to use a Maximum Transmission Unit above 1500? I am
using Ethernet, so I don't ever want this to happen.

Mark.

--
Mark Hobley
Linux User: #370818 http://markhobley.yi.org/

On Sun, 2010-09-19, Mark Hobley wrote:
> I have recently been bitten by a problem with Path Maximum Transmission Unit
> Discovery, and have been having major problems with email, usenet and
> browsing the world wide web. I have now resolved these by disabling PMTUD
> on my computers.
>
> However, I suspect that there is a problem with the way that this works on
> my LAN. I am using Ethernet, and I have two computers side by side on a
> switch. I suspect that PMTUD is not working between them.

There are no routers inbetween them, so I cannot see how PMTU
discovery can be involved. Unless one of them /is/ your router?

> Both computers are running Debian (Squeeze/Testing).
>
> I know that I could use tcpdump to examine the packets, and try to force
> connections from one side to the other to examine this, but I was wondering
> if there is any ready made test suite available.
>
> I am thinking that maybe there is a piece of software that I can run on
> both sides that tries to transmit packets of various sizes, and displays
> on the screen the size of the packets sent, and the size of packets it
> receives together with an interpretation of the Maximum Transmission Unit
> as it stands and the Maximum Segment Size values being transmitted and
> received by the host.

There could be such software I suppose, but something like this should
also help:

- enable the echo and discard services in inetd on one of the machines
- generate traffic using
perl -e 'print "x" x 1500' | nc other-host echo
perl -e 'print "x" x 1500' | nc other-host discard
- try different lengths, and look at it with tcpdump

> Could Linux be trying to use a Maximum Transmission Unit above 1500? I am
> using Ethernet, so I don't ever want this to happen.

Ask tcpdump instead. It suspect it says "no" ...

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

Mark Hobley wrote:
> I have recently been bitten by a problem with Path Maximum Transmission Unit
> Discovery, and have been having major problems with email, usenet and
> browsing the world wide web. I have now resolved these by disabling PMTUD
> on my computers.
>
> However, I suspect that there is a problem with the way that this works on
> my LAN. I am using Ethernet, and I have two computers side by side on a
> switch. I suspect that PMTUD is not working between them.


A common cause for misbehaving PMTU discovery is
that all ICMP packets or responses are disabled
in the firewall (in a vain attempt to hide the
computer from ping).

--

Tauno Voipio
tauno voipio (at) iki fi

On Sun, 19 Sep 2010 21:13:27 +0300, Tauno Voipio wrote:

> A common cause for misbehaving PMTU discovery is that all ICMP packets
> or responses are disabled in the firewall (in a vain attempt to hide the
> computer from ping).

Yeah. There is no firewall between the two hosts and they can ping each
other.

Mark.

--
Mark Hobley
Linux User: #370818 http://markhobley.yi.org/

Mark Hobley <(E-Mail Removed)> wrote:
> Yeah. There is no firewall between the two hosts and they can ping
> each other.

Is there actually a router between them or are they talking to one
another "directly?"

PathMTU discovery requires a router between two nodes. Perhaps two.

Node1 --BigMTUNet-- Router1-- SmallMTUNet -- Router2 --BigMTUNet Node2

At the risk of restating the known...

A large IP datagram sent by Node1 to Node2 hits the router. The when
the router goes to transmit it on SmallMTUNet, it sees the datagram is
larger than the MTU of SmallMTUNet. It also sees the DF bit is set.
It will then drop the datagram and send an ICMP message to Node1,
telling Node1 it dropped the datagram because it was too large to be
forwarded without further fragmentation and the DF bit was set. It
also includes what size could be sent without further fragmentation.

Why two routers? First a bit about MTUs in a local broadcast domain
(that part of a network all on the same side of a router). Broadly
speaking, all nodes in the same broadcast domain must have the same
MTU. Otherwise, a node with a large MTU might try to send a frame to
a node with a small MTU and the node with the small MTU will drop it
(if some switch along the way didn't drop it first) - it is not setup
to accept the larger frame size.

Now, if the communication is TCP, the exchange of MSS (Maximum Segment
Size) at the beginning of the communication will cover the backside of
whomever configured multiple MTUs in the broadcast domain. One side
will send an MSS based on the large MTU, the other will send an MSS
based on its smaller MTU. The smaller MSS is the one used. However,
UDP communication does not have that MSS exchange. The system with
the large MTU just sends to the system with the small and we are back
to packets dropped.

Now, if there is only one router between them, and the comms are TCP,
we still have the MSS exchange and there will be no need for
fragmentation by the router and no PathMTU discovery happens. If the
comms are UDP, then the router (which must have a large MTU configured
on the large MTU side (broadcast domain) will generate the ICMP's per
some of the initial text above.

Now, if there are two routers, and the broadcast domain joining the
routers is the one with the small MTU, and the ends have the big MTU
(aka a "dumbbell network") then UDP behaves as before. Now the TCP
MSS exchange as large MSSes in both directions, and so the TCP traffic
too will need framentation and will trigger PathMTU by the router (I'm
assuming DF is set in the IP header for all this)

rick jones
--
web2.0 n, the dot.com reunion tour...
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Hello,

Mark Hobley a écrit :
>
> Could Linux be trying to use a Maximum Transmission Unit above 1500? I am
> using Ethernet

Yes, if jumbo frames are enabled.

On Wed, 22 Sep 2010 22:16:15 +0000, Rick Jones wrote:

> Is there actually a router between them or are they talking to one
> another "directly?"

There is no router as such. There is a hub between them, and they are both
on the same LAN segment.

> PathMTU discovery requires a router between two nodes. Perhaps two.

Yeah. There is no router. I don't know why PMTUD is effective here, but I
reckon the kernel is making it effective.

> First a bit about MTUs in a local broadcast domain
> (that part of a network all on the same side of a router). Broadly
> speaking, all nodes in the same broadcast domain must have the same MTU.
> Otherwise, a node with a large MTU might try to send a frame to a node
> with a small MTU and the node with the small MTU will drop it (if some
> switch along the way didn't drop it first) - it is not setup to accept
> the larger frame size.
>
> Now, if the communication is TCP, the exchange of MSS (Maximum Segment
> Size) at the beginning of the communication will cover the backside of
> whomever configured multiple MTUs in the broadcast domain. One side
> will send an MSS based on the large MTU, the other will send an MSS
> based on its smaller MTU. The smaller MSS is the one used.

Does that exchange always take place in Linux?

I have configured the MTU by using a parameter in the /etc/network/interfaces
file on both hosts as follows:

iface eth0 inet static
address 10.0.0.101
netmask 255.255.255.0
broadcast 10.0.0.255
gateway 10.0.0.1
mtu 1500

I don't need MTU discovery here. I know that my MSS is 1500, and I never want
this limit to be exceeded.

Incidentally, there is a router on this network for externally bound traffic
(which is also connected via ethernet). If a host external to the router
(such as an internet based host), indicates that it has a different MSS
size, then surely this is a concern between the router and the foreign host
(and possibly a concern for the individual application running on the LAN side
that is communicating with the foreign host). The application that is
communicating to the foreign host, may use a smaller segment size for its
communications to the foreign host, but this segment size should not affect
the segment sizes of packets to other hosts (such as local hosts, where I
still require the 1500 segment size to remain in effect.)

Mark.

--
Mark Hobley
Linux User: #370818 http://markhobley.yi.org/

On Sat, 25 Sep 2010 18:53:41 +0000, Mark Hobley wrote:

> The application that is communicating to the foreign host, may use a
> smaller segment size for its communications to the foreign host

Actually, this raises an interesting question: Supposing that I am using
HTTP protocol, via a remote proxy to a webserver as follows:

Machine A Machine B Machine C
Browser HTTP Proxy Web Server

Machines A and B above have a Maximum Segment Size of 1500. However, the
web server machine (C) has a smaller Maximum Segment Size (lets say 512).

Does Machine B tell Machine A that the Maximum Segment Size is 512, so that
queries from the browser are enclosed in segments of that size? Or does the
browser still send its queries in segments of 1500, and the proxy server
fragments them into smaller 512 segments? If the proxy server is fragmenting
them, then presumably this defeats the object of using PMTUD in the first
place, in which case, wouldn't it be better to not bother with PMTUD?

Mark.

--
Mark Hobley
Linux User: #370818 http://markhobley.yi.org/

On Sun, 26 Sep 2010 12:23:58 +0200, Ersek, Laszlo wrote:

> The discovery and fragmentation work in the IP layer. The HTTP proxy
> works in the TCP (and application) layers. Once Machine B is reached,
> the IP layer's job is done. That job recommences towards Machine C.

> Machine A and Machine B are constant links in the above chain. Machine B
> connects to lots of different HTTP servers (or further proxies) scattered
> all over the net. Machine C is variable, and thus each type of B-C link is
> different. Therefore it makes sense for the discovered A-B and B-C PMTU's
> to differ and to have different lifetimes.

If Machine B is receiving data from Machine A in 1500 byte segments, and
then retransmitting this in smaller segments to suit C, then PMTUD has not
really prevented fragmentation. Wouldn't it have been better to just do away
with PMTUD and let the fragmentation and reassembly take place downstream
towards C?

What benefit do we get from Machine B knowing the Maximum Segment Size for
each of the target hosts?

Incidentally, if Machine B is a Linux machine, does it actually retain the
MSS for each host it connects to? Say, for example Machine B has just
finished communicating with the Yahoo web server using a MSS of 512 bytes.
If a client machine wants to communicate with Yahoo again 5 minutes later,
via the Machine B proxy, does Machine B remember that it needs to use an MSS
of 512 bytes for Yahoo, or does it need to go through the process of PMTUD
again? How many times does Linux set the DF flag for PMTUD in an HTTP
session? Is it at the start of every request? If so, then that seems like an
awful lot of PMTUD going on. Do the benefits of this outweigh the cost of
downstream fragmentation and assembly?

Mark.

--
Mark Hobley
Linux User: #370818 http://markhobley.yi.org/

On Sun, 26 Sep 2010 12:21:16 +0000, Mark Hobley wrote:

> On Sun, 26 Sep 2010 12:23:58 +0200, Ersek, Laszlo wrote:
>
>> The discovery and fragmentation work in the IP layer. The HTTP proxy
>> works in the TCP (and application) layers. Once Machine B is reached,
>> the IP layer's job is done. That job recommences towards Machine C.
>
>> Machine A and Machine B are constant links in the above chain. Machine
>> B connects to lots of different HTTP servers (or further proxies)
>> scattered all over the net. Machine C is variable, and thus each type
>> of B-C link is different. Therefore it makes sense for the discovered
>> A-B and B-C PMTU's to differ and to have different lifetimes.
>
> If Machine B is receiving data from Machine A in 1500 byte segments, and
> then retransmitting this in smaller segments to suit C, then PMTUD has
> not really prevented fragmentation. Wouldn't it have been better to just
> do away with PMTUD and let the fragmentation and reassembly take place
> downstream towards C?

No, no, those are different TCP connections. They have nothing to do with
each other. They are separate connections with their own session
identifiers, their own endpoints. The only connection between them is
that the connection from B to C is initiated as a result from the
connection from A to B, but that is all they have to do with each other.

Let's say a user telnets from A to B. Another user telnets from B to C.
Would you expect the MSSes from these connections to influence each
other? Of course not. Yet this is exactly the same scenario, two separate
TCP connections, one from A to B and one from B to C.

HTH,
M4