Linux 2.4 TCP Performance Problem?

Hi, I'm experiencing what seems to be a TCP throughput problem between
several servers running RedHat Linux 7.3 (kernel 2.4.18). I've also
seen the problem with RedHat AS3.0 (kernel 2.4.21EL). The problem
seems to strike randomly, and could affect one or more hosts
simultaneously. The basic gist is as follows:

From Server A, I'm generating a stream of data that is extremely
dynamic in terms of bandwidth, but peaks at around 6Mbit/sec. There
are multiple clients connecting to Server A, each from different
hosts, reading a copy of the data. This often works exactly as
expected for hours, or even days, at a time without any problems.
Suddenly, I'll notice that the data rate has decreased 95% for one or
more of the clients. Establishing a new connection to Server A during
that time shows that the data rate has in fact not changed, but
remains near the 6Mbit/sec mark. Looking at the output of 'netstat -na
| grep <port>' shows that the TCP send/recv buffers are consistently
around 20K for the send buffer on Server A and around 70K for the recv
buffer on the affected client.
Performing an strace on the affected client applications shows that
the application is often blocking in read() - this really shouldn't be
happening if Server A is still generating a 6Mbit/sec stream.

Looking at the output of tcpdump, I see that, up until the problem
occurs, the receiver is advertising a TCP window of 63712 bytes. At
the point the problem occurs, it appears a TCP segment was missed, and
a series of SACKs were sent for the in-flight segments. Eventually,
the sender retransmits the originally missed packet, and continues
from where it left off. However, the receiver's TCP window now
continues to shrink until it reaches zero. At this point, a cyclical
pattern emerges: the sender pauses, the receiver catches up,
re-advertises a non-zero window (typcially around 15298 bytes), the
sender begins transmitting again, and the receiver closes the window
again. Pattern repeats.

Server A and the problematic clients are connected via a Cisco 2950
switch and 100Mbit full duplex ethernet. There are other clients
connected to Server A via a WAN, and they do not exhibit this problem.

If anyone has any suggestions on this, I'd appreciate it! Below is a
sample of the tcpdump capture highlighting the pattern observed:

18:42:01.168059 q1.35790 > fanout1.23000: . ack 124342586 win 63712
<nop,nop,timestamp 58725704 314379527> (DF)
18:42:01.180378 fanout1.23000 > q1.35790: P 124344034:124345052(1018)
ack 1 win 5792 <nop,nop,timestamp 314379536 58725703> (DF)
18:42:01.180402 q1.35790 > fanout1.23000: . ack 124342586 win 63712
<nop,nop,timestamp 58725705 314379527,nop,nop,sack sack 1
{124344034:124345052} > (DF)
18:42:01.191480 fanout1.23000 > q1.35790: . 124345052:124346500(1448)
ack 1 win 5792 <nop,nop,timestamp 314379542 58725703> (DF)
18:42:01.191522 q1.35790 > fanout1.23000: . ack 124342586 win 63712
<nop,nop,timestamp 58725706 314379527,nop,nop,sack sack 1
{124344034:124346500} > (DF)
18:42:01.197660 fanout1.23000 > q1.35790: . 124346500:124347948(1448)
ack 1 win 5792 <nop,nop,timestamp 314379545 58725704> (DF)
18:42:01.197685 q1.35790 > fanout1.23000: . ack 124342586 win 63712
<nop,nop,timestamp 58725707 314379527,nop,nop,sack sack 1
{124344034:124347948} > (DF)
18:42:01.207593 fanout1.23000 > q1.35790: . 124347948:124349396(1448)
ack 1 win 5792 <nop,nop,timestamp 314379551 58725705> (DF)
18:42:01.207664 q1.35790 > fanout1.23000: . ack 124342586 win 63712
<nop,nop,timestamp 58725708 314379527,nop,nop,sack sack 1
{124344034:124349396} > (DF)
18:42:01.224015 fanout1.23000 > q1.35790: . 124342586:124344034(1448)
ack 1 win 5792 <nop,nop,timestamp 314379559 58725707> (DF)
18:42:01.224076 q1.35790 > fanout1.23000: . ack 124349396 win 59368
<nop,nop,timestamp 58725710 314379559> (DF)
18:42:01.473164 fanout1.23000 > q1.35790: . 124349396:124350844(1448)
ack 1 win 5792 <nop,nop,timestamp 314379687 58725710> (DF)
18:42:01.473210 q1.35790 > fanout1.23000: . ack 124350844 win 63712
<nop,nop,timestamp 58725735 314379687> (DF)
18:42:01.904290 fanout1.23000 > q1.35790: . 124349396:124350844(1448)
ack 1 win 5792 <nop,nop,timestamp 314379915 58725710> (DF)
18:42:01.904329 q1.35790 > fanout1.23000: . ack 124350844 win 63712
<nop,nop,timestamp 58725778 314379915,nop,nop,sack sack 1
{124349396:124350844} > (DF)
18:42:01.907279 fanout1.23000 > q1.35790: . 124350844:124352292(1448)
ack 1 win 5792 <nop,nop,timestamp 314379916 58725778> (DF)
18:42:01.907282 fanout1.23000 > q1.35790: . 124352292:124353740(1448)
ack 1 win 5792 <nop,nop,timestamp 314379916 58725778> (DF)
18:42:01.907372 q1.35790 > fanout1.23000: . ack 124352292 win 63712
<nop,nop,timestamp 58725778 314379916> (DF)
18:42:01.907400 q1.35790 > fanout1.23000: . ack 124353740 win 63712
<nop,nop,timestamp 58725778 314379916> (DF)
18:42:01.910474 fanout1.23000 > q1.35790: P 124353740:124355188(1448)
ack 1 win 5792 <nop,nop,timestamp 314379918 58725778> (DF)
18:42:01.910477 fanout1.23000 > q1.35790: . 124355188:124356636(1448)
ack 1 win 5792 <nop,nop,timestamp 314379918 58725778> (DF)
18:42:01.910479 fanout1.23000 > q1.35790: . 124356636:124358084(1448)
ack 1 win 5792 <nop,nop,timestamp 314379918 58725778> (DF)
18:42:01.910558 q1.35790 > fanout1.23000: . ack 124355188 win 63712
<nop,nop,timestamp 58725778 314379918> (DF)
18:42:01.910579 q1.35790 > fanout1.23000: . ack 124356636 win 63712
<nop,nop,timestamp 58725778 314379918> (DF)
18:42:01.910608 q1.35790 > fanout1.23000: . ack 124358084 win 63712
<nop,nop,timestamp 58725778 314379918> (DF)
18:42:01.911682 fanout1.23000 > q1.35790: . 124358084:124359532(1448)
ack 1 win 5792 <nop,nop,timestamp 314379918 58725778> (DF)
18:42:01.911705 q1.35790 > fanout1.23000: . ack 124359532 win 63712
<nop,nop,timestamp 58725778 314379918> (DF)
18:42:01.913206 fanout1.23000 > q1.35790: . 124359532:124360980(1448)
ack 1 win 5792 <nop,nop,timestamp 314379919 58725778> (DF)
18:42:01.913264 q1.35790 > fanout1.23000: . ack 124360980 win 62264
<nop,nop,timestamp 58725779 314379919> (DF)
18:42:01.914435 fanout1.23000 > q1.35790: P 124360980:124362428(1448)
ack 1 win 5792 <nop,nop,timestamp 314379919 58725778> (DF)
18:42:01.914465 q1.35790 > fanout1.23000: . ack 124362428 win 60816
<nop,nop,timestamp 58725779 314379919> (DF)

18:42:02.252533 q1.35790 > fanout1.23000: . ack 124428575 win 9651
<nop,nop,timestamp 58725813 314380073> (DF)
18:42:02.254857 fanout1.23000 > q1.35790: . 124428575:124430023(1448)
ack 1 win 5792 <nop,nop,timestamp 314380094 58725813> (DF)
18:42:02.256297 fanout1.23000 > q1.35790: . 124430023:124431471(1448)
ack 1 win 5792 <nop,nop,timestamp 314380094 58725813> (DF)
18:42:02.256300 fanout1.23000 > q1.35790: P 124431471:124432919(1448)
ack 1 win 5792 <nop,nop,timestamp 314380094 58725813> (DF)
18:42:02.263115 fanout1.23000 > q1.35790: P 124432919:124434271(1352)
ack 1 win 5792 <nop,nop,timestamp 314380098 58725813> (DF)
18:42:02.274586 fanout1.23000 > q1.35790: . 124434271:124435719(1448)
ack 1 win 5792 <nop,nop,timestamp 314380104 58725813> (DF)
18:42:02.281788 fanout1.23000 > q1.35790: . 124435719:124437167(1448)
ack 1 win 5792 <nop,nop,timestamp 314380107 58725813> (DF)
18:42:02.292533 q1.35790 > fanout1.23000: . ack 124437167 win 1059
<nop,nop,timestamp 58725817 314380094> (DF)
18:42:02.523541 fanout1.23000 > q1.35790: P 124437167:124438226(1059)
ack 1 win 5792 <nop,nop,timestamp 314380232 58725817> (DF)
18:42:02.523564 q1.35790 > fanout1.23000: . ack 124438226 win 0
<nop,nop,timestamp 58725840 314380232> (DF)
18:42:02.748036 fanout1.23000 > q1.35790: . ack 1 win 5792
<nop,nop,timestamp 314380348 58725840> (DF)
18:42:02.748054 q1.35790 > fanout1.23000: . ack 124438226 win 0
<nop,nop,timestamp 58725862 314380232> (DF)
18:42:03.197262 fanout1.23000 > q1.35790: . ack 1 win 5792
<nop,nop,timestamp 314380578 58725862> (DF)
18:42:03.197295 q1.35790 > fanout1.23000: . ack 124438226 win 0
<nop,nop,timestamp 58725907 314380232> (DF)
18:42:03.418887 q1.35790 > fanout1.23000: . ack 124438226 win 17376
<nop,nop,timestamp 58725929 314380232> (DF)
18:42:03.420153 fanout1.23000 > q1.35790: P 124438226:124438615(389)
ack 1 win 5792 <nop,nop,timestamp 314380691 58725929> (DF)
18:42:03.420190 q1.35790 > fanout1.23000: . ack 124438615 win 16987
<nop,nop,timestamp 58725929 314380691> (DF)

18:42:03.452555 q1.35790 > fanout1.23000: . ack 124444407 win 11195
<nop,nop,timestamp 58725933 314380691> (DF)
18:42:03.454904 fanout1.23000 > q1.35790: . 124444407:124445855(1448)
ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
18:42:03.455558 fanout1.23000 > q1.35790: . 124445855:124447303(1448)
ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
18:42:03.455561 fanout1.23000 > q1.35790: . 124447303:124448751(1448)
ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
18:42:03.455564 fanout1.23000 > q1.35790: . 124448751:124450199(1448)
ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
18:42:03.456754 fanout1.23000 > q1.35790: . 124450199:124451647(1448)
ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
18:42:03.492533 q1.35790 > fanout1.23000: . ack 124451647 win 3955
<nop,nop,timestamp 58725937 314380709> (DF)
18:42:03.494777 fanout1.23000 > q1.35790: . 124451647:124453095(1448)
ack 1 win 5792 <nop,nop,timestamp 314380729 58725937> (DF)
18:42:03.496203 fanout1.23000 > q1.35790: P 124453095:124454543(1448)
ack 1 win 5792 <nop,nop,timestamp 314380729 58725937> (DF)
18:42:03.532534 q1.35790 > fanout1.23000: . ack 124454543 win 1059
<nop,nop,timestamp 58725941 314380729> (DF)
18:42:03.761249 fanout1.23000 > q1.35790: P 124454543:124455602(1059)
ack 1 win 5792 <nop,nop,timestamp 314380866 58725941> (DF)
18:42:03.761269 q1.35790 > fanout1.23000: . ack 124455602 win 0
<nop,nop,timestamp 58725963 314380866> (DF)
18:42:03.945576 q1.35790 > fanout1.23000: . ack 124455602 win 14480
<nop,nop,timestamp 58725982 314380866> (DF)

18:42:04.362534 q1.35790 > fanout1.23000: . ack 124483503 win 2896
<nop,nop,timestamp 58726024 314381154> (DF)
18:42:04.365217 fanout1.23000 > q1.35790: . 124483503:124484951(1448)
ack 1 win 5792 <nop,nop,timestamp 314381175 58726024> (DF)
18:42:04.365221 fanout1.23000 > q1.35790: P 124484951:124486399(1448)
ack 1 win 5792 <nop,nop,timestamp 314381175 58726024> (DF)
18:42:04.402534 q1.35790 > fanout1.23000: . ack 124486399 win 0
<nop,nop,timestamp 58726028 314381175> (DF)
18:42:04.415966 q1.35790 > fanout1.23000: . ack 124486399 win 15928
<nop,nop,timestamp 58726029 314381175> (DF)
18:42:04.418776 fanout1.23000 > q1.35790: P 124486399:124487847(1448)
ack 1 win 5792 <nop,nop,timestamp 314381202 58726029> (DF)
18:42:04.418779 fanout1.23000 > q1.35790: . 124487847:124489295(1448)
ack 1 win 5792 <nop,nop,timestamp 314381202 58726029> (DF)
18:42:04.418782 fanout1.23000 > q1.35790: . 124489295:124490743(1448)
ack 1 win 5792 <nop,nop,timestamp 314381202 58726029> (DF)
18:42:04.419975 fanout1.23000 > q1.35790: . 124490743:124492191(1448)
ack 1 win 5792 <nop,nop,timestamp 314381202 58726029> (DF)

18:42:04.455817 fanout1.23000 > q1.35790: P 124499431:124500879(1448)
ack 1 win 5792 <nop,nop,timestamp 314381221 58726033> (DF)
18:42:04.455820 fanout1.23000 > q1.35790: . 124500879:124502327(1448)
ack 1 win 5792 <nop,nop,timestamp 314381221 58726033> (DF)
18:42:04.492534 q1.35790 > fanout1.23000: . ack 124502327 win 0
<nop,nop,timestamp 58726037 314381221> (DF)
18:42:04.722056 fanout1.23000 > q1.35790: . ack 1 win 5792
<nop,nop,timestamp 314381359 58726037> (DF)
18:42:04.722084 q1.35790 > fanout1.23000: . ack 124502327 win 0
<nop,nop,timestamp 58726059 314381221> (DF)
18:42:04.751376 q1.35790 > fanout1.23000: . ack 124502327 win 17376
<nop,nop,timestamp 58726062 314381221> (DF)
18:42:04.753887 fanout1.23000 > q1.35790: P 124502327:124503775(1448)
ack 1 win 5792 <nop,nop,timestamp 314381374 58726062> (DF)
18:42:04.753910 q1.35790 > fanout1.23000: . ack 124503775 win 15928
<nop,nop,timestamp 58726063 314381374> (DF)

Has anyone ever experienced a similiar problem? I've measured the link
bandwidth between the affected hosts with iperf and it is > 90Mbit.

Any suggestions?

Thanks,
-Brian

(E-Mail Removed) (Brian Fellion) wrote in message news:<(E-Mail Removed). com>...
> Hi, I'm experiencing what seems to be a TCP throughput problem between
> several servers running RedHat Linux 7.3 (kernel 2.4.18). I've also
> seen the problem with RedHat AS3.0 (kernel 2.4.21EL). The problem
> seems to strike randomly, and could affect one or more hosts
> simultaneously. The basic gist is as follows:
>
> From Server A, I'm generating a stream of data that is extremely
> dynamic in terms of bandwidth, but peaks at around 6Mbit/sec. There
> are multiple clients connecting to Server A, each from different
> hosts, reading a copy of the data. This often works exactly as
> expected for hours, or even days, at a time without any problems.
> Suddenly, I'll notice that the data rate has decreased 95% for one or
> more of the clients. Establishing a new connection to Server A during
> that time shows that the data rate has in fact not changed, but
> remains near the 6Mbit/sec mark. Looking at the output of 'netstat -na
> | grep <port>' shows that the TCP send/recv buffers are consistently
> around 20K for the send buffer on Server A and around 70K for the recv
> buffer on the affected client.
> Performing an strace on the affected client applications shows that
> the application is often blocking in read() - this really shouldn't be
> happening if Server A is still generating a 6Mbit/sec stream.
>
> Looking at the output of tcpdump, I see that, up until the problem
> occurs, the receiver is advertising a TCP window of 63712 bytes. At
> the point the problem occurs, it appears a TCP segment was missed, and
> a series of SACKs were sent for the in-flight segments. Eventually,
> the sender retransmits the originally missed packet, and continues
> from where it left off. However, the receiver's TCP window now
> continues to shrink until it reaches zero. At this point, a cyclical
> pattern emerges: the sender pauses, the receiver catches up,
> re-advertises a non-zero window (typcially around 15298 bytes), the
> sender begins transmitting again, and the receiver closes the window
> again. Pattern repeats.
>
> Server A and the problematic clients are connected via a Cisco 2950
> switch and 100Mbit full duplex ethernet. There are other clients
> connected to Server A via a WAN, and they do not exhibit this problem.
>
> If anyone has any suggestions on this, I'd appreciate it! Below is a
> sample of the tcpdump capture highlighting the pattern observed:
>
> 18:42:01.168059 q1.35790 > fanout1.23000: . ack 124342586 win 63712
> <nop,nop,timestamp 58725704 314379527> (DF)
> 18:42:01.180378 fanout1.23000 > q1.35790: P 124344034:124345052(1018)
> ack 1 win 5792 <nop,nop,timestamp 314379536 58725703> (DF)
> 18:42:01.180402 q1.35790 > fanout1.23000: . ack 124342586 win 63712
> <nop,nop,timestamp 58725705 314379527,nop,nop,sack sack 1
> {124344034:124345052} > (DF)
> 18:42:01.191480 fanout1.23000 > q1.35790: . 124345052:124346500(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379542 58725703> (DF)
> 18:42:01.191522 q1.35790 > fanout1.23000: . ack 124342586 win 63712
> <nop,nop,timestamp 58725706 314379527,nop,nop,sack sack 1
> {124344034:124346500} > (DF)
> 18:42:01.197660 fanout1.23000 > q1.35790: . 124346500:124347948(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379545 58725704> (DF)
> 18:42:01.197685 q1.35790 > fanout1.23000: . ack 124342586 win 63712
> <nop,nop,timestamp 58725707 314379527,nop,nop,sack sack 1
> {124344034:124347948} > (DF)
> 18:42:01.207593 fanout1.23000 > q1.35790: . 124347948:124349396(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379551 58725705> (DF)
> 18:42:01.207664 q1.35790 > fanout1.23000: . ack 124342586 win 63712
> <nop,nop,timestamp 58725708 314379527,nop,nop,sack sack 1
> {124344034:124349396} > (DF)
> 18:42:01.224015 fanout1.23000 > q1.35790: . 124342586:124344034(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379559 58725707> (DF)
> 18:42:01.224076 q1.35790 > fanout1.23000: . ack 124349396 win 59368
> <nop,nop,timestamp 58725710 314379559> (DF)
> 18:42:01.473164 fanout1.23000 > q1.35790: . 124349396:124350844(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379687 58725710> (DF)
> 18:42:01.473210 q1.35790 > fanout1.23000: . ack 124350844 win 63712
> <nop,nop,timestamp 58725735 314379687> (DF)
> 18:42:01.904290 fanout1.23000 > q1.35790: . 124349396:124350844(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379915 58725710> (DF)
> 18:42:01.904329 q1.35790 > fanout1.23000: . ack 124350844 win 63712
> <nop,nop,timestamp 58725778 314379915,nop,nop,sack sack 1
> {124349396:124350844} > (DF)
> 18:42:01.907279 fanout1.23000 > q1.35790: . 124350844:124352292(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379916 58725778> (DF)
> 18:42:01.907282 fanout1.23000 > q1.35790: . 124352292:124353740(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379916 58725778> (DF)
> 18:42:01.907372 q1.35790 > fanout1.23000: . ack 124352292 win 63712
> <nop,nop,timestamp 58725778 314379916> (DF)
> 18:42:01.907400 q1.35790 > fanout1.23000: . ack 124353740 win 63712
> <nop,nop,timestamp 58725778 314379916> (DF)
> 18:42:01.910474 fanout1.23000 > q1.35790: P 124353740:124355188(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379918 58725778> (DF)
> 18:42:01.910477 fanout1.23000 > q1.35790: . 124355188:124356636(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379918 58725778> (DF)
> 18:42:01.910479 fanout1.23000 > q1.35790: . 124356636:124358084(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379918 58725778> (DF)
> 18:42:01.910558 q1.35790 > fanout1.23000: . ack 124355188 win 63712
> <nop,nop,timestamp 58725778 314379918> (DF)
> 18:42:01.910579 q1.35790 > fanout1.23000: . ack 124356636 win 63712
> <nop,nop,timestamp 58725778 314379918> (DF)
> 18:42:01.910608 q1.35790 > fanout1.23000: . ack 124358084 win 63712
> <nop,nop,timestamp 58725778 314379918> (DF)
> 18:42:01.911682 fanout1.23000 > q1.35790: . 124358084:124359532(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379918 58725778> (DF)
> 18:42:01.911705 q1.35790 > fanout1.23000: . ack 124359532 win 63712
> <nop,nop,timestamp 58725778 314379918> (DF)
> 18:42:01.913206 fanout1.23000 > q1.35790: . 124359532:124360980(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379919 58725778> (DF)
> 18:42:01.913264 q1.35790 > fanout1.23000: . ack 124360980 win 62264
> <nop,nop,timestamp 58725779 314379919> (DF)
> 18:42:01.914435 fanout1.23000 > q1.35790: P 124360980:124362428(1448)
> ack 1 win 5792 <nop,nop,timestamp 314379919 58725778> (DF)
> 18:42:01.914465 q1.35790 > fanout1.23000: . ack 124362428 win 60816
> <nop,nop,timestamp 58725779 314379919> (DF)
>
> 18:42:02.252533 q1.35790 > fanout1.23000: . ack 124428575 win 9651
> <nop,nop,timestamp 58725813 314380073> (DF)
> 18:42:02.254857 fanout1.23000 > q1.35790: . 124428575:124430023(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380094 58725813> (DF)
> 18:42:02.256297 fanout1.23000 > q1.35790: . 124430023:124431471(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380094 58725813> (DF)
> 18:42:02.256300 fanout1.23000 > q1.35790: P 124431471:124432919(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380094 58725813> (DF)
> 18:42:02.263115 fanout1.23000 > q1.35790: P 124432919:124434271(1352)
> ack 1 win 5792 <nop,nop,timestamp 314380098 58725813> (DF)
> 18:42:02.274586 fanout1.23000 > q1.35790: . 124434271:124435719(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380104 58725813> (DF)
> 18:42:02.281788 fanout1.23000 > q1.35790: . 124435719:124437167(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380107 58725813> (DF)
> 18:42:02.292533 q1.35790 > fanout1.23000: . ack 124437167 win 1059
> <nop,nop,timestamp 58725817 314380094> (DF)
> 18:42:02.523541 fanout1.23000 > q1.35790: P 124437167:124438226(1059)
> ack 1 win 5792 <nop,nop,timestamp 314380232 58725817> (DF)
> 18:42:02.523564 q1.35790 > fanout1.23000: . ack 124438226 win 0
> <nop,nop,timestamp 58725840 314380232> (DF)
> 18:42:02.748036 fanout1.23000 > q1.35790: . ack 1 win 5792
> <nop,nop,timestamp 314380348 58725840> (DF)
> 18:42:02.748054 q1.35790 > fanout1.23000: . ack 124438226 win 0
> <nop,nop,timestamp 58725862 314380232> (DF)
> 18:42:03.197262 fanout1.23000 > q1.35790: . ack 1 win 5792
> <nop,nop,timestamp 314380578 58725862> (DF)
> 18:42:03.197295 q1.35790 > fanout1.23000: . ack 124438226 win 0
> <nop,nop,timestamp 58725907 314380232> (DF)
> 18:42:03.418887 q1.35790 > fanout1.23000: . ack 124438226 win 17376
> <nop,nop,timestamp 58725929 314380232> (DF)
> 18:42:03.420153 fanout1.23000 > q1.35790: P 124438226:124438615(389)
> ack 1 win 5792 <nop,nop,timestamp 314380691 58725929> (DF)
> 18:42:03.420190 q1.35790 > fanout1.23000: . ack 124438615 win 16987
> <nop,nop,timestamp 58725929 314380691> (DF)
>
> 18:42:03.452555 q1.35790 > fanout1.23000: . ack 124444407 win 11195
> <nop,nop,timestamp 58725933 314380691> (DF)
> 18:42:03.454904 fanout1.23000 > q1.35790: . 124444407:124445855(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
> 18:42:03.455558 fanout1.23000 > q1.35790: . 124445855:124447303(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
> 18:42:03.455561 fanout1.23000 > q1.35790: . 124447303:124448751(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
> 18:42:03.455564 fanout1.23000 > q1.35790: . 124448751:124450199(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
> 18:42:03.456754 fanout1.23000 > q1.35790: . 124450199:124451647(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380709 58725933> (DF)
> 18:42:03.492533 q1.35790 > fanout1.23000: . ack 124451647 win 3955
> <nop,nop,timestamp 58725937 314380709> (DF)
> 18:42:03.494777 fanout1.23000 > q1.35790: . 124451647:124453095(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380729 58725937> (DF)
> 18:42:03.496203 fanout1.23000 > q1.35790: P 124453095:124454543(1448)
> ack 1 win 5792 <nop,nop,timestamp 314380729 58725937> (DF)
> 18:42:03.532534 q1.35790 > fanout1.23000: . ack 124454543 win 1059
> <nop,nop,timestamp 58725941 314380729> (DF)
> 18:42:03.761249 fanout1.23000 > q1.35790: P 124454543:124455602(1059)
> ack 1 win 5792 <nop,nop,timestamp 314380866 58725941> (DF)
> 18:42:03.761269 q1.35790 > fanout1.23000: . ack 124455602 win 0
> <nop,nop,timestamp 58725963 314380866> (DF)
> 18:42:03.945576 q1.35790 > fanout1.23000: . ack 124455602 win 14480
> <nop,nop,timestamp 58725982 314380866> (DF)
>
> 18:42:04.362534 q1.35790 > fanout1.23000: . ack 124483503 win 2896
> <nop,nop,timestamp 58726024 314381154> (DF)
> 18:42:04.365217 fanout1.23000 > q1.35790: . 124483503:124484951(1448)
> ack 1 win 5792 <nop,nop,timestamp 314381175 58726024> (DF)
> 18:42:04.365221 fanout1.23000 > q1.35790: P 124484951:124486399(1448)
> ack 1 win 5792 <nop,nop,timestamp 314381175 58726024> (DF)
> 18:42:04.402534 q1.35790 > fanout1.23000: . ack 124486399 win 0
> <nop,nop,timestamp 58726028 314381175> (DF)
> 18:42:04.415966 q1.35790 > fanout1.23000: . ack 124486399 win 15928
> <nop,nop,timestamp 58726029 314381175> (DF)
> 18:42:04.418776 fanout1.23000 > q1.35790: P 124486399:124487847(1448)
> ack 1 win 5792 <nop,nop,timestamp 314381202 58726029> (DF)
> 18:42:04.418779 fanout1.23000 > q1.35790: . 124487847:124489295(1448)
> ack 1 win 5792 <nop,nop,timestamp 314381202 58726029> (DF)
> 18:42:04.418782 fanout1.23000 > q1.35790: . 124489295:124490743(1448)
> ack 1 win 5792 <nop,nop,timestamp 314381202 58726029> (DF)
> 18:42:04.419975 fanout1.23000 > q1.35790: . 124490743:124492191(1448)
> ack 1 win 5792 <nop,nop,timestamp 314381202 58726029> (DF)
>
> 18:42:04.455817 fanout1.23000 > q1.35790: P 124499431:124500879(1448)
> ack 1 win 5792 <nop,nop,timestamp 314381221 58726033> (DF)
> 18:42:04.455820 fanout1.23000 > q1.35790: . 124500879:124502327(1448)
> ack 1 win 5792 <nop,nop,timestamp 314381221 58726033> (DF)
> 18:42:04.492534 q1.35790 > fanout1.23000: . ack 124502327 win 0
> <nop,nop,timestamp 58726037 314381221> (DF)
> 18:42:04.722056 fanout1.23000 > q1.35790: . ack 1 win 5792
> <nop,nop,timestamp 314381359 58726037> (DF)
> 18:42:04.722084 q1.35790 > fanout1.23000: . ack 124502327 win 0
> <nop,nop,timestamp 58726059 314381221> (DF)
> 18:42:04.751376 q1.35790 > fanout1.23000: . ack 124502327 win 17376
> <nop,nop,timestamp 58726062 314381221> (DF)
> 18:42:04.753887 fanout1.23000 > q1.35790: P 124502327:124503775(1448)
> ack 1 win 5792 <nop,nop,timestamp 314381374 58726062> (DF)
> 18:42:04.753910 q1.35790 > fanout1.23000: . ack 124503775 win 15928
> <nop,nop,timestamp 58726063 314381374> (DF)