Odd TCP client/server throughput problem

Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
machine patched after install from CDs.

Communicating over: GigE LAN

1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
2) Server consistently has TCP RECV-Q of 0.
3) Client (sender) consistently shows a TCP SEND-Q of 80K.
4) Socket is up and connection is ESTABLISHED from both sides.
5) No data is transmitted.

To troubleshoot, I've torn down and re-established to connection
countless times. There may be a trickle of data initially, but
within a few seconds the client SEND-Q builds and transmission stops.
Receiver's window size never goes below 32K.

Never seen this kind of behavior before. If the server process was
slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.

Richard Eich wrote:
> Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> machine patched after install from CDs.
>
> Communicating over: GigE LAN
>
> 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> 2) Server consistently has TCP RECV-Q of 0.
> 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> 4) Socket is up and connection is ESTABLISHED from both sides.
> 5) No data is transmitted.
>
> To troubleshoot, I've torn down and re-established to connection
> countless times. There may be a trickle of data initially, but
> within a few seconds the client SEND-Q builds and transmission stops.
> Receiver's window size never goes below 32K.
>
> Never seen this kind of behavior before. If the server process was
> slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.

I would use tcpdump or wireshark/ethereal to see what is being sent and
received.

(E-Mail Removed) wrote...
>
> Richard Eich wrote:
> > Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> > machine patched after install from CDs.
> >
> > Communicating over: GigE LAN
> >
> > 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> > 2) Server consistently has TCP RECV-Q of 0.
> > 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> > 4) Socket is up and connection is ESTABLISHED from both sides.
> > 5) No data is transmitted.
> >
> > To troubleshoot, I've torn down and re-established to connection
> > countless times. There may be a trickle of data initially, but
> > within a few seconds the client SEND-Q builds and transmission stops.
> > Receiver's window size never goes below 32K.
> >
> > Never seen this kind of behavior before. If the server process was
> > slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.
>
> I would use tcpdump or wireshark/ethereal to see what is being sent and
> received.

How do you think I know the TCP RWIN size for the server?

Richard Eich wrote:
> (E-Mail Removed) wrote...
> >
> > Richard Eich wrote:
> > > Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> > > machine patched after install from CDs.
> > >
> > > Communicating over: GigE LAN
> > >
> > > 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> > > 2) Server consistently has TCP RECV-Q of 0.
> > > 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> > > 4) Socket is up and connection is ESTABLISHED from both sides.
> > > 5) No data is transmitted.
> > >
> > > To troubleshoot, I've torn down and re-established to connection
> > > countless times. There may be a trickle of data initially, but
> > > within a few seconds the client SEND-Q builds and transmission stops.
> > > Receiver's window size never goes below 32K.
> > >
> > > Never seen this kind of behavior before. If the server process was
> > > slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.
> >
> > I would use tcpdump or wireshark/ethereal to see what is being sent and
> > received.
>
> How do you think I know the TCP RWIN size for the server?

So, what does tcpdump shows you before the transfer stops?

Richard Eich wrote:
> (E-Mail Removed) wrote...
> >
> > Richard Eich wrote:
> > > Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> > > machine patched after install from CDs.
> > >
> > > Communicating over: GigE LAN
> > >
> > > 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> > > 2) Server consistently has TCP RECV-Q of 0.
> > > 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> > > 4) Socket is up and connection is ESTABLISHED from both sides.
> > > 5) No data is transmitted.
> > >
> > > To troubleshoot, I've torn down and re-established to connection
> > > countless times. There may be a trickle of data initially, but
> > > within a few seconds the client SEND-Q builds and transmission stops.
> > > Receiver's window size never goes below 32K.
> > >
> > > Never seen this kind of behavior before. If the server process was
> > > slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.
> >
> > I would use tcpdump or wireshark/ethereal to see what is being sent and
> > received.
>
> How do you think I know the TCP RWIN size for the server?

Is this the same problem?
http://groups.google.com/group/comp....55a65936967a36

(E-Mail Removed) wrote...
>
> Richard Eich wrote:
> > (E-Mail Removed) wrote...
> > >
> > > Richard Eich wrote:
> > > > Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> > > > machine patched after install from CDs.
> > > >
> > > > Communicating over: GigE LAN
> > > >
> > > > 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> > > > 2) Server consistently has TCP RECV-Q of 0.
> > > > 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> > > > 4) Socket is up and connection is ESTABLISHED from both sides.
> > > > 5) No data is transmitted.
> > > >
> > > > To troubleshoot, I've torn down and re-established to connection
> > > > countless times. There may be a trickle of data initially, but
> > > > within a few seconds the client SEND-Q builds and transmission stops.
> > > > Receiver's window size never goes below 32K.
> > > >
> > > > Never seen this kind of behavior before. If the server process was
> > > > slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.
> > >
> > > I would use tcpdump or wireshark/ethereal to see what is being sent and
> > > received.
> >
> > How do you think I know the TCP RWIN size for the server?
>
> So, what does tcpdump shows you before the transfer stops?

SYN =>
SYN ACK <=
SYN ACK =>

then either nothing, or

Packet =>
ACK <= RWIN 32K // immediate reply
Packet =>
ACK <= RWIN 32K // immediate reply
....
nothing. No shrinking window, no slowdown in the rate of ACKs.

--
Taxes are not "punishment for success". Nor are they "theft". Taxes
are a royalty paid commensurate to the economic benefit obtained from
a shared socio-economic system.

"Those who gain the benefit should also bear the disadvantage."
- Common Law maxim

(E-Mail Removed) wrote...
>
> Richard Eich wrote:
> > (E-Mail Removed) wrote...
> > >
> > > Richard Eich wrote:
> > > > Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> > > > machine patched after install from CDs.
> > > >
> > > > Communicating over: GigE LAN
> > > >
> > > > 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> > > > 2) Server consistently has TCP RECV-Q of 0.
> > > > 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> > > > 4) Socket is up and connection is ESTABLISHED from both sides.
> > > > 5) No data is transmitted.
> > > >
> > > > To troubleshoot, I've torn down and re-established to connection
> > > > countless times. There may be a trickle of data initially, but
> > > > within a few seconds the client SEND-Q builds and transmission stops.
> > > > Receiver's window size never goes below 32K.
> > > >
> > > > Never seen this kind of behavior before. If the server process was
> > > > slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.
> > >
> > > I would use tcpdump or wireshark/ethereal to see what is being sent and
> > > received.
> >
> > How do you think I know the TCP RWIN size for the server?
>
> Is this the same problem?
> http://groups.google.com/group/comp....55a65936967a36

Yes, except that we now have a minimal case (described above) that
exhibits the behavior without what we have learned since apparently
are red herrings.

Simple Simon wrote:
> (E-Mail Removed) wrote...
> >
> > Richard Eich wrote:
> > > (E-Mail Removed) wrote...
> > > >
> > > > Richard Eich wrote:
> > > > > Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> > > > > machine patched after install from CDs.
> > > > >
> > > > > Communicating over: GigE LAN
> > > > >
> > > > > 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> > > > > 2) Server consistently has TCP RECV-Q of 0.
> > > > > 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> > > > > 4) Socket is up and connection is ESTABLISHED from both sides.
> > > > > 5) No data is transmitted.
> > > > >
> > > > > To troubleshoot, I've torn down and re-established to connection
> > > > > countless times. There may be a trickle of data initially, but
> > > > > within a few seconds the client SEND-Q builds and transmission stops.
> > > > > Receiver's window size never goes below 32K.
> > > > >
> > > > > Never seen this kind of behavior before. If the server process was
> > > > > slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.
> > > >
> > > > I would use tcpdump or wireshark/ethereal to see what is being sent and
> > > > received.
> > >
> > > How do you think I know the TCP RWIN size for the server?
> >
> > So, what does tcpdump shows you before the transfer stops?
>
> SYN =>
> SYN ACK <=
> SYN ACK =>
>
> then either nothing, or
>
> Packet =>
> ACK <= RWIN 32K // immediate reply
> Packet =>
> ACK <= RWIN 32K // immediate reply
> ...
> nothing. No shrinking window, no slowdown in the rate of ACKs.

tcpdump output might provide more details.

strace'ing the sender while reproducing the problem may give you a
clue. You may like to find out what the sending thread is doing or
where it is blocked when the data transfer stops.

(E-Mail Removed) wrote...
>
> Richard Eich wrote:
> > (E-Mail Removed) wrote...
> > >
> > > Richard Eich wrote:
> > > > Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> > > > machine patched after install from CDs.
> > > >
> > > > Communicating over: GigE LAN
> > > >
> > > > 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> > > > 2) Server consistently has TCP RECV-Q of 0.
> > > > 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> > > > 4) Socket is up and connection is ESTABLISHED from both sides.
> > > > 5) No data is transmitted.
> > > >
> > > > To troubleshoot, I've torn down and re-established to connection
> > > > countless times. There may be a trickle of data initially, but
> > > > within a few seconds the client SEND-Q builds and transmission stops.
> > > > Receiver's window size never goes below 32K.
> > > >
> > > > Never seen this kind of behavior before. If the server process was
> > > > slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.
> > >
> > > I would use tcpdump or wireshark/ethereal to see what is being sent and
> > > received.
> >
> > How do you think I know the TCP RWIN size for the server?
>
> So, what does tcpdump shows you before the transfer stops?

Looks like we're getting a lot of retransmissions, which I believe
would clearly explain the full sender tcp send-q, the receiver empty
tcp recv-q, and the receiver's normal receive window size.

(Relative ACKs used for clarity).

|Time | 192.168.75.100 | 192.168.76.15 |
|0.000 | PSH, ACK - Len: 4 |Seq = 0 Ack = 0
| |(43349) ------------------> (9550) |
|0.000 | PSH, ACK - Len: 1448 |Seq = 4 Ack = 0
| |(43349) ------------------> (9550) |
|0.000 | ACK | |Seq = 0 Ack = 4
| |(43349) <------------------ (9550) |
|0.000 | ACK - Len: 1448 |Seq = 1452 Ack = 0
| |(43349) ------------------> (9550) |
|0.000 | ACK - Len: 1448 |Seq = 2900 Ack = 0
| |(43349) ------------------> (9550) |
|0.001 | ACK | |Seq = 0 Ack = 1452
| |(43349) <------------------ (9550) |
|0.001 | ACK - Len: 1448 |Seq = 4348 Ack = 0
| |(43349) ------------------> (9550) |
|0.002 | ACK | |Seq = 0 Ack = 2900
| |(43349) <------------------ (9550) |
|0.002 | PSH, ACK - Len: 1448 |Seq = 5796 Ack = 0
| |(43349) ------------------> (9550) |
|0.002 | ACK | |Seq = 0 Ack = 4348
| |(43349) <------------------ (9550) |
|0.002 | ACK - Len: 1448 |Seq = 7244 Ack = 0
| |(43349) ------------------> (9550) |
|0.002 | ACK | |Seq = 0 Ack = 5796
| |(43349) <------------------ (9550) |
|0.003 | ACK - Len: 1448 |Seq = 8692 Ack = 0
| |(43349) ------------------> (9550) |
|0.003 | ACK - Len: 1448 |Seq = 10140 Ack =
0
| |(43349) ------------------> (9550) |
|0.003 | ACK | |Seq = 0 Ack = 7244
| |(43349) <------------------ (9550) |
|0.003 | ACK - Len: 1448 |Seq = 11588 Ack =
0
| |(43349) ------------------> (9550) |
|0.003 | ACK | |Seq = 0 Ack = 8692
| |(43349) <------------------ (9550) |
|0.003 | ACK - Len: 1448 |Seq = 13036 Ack =
0
| |(43349) ------------------> (9550) |
|0.004 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.004 | PSH, ACK - Len: 1448 |Seq = 14484 Ack =
0
| |(43349) ------------------> (9550) |
|0.004 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.004 | ACK - Len: 1448 |Seq = 15932 Ack =
0
| |(43349) ------------------> (9550) |
|0.004 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.004 | ACK - Len: 1448 |Seq = 17380 Ack =
0
| |(43349) ------------------> (9550) |
|0.005 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.005 | ACK - Len: 1448 |Seq = 10140 Ack =
0
| |(43349) ------------------> (9550) |
|0.005 | ACK | |Seq = 0 Ack =
10140
| |(43349) <------------------ (9550) |
|0.006 | ACK | |Seq = 0 Ack =
14484
| |(43349) <------------------ (9550) |
|0.006 | ACK - Len: 1448 |Seq = 18828 Ack =
0
| |(43349) ------------------> (9550) |
|0.007 | ACK | |Seq = 0 Ack =
14484
| |(43349) <------------------ (9550) |
|0.007 | PSH, ACK - Len: 1448 |Seq = 14484 Ack =
0
| |(43349) ------------------> (9550) |
|0.008 | ACK | |Seq = 0 Ack =
20276
| |(43349) <------------------ (9550) |
|0.008 | ACK - Len: 1448 |Seq = 20276 Ack =
0
| |(43349) ------------------> (9550) |
|0.010 | ACK | |Seq = 0 Ack =
21724
| |(43349) <------------------ (9550) |
|0.010 | ACK - Len: 1448 |Seq = 21724 Ack =
0
| |(43349) ------------------> (9550) |
|0.010 | ACK - Len: 1448 |Seq = 23172 Ack =
0
| |(43349) ------------------> (9550) |
|0.011 | ACK | |Seq = 0 Ack =
23172
| |(43349) <------------------ (9550) |
|0.011 | ACK - Len: 1448 |Seq = 24620 Ack =
0
| |(43349) ------------------> (9550) |
|0.011 | ACK - Len: 1448 |Seq = 26068 Ack =
0
| |(43349) ------------------> (9550) |
|0.011 | ACK | |Seq = 0 Ack =
24620
| |(43349) <------------------ (9550) |
|0.011 | ACK - Len: 1448 |Seq = 27516 Ack =
0
| |(43349) ------------------> (9550) |
|0.012 | ACK | |Seq = 0 Ack =
24620
| |(43349) <------------------ (9550) |
|0.012 | ACK - Len: 1448 |Seq = 28964 Ack =
0
| |(43349) ------------------> (9550) |
|0.012 | ACK | |Seq = 0 Ack =
24620
| |(43349) <------------------ (9550) |
|0.012 | ACK - Len: 1448 |Seq = 30412 Ack =
0
| |(43349) ------------------> (9550) |
|0.013 | ACK | |Seq = 0 Ack =
24620
| |(43349) <------------------ (9550) |
|0.013 | ACK - Len: 1448 |Seq = 24620 Ack =
0
| |(43349) ------------------> (9550) |
|0.014 | ACK | |Seq = 0 Ack =
28964
| |(43349) <------------------ (9550) |
|0.216 | ACK - Len: 1448 |Seq = 28964 Ack =
0
| |(43349) ------------------> (9550) |
|0.218 | ACK | |Seq = 0 Ack =
31860
| |(43349) <------------------ (9550) |
|0.218 | ACK - Len: 1448 |Seq = 31860 Ack =
0
| |(43349) ------------------> (9550) |
|0.218 | ACK - Len: 1448 |Seq = 33308 Ack =
0
| |(43349) ------------------> (9550) |
|0.219 | ACK | |Seq = 0 Ack =
33308
| |(43349) <------------------ (9550) |
|0.219 | ACK - Len: 1448 |Seq = 34756 Ack =
0
| |(43349) ------------------> (9550) |
|0.219 | ACK - Len: 1448 |Seq = 36204 Ack =
0
| |(43349) ------------------> (9550) |
|0.219 | ACK | |Seq = 0 Ack =
34756
| |(43349) <------------------ (9550) |
|0.219 | ACK - Len: 1448 |Seq = 37652 Ack =
0
| |(43349) ------------------> (9550) |
|0.220 | ACK | |Seq = 0 Ack =
34756
| |(43349) <------------------ (9550) |
|0.220 | ACK - Len: 1448 |Seq = 39100 Ack =
0
| |(43349) ------------------> (9550) |
|0.220 | ACK | |Seq = 0 Ack =
34756
| |(43349) <------------------ (9550) |
|0.220 | ACK - Len: 1448 |Seq = 40548 Ack =
0
| |(43349) ------------------> (9550) |
|0.221 | ACK | |Seq = 0 Ack =
34756
| |(43349) <------------------ (9550) |
|0.221 | ACK - Len: 1448 |Seq = 34756 Ack =
0
| |(43349) ------------------> (9550) |
|0.222 | ACK | |Seq = 0 Ack =
39100
| |(43349) <------------------ (9550) |
|0.424 | ACK - Len: 1448 |Seq = 39100 Ack =
0
| |(43349) ------------------> (9550) |
|0.426 | ACK | |Seq = 0 Ack =
41996
| |(43349) <------------------ (9550) |
|0.426 | ACK - Len: 1448 |Seq = 41996 Ack =
0
| |(43349) ------------------> (9550) |
|0.426 | ACK - Len: 1448 |Seq = 43444 Ack =
0
| |(43349) ------------------> (9550) |
|0.427 | ACK | |Seq = 0 Ack =
43444
| |(43349) <------------------ (9550) |
|0.427 | ACK - Len: 1448 |Seq = 44892 Ack =
0
| |(43349) ------------------> (9550) |
|0.427 | ACK - Len: 1448 |Seq = 46340 Ack =
0
| |(43349) ------------------> (9550) |
|0.427 | ACK | |Seq = 0 Ack =
44892
| |(43349) <------------------ (9550) |
|0.427 | ACK - Len: 1448 |Seq = 47788 Ack =
0
| |(43349) ------------------> (9550) |
|0.428 | ACK | |Seq = 0 Ack =
44892
| |(43349) <------------------ (9550) |
|0.428 | PSH, ACK - Len: 1448 |Seq = 49236 Ack =
0
| |(43349) ------------------> (9550) |
|0.428 | ACK | |Seq = 0 Ack =
44892
| |(43349) <------------------ (9550) |
|0.428 | ACK - Len: 1448 |Seq = 50684 Ack =
0
| |(43349) ------------------> (9550) |
|0.429 | ACK | |Seq = 0 Ack =
44892
| |(43349) <------------------ (9550) |
|0.429 | ACK - Len: 1448 |Seq = 44892 Ack =
0
| |(43349) ------------------> (9550) |
|0.430 | ACK | |Seq = 0 Ack =
49236
| |(43349) <------------------ (9550) |
|0.632 | PSH, ACK - Len: 1448 |Seq = 49236 Ack =
0
| |(43349) ------------------> (9550) |
|0.633 | ACK | |Seq = 0 Ack =
52132
| |(43349) <------------------ (9550) |
|0.634 | ACK - Len: 1448 |Seq = 52132 Ack =
0
| |(43349) ------------------> (9550) |
|0.634 | ACK - Len: 1448 |Seq = 53580 Ack =
0
| |(43349) ------------------> (9550) |
|0.635 | ACK | |Seq = 0 Ack =
53580
| |(43349) <------------------ (9550) |
|0.635 | ACK - Len: 1448 |Seq = 55028 Ack =
0
| |(43349) ------------------> (9550) |
|0.635 | ACK - Len: 1448 |Seq = 56476 Ack =
0
| |(43349) ------------------> (9550) |
|0.635 | ACK | |Seq = 0 Ack =
55028
| |(43349) <------------------ (9550) |
|0.635 | ACK - Len: 1448 |Seq = 57924 Ack =
0
| |(43349) ------------------> (9550) |
|0.636 | ACK | |Seq = 0 Ack =
55028
| |(43349) <------------------ (9550) |
|0.636 | ACK - Len: 1448 |Seq = 59372 Ack =
0
| |(43349) ------------------> (9550) |
|0.636 | ACK | |Seq = 0 Ack =
55028
| |(43349) <------------------ (9550) |
|0.636 | ACK - Len: 1448 |Seq = 60820 Ack =
0
| |(43349) ------------------> (9550) |
|0.637 | ACK | |Seq = 0 Ack =
55028
| |(43349) <------------------ (9550) |
|0.637 | ACK - Len: 1448 |Seq = 55028 Ack =
0
| |(43349) ------------------> (9550) |
|0.638 | ACK | |Seq = 0 Ack =
59372
| |(43349) <------------------ (9550) |
|0.840 | ACK - Len: 1448 |Seq = 59372 Ack =
0
| |(43349) ------------------> (9550) |
|0.841 | ACK | |Seq = 0 Ack =
62268
| |(43349) <------------------ (9550) |
|0.841 | ACK - Len: 1448 |Seq = 62268 Ack =
0
| |(43349) ------------------> (9550) |
|0.841 | ACK - Len: 1448 |Seq = 63716 Ack =
0
| |(43349) ------------------> (9550) |
|0.843 | ACK | |Seq = 0 Ack =
63716
| |(43349) <------------------ (9550) |
|0.843 | ACK - Len: 1448 |Seq = 65164 Ack =
0
| |(43349) ------------------> (9550) |
|0.843 | ACK - Len: 1448 |Seq = 66612 Ack =
0
| |(43349) ------------------> (9550) |
|0.843 | ACK | |Seq = 0 Ack =
65164
| |(43349) <------------------ (9550) |
|0.843 | ACK - Len: 1448 |Seq = 68060 Ack =
0
| |(43349) ------------------> (9550) |
|0.844 | ACK | |Seq = 0 Ack =
65164
| |(43349) <------------------ (9550) |
|0.844 | ACK - Len: 1448 |Seq = 69508 Ack =
0
| |(43349) ------------------> (9550) |
|0.844 | ACK | |Seq = 0 Ack =
65164
| |(43349) <------------------ (9550) |
|0.844 | ACK - Len: 1448 |Seq = 70956 Ack =
0
| |(43349) ------------------> (9550) |
|0.845 | ACK | |Seq = 0 Ack =
65164
| |(43349) <------------------ (9550) |
|0.845 | ACK - Len: 1448 |Seq = 65164 Ack =
0
| |(43349) ------------------> (9550) |
|0.847 | ACK | |Seq = 0 Ack =
69508
| |(43349) <------------------ (9550) |
|1.047 | ACK - Len: 1448 |Seq = 69508 Ack =
0
| |(43349) ------------------> (9550) |
|1.048 | ACK | |Seq = 0 Ack =
72404
| |(43349) <------------------ (9550) |
|1.048 | ACK - Len: 1448 |Seq = 72404 Ack =
0
| |(43349) ------------------> (9550) |
|1.048 | ACK - Len: 1448 |Seq = 73852 Ack =
0
| |(43349) ------------------> (9550) |
|1.050 | ACK | |Seq = 0 Ack =
73852
| |(43349) <------------------ (9550) |
|1.050 | ACK - Len: 1448 |Seq = 75300 Ack =
0
| |(43349) ------------------> (9550) |
|1.050 | ACK - Len: 1448 |Seq = 76748 Ack =
0
| |(43349) ------------------> (9550) |
|1.050 | ACK | |Seq = 0 Ack =
75300
| |(43349) <------------------ (9550) |
|1.050 | ACK - Len: 1448 |Seq = 78196 Ack =
0
| |(43349) ------------------> (9550) |
|1.051 | ACK | |Seq = 0 Ack =
75300
| |(43349) <------------------ (9550) |
|1.051 | ACK - Len: 1448 |Seq = 79644 Ack =
0
| |(43349) ------------------> (9550) |
|1.051 | ACK | |Seq = 0 Ack =
75300
| |(43349) <------------------ (9550) |
|1.051 | ACK - Len: 1448 |Seq = 81092 Ack =
0
| |(43349) ------------------> (9550) |
|1.052 | ACK | |Seq = 0 Ack =
75300
| |(43349) <------------------ (9550) |
|1.052 | ACK - Len: 1448 |Seq = 75300 Ack =
0
| |(43349) ------------------> (9550) |
|1.053 | ACK | |Seq = 0 Ack =
79644
| |(43349) <------------------ (9550) |
|1.255 | ACK - Len: 1448 |Seq = 79644 Ack =
0
| |(43349) ------------------> (9550) |
|1.256 | ACK | |Seq = 0 Ack =
82540
| |(43349) <------------------ (9550) |
|1.256 | ACK - Len: 1448 |Seq = 82540 Ack =
0
| |(43349) ------------------> (9550) |
|1.256 | ACK - Len: 1448 |Seq = 83988 Ack =
0
| |(43349) ------------------> (9550) |
|1.258 | ACK | |Seq = 0 Ack =
83988
| |(43349) <------------------ (9550) |
|1.258 | ACK - Len: 1448 |Seq = 85436 Ack =
0
| |(43349) ------------------> (9550) |
|1.258 | ACK - Len: 1448 |Seq = 86884 Ack =
0
| |(43349) ------------------> (9550) |
|1.258 | ACK | |Seq = 0 Ack =
85436
| |(43349) <------------------ (9550) |
|1.258 | ACK - Len: 1448 |Seq = 88332 Ack =
0
| |(43349) ------------------> (9550) |
|1.259 | ACK | |Seq = 0 Ack =
85436
| |(43349) <------------------ (9550) |
|1.259 | ACK - Len: 1448 |Seq = 89780 Ack =
0
| |(43349) ------------------> (9550) |
|1.259 | ACK | |Seq = 0 Ack =
85436
| |(43349) <------------------ (9550) |
|1.259 | ACK - Len: 1448 |Seq = 91228 Ack =
0
| |(43349) ------------------> (9550) |
|1.260 | ACK | |Seq = 0 Ack =
85436
| |(43349) <------------------ (9550) |
|1.260 | ACK - Len: 1448 |Seq = 85436 Ack =
0
| |(43349) ------------------> (9550) |
|1.261 | ACK | |Seq = 0 Ack =
89780
| |(43349) <------------------ (9550) |
|1.463 | ACK - Len: 1448 |Seq = 89780 Ack =
0
| |(43349) ------------------> (9550) |
|1.464 | ACK | |Seq = 0 Ack =
92676
| |(43349) <------------------ (9550) |
|1.464 | PSH, ACK - Len: 1448 |Seq = 92676 Ack =
0
| |(43349) ------------------> (9550) |
|1.464 | ACK - Len: 1448 |Seq = 94124 Ack =
0
| |(43349) ------------------> (9550) |
|1.466 | ACK | |Seq = 0 Ack =
94124
| |(43349) <------------------ (9550) |
|1.466 | ACK - Len: 1448 |Seq = 95572 Ack =
0
| |(43349) ------------------> (9550) |
|1.466 | ACK - Len: 1448 |Seq = 97020 Ack =
0
| |(43349) ------------------> (9550) |
|1.466 | ACK | |Seq = 0 Ack =
95572
| |(43349) <------------------ (9550) |
|1.466 | ACK - Len: 1448 |Seq = 98468 Ack =
0
| |(43349) ------------------> (9550) |
|1.467 | ACK | |Seq = 0 Ack =
95572
| |(43349) <------------------ (9550) |
|1.467 | ACK - Len: 1448 |Seq = 99916 Ack =
0
| |(43349) ------------------> (9550) |
|1.467 | ACK | |Seq = 0 Ack =
95572
| |(43349) <------------------ (9550) |
|1.467 | ACK - Len: 1448 |Seq = 101364 Ack =
0
| |(43349) ------------------> (9550) |
|1.468 | ACK | |Seq = 0 Ack =
95572
| |(43349) <------------------ (9550) |
|1.468 | ACK - Len: 1448 |Seq = 95572 Ack =
0
| |(43349) ------------------> (9550) |
|1.469 | ACK | |Seq = 0 Ack =
99916
| |(43349) <------------------ (9550) |
|1.670 | ACK - Len: 1448 |Seq = 99916 Ack =
0
| |(43349) ------------------> (9550) |
|1.671 | ACK | |Seq = 0 Ack =
102812
| |(43349) <------------------ (9550) |
|1.671 | ACK - Len: 1448 |Seq = 102812 Ack =
0
| |(43349) ------------------> (9550) |
|1.671 | ACK - Len: 1448 |Seq = 104260 Ack =
0
| |(43349) ------------------> (9550) |
|1.672 | ACK | |Seq = 0 Ack =
104260
| |(43349) <------------------ (9550) |
|1.672 | ACK - Len: 1448 |Seq = 105708 Ack =
0
| |(43349) ------------------> (9550) |
|1.673 | ACK - Len: 1448 |Seq = 107156 Ack =
0
| |(43349) ------------------> (9550) |
|1.673 | ACK | |Seq = 0 Ack =
105708
| |(43349) <------------------ (9550) |
|1.673 | ACK - Len: 1448 |Seq = 108604 Ack =
0
| |(43349) ------------------> (9550) |
|1.674 | ACK | |Seq = 0 Ack =
105708
| |(43349) <------------------ (9550) |
|1.674 | ACK - Len: 1448 |Seq = 110052 Ack =
0
| |(43349) ------------------> (9550) |
|1.674 | ACK | |Seq = 0 Ack =
105708
| |(43349) <------------------ (9550) |
|1.674 | PSH, ACK - Len: 1448 |Seq = 111500 Ack =
0
| |(43349) ------------------> (9550) |
|1.675 | ACK | |Seq = 0 Ack =
105708
| |(43349) <------------------ (9550) |
|1.675 | ACK - Len: 1448 |Seq = 105708 Ack =
0
| |(43349) ------------------> (9550) |
|1.676 | ACK | |Seq = 0 Ack =
110052
| |(43349) <------------------ (9550) |
|1.878 | ACK - Len: 1448 |Seq = 110052 Ack =
0
| |(43349) ------------------> (9550) |
|1.879 | ACK | |Seq = 0 Ack =
112948
| |(43349) <------------------ (9550) |
|1.879 | ACK - Len: 1448 |Seq = 112948 Ack =
0
| |(43349) ------------------> (9550) |
|1.879 | ACK - Len: 1448 |Seq = 114396 Ack =
0
| |(43349) ------------------> (9550) |
|1.880 | ACK | |Seq = 0 Ack =
114396
| |(43349) <------------------ (9550) |
|1.881 | ACK - Len: 1448 |Seq = 115844 Ack =
0
| |(43349) ------------------> (9550) |
|1.881 | ACK - Len: 1448 |Seq = 117292 Ack =
0
| |(43349) ------------------> (9550) |
|1.881 | ACK | |Seq = 0 Ack =
115844
| |(43349) <------------------ (9550) |
|1.881 | ACK - Len: 1448 |Seq = 118740 Ack =
0
| |(43349) ------------------> (9550) |
|1.882 | ACK | |Seq = 0 Ack =
115844
| |(43349) <------------------ (9550) |
|1.882 | ACK - Len: 1448 |Seq = 120188 Ack =
0
| |(43349) ------------------> (9550) |
|1.882 | ACK | |Seq = 0 Ack =
115844
| |(43349) <------------------ (9550) |
|1.882 | PSH, ACK - Len: 1448 |Seq = 121636 Ack =
0
| |(43349) ------------------> (9550) |
|1.883 | ACK | |Seq = 0 Ack =
115844
| |(43349) <------------------ (9550) |
|1.883 | ACK - Len: 1448 |Seq = 115844 Ack =
0
| |(43349) ------------------> (9550) |
|1.884 | ACK | |Seq = 0 Ack =
120188
| |(43349) <------------------ (9550) |
|2.086 | ACK - Len: 1448 |Seq = 120188 Ack =
0
| |(43349) ------------------> (9550) |
|2.087 | ACK | |Seq = 0 Ack =
123084
| |(43349) <------------------ (9550) |
|2.087 | ACK - Len: 1448 |Seq = 123084 Ack =
0
| |(43349) ------------------> (9550) |
|2.087 | ACK - Len: 1448 |Seq = 124532 Ack =
0
| |(43349) ------------------> (9550) |
|2.088 | ACK | |Seq = 0 Ack =
124532
| |(43349) <------------------ (9550) |
|2.088 | ACK - Len: 1448 |Seq = 125980 Ack =
0
| |(43349) ------------------> (9550) |
|2.088 | ACK - Len: 1448 |Seq = 127428 Ack =
0
| |(43349) ------------------> (9550) |
|2.089 | ACK | |Seq = 0 Ack =
125980
| |(43349) <------------------ (9550) |
|2.089 | ACK - Len: 1448 |Seq = 128876 Ack =
0
| |(43349) ------------------> (9550) |
|2.090 | ACK | |Seq = 0 Ack =
125980
| |(43349) <------------------ (9550) |
|2.090 | ACK - Len: 1448 |Seq = 130324 Ack =
0
| |(43349) ------------------> (9550) |
|2.090 | ACK | |Seq = 0 Ack =
125980
| |(43349) <------------------ (9550) |
|2.090 | ACK - Len: 1448 |Seq = 131772 Ack =
0
| |(43349) ------------------> (9550) |
|2.091 | ACK | |Seq = 0 Ack =
125980
| |(43349) <------------------ (9550) |
|2.091 | ACK - Len: 1448 |Seq = 125980 Ack =
0
| |(43349) ------------------> (9550) |
|2.092 | ACK | |Seq = 0 Ack =
130324
| |(43349) <------------------ (9550) |
|2.294 | ACK - Len: 1448 |Seq = 130324 Ack =
0
| |(43349) ------------------> (9550) |
|2.295 | ACK | |Seq = 0 Ack =
133220
| |(43349) <------------------ (9550) |
|2.295 | ACK - Len: 1448 |Seq = 133220 Ack =
0
| |(43349) ------------------> (9550) |
|2.295 | ACK - Len: 1448 |Seq = 134668 Ack =
0
| |(43349) ------------------> (9550) |
|2.297 | ACK | |Seq = 0 Ack =
134668
| |(43349) <------------------ (9550) |
|2.297 | ACK - Len: 1448 |Seq = 136116 Ack =
0
| |(43349) ------------------> (9550) |
|2.297 | ACK - Len: 1448 |Seq = 137564 Ack =
0
| |(43349) ------------------> (9550) |
|2.297 | ACK | |Seq = 0 Ack =
136116
| |(43349) <------------------ (9550) |
|2.297 | ACK - Len: 1448 |Seq = 139012 Ack =
0
| |(43349) ------------------> (9550) |
|2.298 | ACK | |Seq = 0 Ack =
136116
| |(43349) <------------------ (9550) |
|2.298 | ACK - Len: 1448 |Seq = 140460 Ack =
0
| |(43349) ------------------> (9550) |
|2.298 | ACK | |Seq = 0 Ack =
136116
| |(43349) <------------------ (9550) |
|2.298 | ACK - Len: 1448 |Seq = 141908 Ack =
0
| |(43349) ------------------> (9550) |
|2.299 | ACK | |Seq = 0 Ack =
136116
| |(43349) <------------------ (9550) |
|2.299 | ACK - Len: 1448 |Seq = 136116 Ack =
0
| |(43349) ------------------> (9550) |
|2.300 | ACK | |Seq = 0 Ack =
140460
| |(43349) <------------------ (9550) |
|2.501 | ACK - Len: 1448 |Seq = 140460 Ack =
0
| |(43349) ------------------> (9550) |
|2.502 | ACK | |Seq = 0 Ack =
143356
| |(43349) <------------------ (9550) |
|2.502 | ACK - Len: 1448 |Seq = 143356 Ack =
0
| |(43349) ------------------> (9550) |
|2.502 | ACK - Len: 1448 |Seq = 144804 Ack =
0
| |(43349) ------------------> (9550) |
|2.503 | ACK | |Seq = 0 Ack =
144804
| |(43349) <------------------ (9550) |
|2.503 | ACK - Len: 1448 |Seq = 146252 Ack =
0
| |(43349) ------------------> (9550) |
|2.503 | PSH, ACK - Len: 1448 |Seq = 147700 Ack =
0
| |(43349) ------------------> (9550) |
|2.503 | ACK | |Seq = 0 Ack =
146252
| |(43349) <------------------ (9550) |
|2.503 | ACK - Len: 1448 |Seq = 149148 Ack =
0
| |(43349) ------------------> (9550) |
|2.505 | ACK | |Seq = 0 Ack =
146252
| |(43349) <------------------ (9550) |
|2.505 | ACK - Len: 1448 |Seq = 150596 Ack =
0
| |(43349) ------------------> (9550) |
|2.505 | ACK | |Seq = 0 Ack =
146252
| |(43349) <------------------ (9550) |
|2.505 | PSH, ACK - Len: 1448 |Seq = 152044 Ack =
0
| |(43349) ------------------> (9550) |
|2.506 | ACK | |Seq = 0 Ack =
146252
| |(43349) <------------------ (9550) |
|2.506 | ACK - Len: 1448 |Seq = 146252 Ack =
0
| |(43349) ------------------> (9550) |
|2.507 | ACK | |Seq = 0 Ack =
150596
| |(43349) <------------------ (9550) |
|2.709 | ACK - Len: 1448 |Seq = 150596 Ack =
0
| |(43349) ------------------> (9550) |
|2.710 | ACK | |Seq = 0 Ack =
153492
| |(43349) <------------------ (9550) |
|2.710 | ACK - Len: 1448 |Seq = 153492 Ack =
0
| |(43349) ------------------> (9550) |
|2.710 | PSH, ACK - Len: 1448 |Seq = 154940 Ack =
0
| |(43349) ------------------> (9550) |
|2.711 | ACK | |Seq = 0 Ack =
154940
| |(43349) <------------------ (9550) |
|2.711 | ACK - Len: 1448 |Seq = 156388 Ack =
0
| |(43349) ------------------> (9550) |
|2.711 | ACK - Len: 1448 |Seq = 157836 Ack =
0
| |(43349) ------------------> (9550) |
|2.711 | ACK | |Seq = 0 Ack =
156388
| |(43349) <------------------ (9550) |
|2.711 | ACK - Len: 1448 |Seq = 159284 Ack =
0
| |(43349) ------------------> (9550) |
|2.713 | ACK | |Seq = 0 Ack =
156388
| |(43349) <------------------ (9550) |
|2.713 | ACK - Len: 1448 |Seq = 160732 Ack =
0
| |(43349) ------------------> (9550) |
|2.713 | ACK | |Seq = 0 Ack =
156388
| |(43349) <------------------ (9550) |
|2.713 | ACK - Len: 1448 |Seq = 162180 Ack =
0
| |(43349) ------------------> (9550) |
|2.714 | ACK | |Seq = 0 Ack =
156388
| |(43349) <------------------ (9550) |
|2.714 | ACK - Len: 1448 |Seq = 156388 Ack =
0
| |(43349) ------------------> (9550) |
|2.715 | ACK | |Seq = 0 Ack =
160732
| |(43349) <------------------ (9550) |
|2.917 | ACK - Len: 1448 |Seq = 160732 Ack =
0
| |(43349) ------------------> (9550) |
|2.918 | ACK | |Seq = 0 Ack =
163628
| |(43349) <------------------ (9550) |
|2.918 | ACK - Len: 1448 |Seq = 163628 Ack =
0
| |(43349) ------------------> (9550) |
|2.918 | ACK - Len: 1448 |Seq = 165076 Ack =
0
| |(43349) ------------------> (9550) |
|2.919 | ACK | |Seq = 0 Ack =
165076
| |(43349) <------------------ (9550) |
|2.919 | ACK - Len: 1448 |Seq = 166524 Ack =
0
| |(43349) ------------------> (9550) |
|2.919 | ACK - Len: 1448 |Seq = 167972 Ack =
0
| |(43349) ------------------> (9550) |
|2.919 | ACK | |Seq = 0 Ack =
166524
| |(43349) <------------------ (9550) |
|2.919 | ACK - Len: 1448 |Seq = 169420 Ack =
0
| |(43349) ------------------> (9550) |
|2.921 | ACK | |Seq = 0 Ack =
166524
| |(43349) <------------------ (9550) |
|2.921 | ACK - Len: 1448 |Seq = 170868 Ack =
0
| |(43349) ------------------> (9550) |
|2.921 | ACK | |Seq = 0 Ack =
166524
| |(43349) <------------------ (9550) |
|2.921 | ACK - Len: 1448 |Seq = 172316 Ack =
0
| |(43349) ------------------> (9550) |
|2.922 | ACK | |Seq = 0 Ack =
166524
| |(43349) <------------------ (9550) |
|2.922 | ACK - Len: 1448 |Seq = 166524 Ack =
0
| |(43349) ------------------> (9550) |
|2.923 | ACK | |Seq = 0 Ack =
170868
| |(43349) <------------------ (9550) |
|3.125 | ACK - Len: 1448 |Seq = 170868 Ack =
0
| |(43349) ------------------> (9550) |
|3.126 | ACK | |Seq = 0 Ack =
173764
| |(43349) <------------------ (9550) |
|3.126 | PSH, ACK - Len: 1448 |Seq = 173764 Ack =
0
| |(43349) ------------------> (9550) |
|3.126 | ACK - Len: 1448 |Seq = 175212 Ack =
0
| |(43349) ------------------> (9550) |
|3.127 | ACK | |Seq = 0 Ack =
175212
| |(43349) <------------------ (9550) |
|3.127 | ACK - Len: 1448 |Seq = 176660 Ack =
0
| |(43349) ------------------> (9550) |
|3.127 | ACK - Len: 1448 |Seq = 178108 Ack =
0
| |(43349) ------------------> (9550) |
|3.127 | ACK | |Seq = 0 Ack =
176660
| |(43349) <------------------ (9550) |
|3.127 | PSH, ACK - Len: 1448 |Seq = 179556 Ack =
0
| |(43349) ------------------> (9550) |
|3.128 | ACK | |Seq = 0 Ack =
176660
| |(43349) <------------------ (9550) |
|3.128 | ACK - Len: 1448 |Seq = 181004 Ack =
0
| |(43349) ------------------> (9550) |
|3.129 | ACK | |Seq = 0 Ack =
176660
| |(43349) <------------------ (9550) |
|3.129 | PSH, ACK - Len: 1448 |Seq = 182452 Ack =
0
| |(43349) ------------------> (9550) |
|3.130 | ACK | |Seq = 0 Ack =
176660
| |(43349) <------------------ (9550) |
|3.130 | ACK - Len: 1448 |Seq = 176660 Ack =
0
| |(43349) ------------------> (9550) |
|3.131 | ACK | |Seq = 0 Ack =
181004
| |(43349) <------------------ (9550) |

Richard Eich wrote:
> (E-Mail Removed) wrote...
> >
> > Richard Eich wrote:
> > > (E-Mail Removed) wrote...
> > > >
> > > > Richard Eich wrote:
> > > > > Client and Server OS: SuSE 9.3 Pro 2.6.11 default kernel, neither
> > > > > machine patched after install from CDs.
> > > > >
> > > > > Communicating over: GigE LAN
> > > > >
> > > > > 1) Server (receiver) is consistently adversting a TCP RWIN of 32K.
> > > > > 2) Server consistently has TCP RECV-Q of 0.
> > > > > 3) Client (sender) consistently shows a TCP SEND-Q of 80K.
> > > > > 4) Socket is up and connection is ESTABLISHED from both sides.
> > > > > 5) No data is transmitted.
> > > > >
> > > > > To troubleshoot, I've torn down and re-established to connection
> > > > > countless times. There may be a trickle of data initially, but
> > > > > within a few seconds the client SEND-Q builds and transmission stops.
> > > > > Receiver's window size never goes below 32K.
> > > > >
> > > > > Never seen this kind of behavior before. If the server process was
> > > > > slow, I'd expect to see a RECV-Q buildup to go with the big SEND-Q.
> > > >
> > > > I would use tcpdump or wireshark/ethereal to see what is being sent and
> > > > received.
> > >
> > > How do you think I know the TCP RWIN size for the server?
> >
> > So, what does tcpdump shows you before the transfer stops?
>
> Looks like we're getting a lot of retransmissions, which I believe
> would clearly explain the full sender tcp send-q, the receiver empty
> tcp recv-q, and the receiver's normal receive window size.
>
> (Relative ACKs used for clarity).

[]

In the final six tcp segments show that the last two 1448-byte segments
remain unacknowledged. The sender's tcp stack must retransmit the
segments as it happened in the packet dump before. The only explanation
I can think of why we can not see retransmissions in the dump, is that
the retransmissions never get to the network interface egress queue.
The queue is full when the network is congested or when the driver is
misbehaving. You may like to check ifconfig output, particularly if
errors, dropped and collisions fields are non zero for the related
interface.

Does send/write in the client return with an error and if so what is
errno value?