Understanding Send-Q and Recv-Q by netstat

Hello,
I do not understand the relationship that may exist between the socket
option SO_RCVBUF / SO_SNDBUF and values Recv-Q/Send-Q given by netstat.

Consider the following example, consisting of a TCP client (sending
zeros) and a lazy TCP server (which does not consume received data).



$ socat -ddd OPEN:/dev/zero TCP:localhost:8003,sndbuf=2000,rcvbuf=2000
2009/12/04 17:30:21 socat[2880] E write(4, 0x954a758, 8192): Broken pipe


$ socat -ddd TCP-LISTEN:8003,reuseaddr,sndbuf=2000,rcvbuf=2000
EXEC:'sleep 30'
2009/12/04 17:30:21 socat[2879] E write(3, 0x89c2008, 3008): Broken pipe



Thus, the values Recv-Q/Send-Q should match the values SO_RCVBUF /
SO_SNDBUF configured. Isn't it? Yet this is not the case... why?


$ netstat -tpn
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name

tcp 3008 0 127.0.0.1:8003 127.0.0.1:54629
ESTABLISHED 2879/socat

tcp 0 2176 127.0.0.1:54629 127.0.0.1:8003
ESTABLISHED 2880/socat



Note that the server says he received 8003 bytes, which is consistent
with Recv-Q.
But the client said he sent 8192 bytes, which does not correspond to the
Send-Q:2176.
Moreover, none of its values is consistent with the setting of
SO_RCVBUF/SO_SNDBUF.


Does someone could explain?
Best regards

 

SO_RCVBUF and SO_SNDBUF are, ostensibly, the limits to how much can be
queued to the socket. Recv-Q and Send-Q are how much are actually
there.

Recv-Q will be that data which has not yet been pulled from the socket
buffer by the application.

Send-Q will be that data which the sending application has given to
the transport, but has yet to be ACKnowledged by the receiving TCP.

If the sndbuf and rcvbuf settings correspond to setsockopt() calls for
SO_SNDBUF and SO_RCVBUF respectively, it is important to keep in mind
that Linux, unlike virtually every other *nix I've encountered, very
much considers the setsockopt() call a "suggestion" rather than a
"demand." It will set the actual socket buffer size to something
else, which one can see with a subsequent getsockopt() call (eg what
netperf does). It adds-in space for overhead (overheads of the
buffers that get queued to the socket buffers IIRC)

rick jones

 

perhaps it is out of date, but the netstat manpage on my linux system
has this to say about the Q's:

Recv-Q
The count of bytes not copied by the user program connected
to this socket.

Send-Q
The count of bytes not acknowledged by the remote host.

Which certainly sounds like application-level bytes to me.

rick jones

 

So, there are two points:
- set and read socket buffer size are not the same (2 times larger)
- Send-Q and Recv-Q may containts headers (but in TCP it contains user
data only, isn't it?)


So let's play with ad-hoc lient / server
(see attached files)

$ ./lazyServerTCP 8003
SO_RCVBUF=4000
SO_SNDBUF=4000

$ ./clientTCP localhost 8003
SO_RCVBUF=4000
SO_SNDBUF=4000
80715


$ netstat -tpn
Proto Recv-Q Send-Q Adresse locale Adresse distante Etat
PID/Program name
tcp 65930 0 127.0.0.1:8003 127.0.0.1:47969
ESTABLISHED 16937/lazyServerTCP
tcp 0 14784 127.0.0.1:47969 127.0.0.1:8003
ESTABLISHED 16939/clientTCP


I see that Recv-Q plus Sed-Q equals the amount of user data sent.

However I can't figure out why Recv-Q is 16 times larger than the actual
socket buffer...?

Regards.
--

 

David Schwartz a écrit :

Let's play with wireshark. Do a "Follow TCP stream".
The amount of user data sent before the TCP window becomes zero is equal
to Recv-Q

So, I understand the definition of Recv-Q:
The count of bytes not copied by the user program connected to
this socket.

Well, but netstat gives a Recv-Q per socket... (a TCP socket in my case)

.... sorry, I don't understand what are socket receive/send buffers :-(

This is another level of buffer?
What is in it?
Is it after or before Recv-Q (along data flow within the socket)

Regards
Thank you for your explanations and your patience

 

Perhaps this is decades of BSD-based precedent interacting with
Linux's desire to be different?

rick jones

 

Off to the side, with the rest of the meta-data However, that may
not be the case under Linux, which relates to how it likes to
effectively double (up to a limit) what one requests in a setsockopt()
call.

rick jones

 

Christophe Lohr a écrit :

According to "A User's Guide to TCP Windows", there is a direct
relationship between TCP window and SO_RCVBUF
http://www.ncsa.illinois.edu/People/vwelch/net_perf/tcp_windows.html

so... I'm lost...

What are socket buffers?

Regards.

 

David Schwartz a écrit :

Yes, that's it.
I'm just looking for a high-level information

Regards

 

David Schwartz a écrit :

ok

Thank you for your patience
Regards

 

I wonder how to get the attached files?
Pls let me know, thanks