Understanding Send-Q and Recv-Q by netstat

Hello,
I do not understand the relationship that may exist between the socket
option SO_RCVBUF / SO_SNDBUF and values Recv-Q/Send-Q given by netstat.

Consider the following example, consisting of a TCP client (sending
zeros) and a lazy TCP server (which does not consume received data).



$ socat -ddd OPEN:/dev/zero TCP:localhost:8003,sndbuf=2000,rcvbuf=2000
2009/12/04 17:30:21 socat[2880] E write(4, 0x954a758, 8192): Broken pipe


$ socat -ddd TCP-LISTEN:8003,reuseaddr,sndbuf=2000,rcvbuf=2000
EXEC:'sleep 30'
2009/12/04 17:30:21 socat[2879] E write(3, 0x89c2008, 3008): Broken pipe



Thus, the values Recv-Q/Send-Q should match the values SO_RCVBUF /
SO_SNDBUF configured. Isn't it? Yet this is not the case... why?


$ netstat -tpn
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name

tcp 3008 0 127.0.0.1:8003 127.0.0.1:54629
ESTABLISHED 2879/socat

tcp 0 2176 127.0.0.1:54629 127.0.0.1:8003
ESTABLISHED 2880/socat



Note that the server says he received 8003 bytes, which is consistent
with Recv-Q.
But the client said he sent 8192 bytes, which does not correspond to the
Send-Q:2176.
Moreover, none of its values is consistent with the setting of
SO_RCVBUF/SO_SNDBUF.


Does someone could explain?
Best regards

Christophe Lohr <(E-Mail Removed)> wrote:
> Hello,
> I do not understand the relationship that may exist between the socket
> option SO_RCVBUF / SO_SNDBUF and values Recv-Q/Send-Q given by netstat.

SO_RCVBUF and SO_SNDBUF are, ostensibly, the limits to how much can be
queued to the socket. Recv-Q and Send-Q are how much are actually
there.

Recv-Q will be that data which has not yet been pulled from the socket
buffer by the application.

Send-Q will be that data which the sending application has given to
the transport, but has yet to be ACKnowledged by the receiving TCP.

> $ socat -ddd OPEN:/dev/zero TCP:localhost:8003,sndbuf=2000,rcvbuf=2000
> 2009/12/04 17:30:21 socat[2880] E write(4, 0x954a758, 8192): Broken pipe

If the sndbuf and rcvbuf settings correspond to setsockopt() calls for
SO_SNDBUF and SO_RCVBUF respectively, it is important to keep in mind
that Linux, unlike virtually every other *nix I've encountered, very
much considers the setsockopt() call a "suggestion" rather than a
"demand." It will set the actual socket buffer size to something
else, which one can see with a subsequent getsockopt() call (eg what
netperf does). It adds-in space for overhead (overheads of the
buffers that get queued to the socket buffers IIRC)

rick jones
--
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

David Schwartz <(E-Mail Removed)> wrote:
> You are making the naive assumption that the send queue and the
> receive queue are set in units of application-level data bytes. They
> are not.

perhaps it is out of date, but the netstat manpage on my linux system
has this to say about the Q's:

Recv-Q
The count of bytes not copied by the user program connected
to this socket.

Send-Q
The count of bytes not acknowledged by the remote host.

Which certainly sounds like application-level bytes to me.

rick jones
--
a wide gulf separates "what if" from "if only"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

So, there are two points:
- set and read socket buffer size are not the same (2 times larger)
- Send-Q and Recv-Q may containts headers (but in TCP it contains user
data only, isn't it?)


So let's play with ad-hoc lient / server
(see attached files)

$ ./lazyServerTCP 8003
SO_RCVBUF=4000
SO_SNDBUF=4000

$ ./clientTCP localhost 8003
SO_RCVBUF=4000
SO_SNDBUF=4000
80715


$ netstat -tpn
Proto Recv-Q Send-Q Adresse locale Adresse distante Etat
PID/Program name
tcp 65930 0 127.0.0.1:8003 127.0.0.1:47969
ESTABLISHED 16937/lazyServerTCP
tcp 0 14784 127.0.0.1:47969 127.0.0.1:8003
ESTABLISHED 16939/clientTCP


I see that Recv-Q plus Sed-Q equals the amount of user data sent.

However I can't figure out why Recv-Q is 16 times larger than the actual
socket buffer...?

Regards.
--

David Schwartz a écrit :
>
>> However I can't figure out why Recv-Q is 16 times larger than the actual
>> socket buffer...?
>
> Why should one have anything to do with the other? You're measuring
> two completely different things and wondering why the measurements are
> different -- well why shouldn't they be?

Let's play with wireshark. Do a "Follow TCP stream".
The amount of user data sent before the TCP window becomes zero is equal
to Recv-Q

So, I understand the definition of Recv-Q:
The count of bytes not copied by the user program connected to
this socket.


> The receive buffer is
> something you at socket level. The Recv-Q is something at TCP level.

Well, but netstat gives a Recv-Q per socket... (a TCP socket in my case)

> The receive queue size may affect how fast the receive buffer can
> grow, but it won't directly affect its ultimate maximum size.

.... sorry, I don't understand what are socket receive/send buffers :-(

This is another level of buffer?
What is in it?
Is it after or before Recv-Q (along data flow within the socket)

Regards
Thank you for your explanations and your patience

David Schwartz <(E-Mail Removed)> wrote:
> Why should one have anything to do with the other?

Perhaps this is decades of BSD-based precedent interacting with
Linux's desire to be different?

rick jones
--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

David Schwartz <(E-Mail Removed)> wrote:
> On Dec 4, 12:48?pm, Rick Jones <rick.jon...@hp.com> wrote:

> > perhaps it is out of date, but the netstat manpage on my linux system
> > has this to say about the Q's:
> >
> > ? ?Recv-Q
> > ? ? ? ?The ?count ?of ?bytes ?not copied by the user program connected
> > ? ? ? ?to this socket.
> >
> > ? ?Send-Q
> > ? ? ? ?The count of bytes not acknowledged by the remote host.
> >
> > Which certainly sounds like application-level bytes to me.

> I was talking about how the queue sizes are set, not how they are
> measured. This description is TCP-specific, but the SO_SNDBUF/
> SO_RCVBUF socket options are protocol-neutral.

> Think about UDP. Where do you think the source IP and port are stored
> if not in the receive queue?

Off to the side, with the rest of the meta-data However, that may
not be the case under Linux, which relates to how it likes to
effectively double (up to a limit) what one requests in a setsockopt()
call.

rick jones
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

Christophe Lohr a écrit :
> ... sorry, I don't understand what are socket receive/send buffers :-(
>
> This is another level of buffer?
> What is in it?
> Is it after or before Recv-Q (along data flow within the socket)

According to "A User's Guide to TCP Windows", there is a direct
relationship between TCP window and SO_RCVBUF
http://www.ncsa.illinois.edu/People/...p_windows.html

so... I'm lost...

What are socket buffers?

Regards.

David Schwartz a écrit :
>
>> What are socket buffers?
>
> It's implementation-defined, and you will have to get into the
> horribly gory details of the implementation to understand it.
>
> Why do you care? Is this for curiosity?

Yes, that's it.
I'm just looking for a high-level information

Regards

David Schwartz a écrit :
> Unfortunately, the high-level information is that the underlying
> network protocol provides you an ability to set something it calls a
> "send buffer" and something it calls a "receive buffer" in units of
> bytes. And that's it.

ok

Thank you for your patience
Regards