Understanding Send-Q and Recv-Q by netstat

Discussion in 'Linux Networking' started by Christophe Lohr, Dec 4, 2009.

  1. Hello,
    I do not understand the relationship that may exist between the socket
    option SO_RCVBUF / SO_SNDBUF and values Recv-Q/Send-Q given by netstat.

    Consider the following example, consisting of a TCP client (sending
    zeros) and a lazy TCP server (which does not consume received data).



    $ socat -ddd OPEN:/dev/zero TCP:localhost:8003,sndbuf=2000,rcvbuf=2000
    2009/12/04 17:30:21 socat[2880] E write(4, 0x954a758, 8192): Broken pipe


    $ socat -ddd TCP-LISTEN:8003,reuseaddr,sndbuf=2000,rcvbuf=2000
    EXEC:'sleep 30'
    2009/12/04 17:30:21 socat[2879] E write(3, 0x89c2008, 3008): Broken pipe



    Thus, the values Recv-Q/Send-Q should match the values SO_RCVBUF /
    SO_SNDBUF configured. Isn't it? Yet this is not the case... why?


    $ netstat -tpn
    Proto Recv-Q Send-Q Local Address Foreign Address
    State PID/Program name

    tcp 3008 0 127.0.0.1:8003 127.0.0.1:54629
    ESTABLISHED 2879/socat

    tcp 0 2176 127.0.0.1:54629 127.0.0.1:8003
    ESTABLISHED 2880/socat



    Note that the server says he received 8003 bytes, which is consistent
    with Recv-Q.
    But the client said he sent 8192 bytes, which does not correspond to the
    Send-Q:2176.
    Moreover, none of its values is consistent with the setting of
    SO_RCVBUF/SO_SNDBUF.


    Does someone could explain?
    Best regards
     
    Christophe Lohr, Dec 4, 2009
    #1
    1. Advertisements

  2. Christophe Lohr

    Rick Jones Guest

    SO_RCVBUF and SO_SNDBUF are, ostensibly, the limits to how much can be
    queued to the socket. Recv-Q and Send-Q are how much are actually
    there.

    Recv-Q will be that data which has not yet been pulled from the socket
    buffer by the application.

    Send-Q will be that data which the sending application has given to
    the transport, but has yet to be ACKnowledged by the receiving TCP.
    If the sndbuf and rcvbuf settings correspond to setsockopt() calls for
    SO_SNDBUF and SO_RCVBUF respectively, it is important to keep in mind
    that Linux, unlike virtually every other *nix I've encountered, very
    much considers the setsockopt() call a "suggestion" rather than a
    "demand." It will set the actual socket buffer size to something
    else, which one can see with a subsequent getsockopt() call (eg what
    netperf does). It adds-in space for overhead (overheads of the
    buffers that get queued to the socket buffers IIRC)

    rick jones
     
    Rick Jones, Dec 4, 2009
    #2
    1. Advertisements

  3. Christophe Lohr

    Rick Jones Guest

    perhaps it is out of date, but the netstat manpage on my linux system
    has this to say about the Q's:

    Recv-Q
    The count of bytes not copied by the user program connected
    to this socket.

    Send-Q
    The count of bytes not acknowledged by the remote host.

    Which certainly sounds like application-level bytes to me.

    rick jones
     
    Rick Jones, Dec 4, 2009
    #3
  4. So, there are two points:
    - set and read socket buffer size are not the same (2 times larger)
    - Send-Q and Recv-Q may containts headers (but in TCP it contains user
    data only, isn't it?)


    So let's play with ad-hoc lient / server
    (see attached files)

    $ ./lazyServerTCP 8003
    SO_RCVBUF=4000
    SO_SNDBUF=4000

    $ ./clientTCP localhost 8003
    SO_RCVBUF=4000
    SO_SNDBUF=4000
    80715


    $ netstat -tpn
    Proto Recv-Q Send-Q Adresse locale Adresse distante Etat
    PID/Program name
    tcp 65930 0 127.0.0.1:8003 127.0.0.1:47969
    ESTABLISHED 16937/lazyServerTCP
    tcp 0 14784 127.0.0.1:47969 127.0.0.1:8003
    ESTABLISHED 16939/clientTCP


    I see that Recv-Q plus Sed-Q equals the amount of user data sent.

    However I can't figure out why Recv-Q is 16 times larger than the actual
    socket buffer...?

    Regards.
    --
     
    Christophe Lohr, Dec 7, 2009
    #4
  5. David Schwartz a écrit :
    Let's play with wireshark. Do a "Follow TCP stream".
    The amount of user data sent before the TCP window becomes zero is equal
    to Recv-Q

    So, I understand the definition of Recv-Q:
    The count of bytes not copied by the user program connected to
    this socket.

    Well, but netstat gives a Recv-Q per socket... (a TCP socket in my case)
    .... sorry, I don't understand what are socket receive/send buffers :-(

    This is another level of buffer?
    What is in it?
    Is it after or before Recv-Q (along data flow within the socket)

    Regards
    Thank you for your explanations and your patience
     
    Christophe Lohr, Dec 7, 2009
    #5
  6. Christophe Lohr

    Rick Jones Guest

    Perhaps this is decades of BSD-based precedent interacting with
    Linux's desire to be different?

    rick jones
     
    Rick Jones, Dec 7, 2009
    #6
  7. Christophe Lohr

    Rick Jones Guest

    Off to the side, with the rest of the meta-data :) However, that may
    not be the case under Linux, which relates to how it likes to
    effectively double (up to a limit) what one requests in a setsockopt()
    call.

    rick jones
     
    Rick Jones, Dec 7, 2009
    #7
  8. Christophe Lohr a écrit :
    According to "A User's Guide to TCP Windows", there is a direct
    relationship between TCP window and SO_RCVBUF
    http://www.ncsa.illinois.edu/People/vwelch/net_perf/tcp_windows.html

    so... I'm lost...

    What are socket buffers?

    Regards.
     
    Christophe Lohr, Dec 8, 2009
    #8
  9. David Schwartz a écrit :
    Yes, that's it.
    I'm just looking for a high-level information

    Regards
     
    Christophe Lohr, Dec 8, 2009
    #9
  10. David Schwartz a écrit :
    ok

    Thank you for your patience
    Regards
     
    Christophe Lohr, Dec 8, 2009
    #10
  11. Christophe Lohr

    ywssxt

    Joined:
    Sep 28, 2010
    Messages:
    1
    Likes Received:
    0
    I wonder how to get the attached files?
    Pls let me know, thanks
     
    ywssxt, Sep 28, 2010
    #11
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.