Understanding Send-Q and Recv-Q by netstat
in [Linux Networking]
|
Prev: 2.6.31 and connmark restore are not really working for fwmarking and policy routing
Next: ug+w problem on share mounted from XP
From: Christophe Lohr on 4 Dec 2009 11:55 Hello, I do not understand the relationship that may exist between the socket option SO_RCVBUF / SO_SNDBUF and values Recv-Q/Send-Q given by netstat. Consider the following example, consisting of a TCP client (sending zeros) and a lazy TCP server (which does not consume received data). $ socat -ddd OPEN:/dev/zero TCP:localhost:8003,sndbuf=2000,rcvbuf=2000 2009/12/04 17:30:21 socat[2880] E write(4, 0x954a758, 8192): Broken pipe $ socat -ddd TCP-LISTEN:8003,reuseaddr,sndbuf=2000,rcvbuf=2000 EXEC:'sleep 30' 2009/12/04 17:30:21 socat[2879] E write(3, 0x89c2008, 3008): Broken pipe Thus, the values Recv-Q/Send-Q should match the values SO_RCVBUF / SO_SNDBUF configured. Isn't it? Yet this is not the case... why? $ netstat -tpn Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 3008 0 127.0.0.1:8003 127.0.0.1:54629 ESTABLISHED 2879/socat tcp 0 2176 127.0.0.1:54629 127.0.0.1:8003 ESTABLISHED 2880/socat Note that the server says he received 8003 bytes, which is consistent with Recv-Q. But the client said he sent 8192 bytes, which does not correspond to the Send-Q:2176. Moreover, none of its values is consistent with the setting of SO_RCVBUF/SO_SNDBUF. Does someone could explain? Best regards
From: Rick Jones on 4 Dec 2009 12:40 Christophe Lohr <christophe.lohr(a)enst-bretagne.fr> wrote: > Hello, > I do not understand the relationship that may exist between the socket > option SO_RCVBUF / SO_SNDBUF and values Recv-Q/Send-Q given by netstat. SO_RCVBUF and SO_SNDBUF are, ostensibly, the limits to how much can be queued to the socket. Recv-Q and Send-Q are how much are actually there. Recv-Q will be that data which has not yet been pulled from the socket buffer by the application. Send-Q will be that data which the sending application has given to the transport, but has yet to be ACKnowledged by the receiving TCP. > $ socat -ddd OPEN:/dev/zero TCP:localhost:8003,sndbuf=2000,rcvbuf=2000 > 2009/12/04 17:30:21 socat[2880] E write(4, 0x954a758, 8192): Broken pipe If the sndbuf and rcvbuf settings correspond to setsockopt() calls for SO_SNDBUF and SO_RCVBUF respectively, it is important to keep in mind that Linux, unlike virtually every other *nix I've encountered, very much considers the setsockopt() call a "suggestion" rather than a "demand." It will set the actual socket buffer size to something else, which one can see with a subsequent getsockopt() call (eg what netperf does). It adds-in space for overhead (overheads of the buffers that get queued to the socket buffers IIRC) rick jones -- firebug n, the idiot who tosses a lit cigarette out his car window these opinions are mine, all mine; HP might not want them anyway... :) feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...
From: David Schwartz on 4 Dec 2009 14:36 On Dec 4, 8:55 am, Christophe Lohr <christophe.l...(a)enst-bretagne.fr> wrote: > Does someone could explain? You are making the naive assumption that the send queue and the receive queue are set in units of application-level data bytes. They are not. DS
From: Rick Jones on 4 Dec 2009 15:48 David Schwartz <davids(a)webmaster.com> wrote: > You are making the naive assumption that the send queue and the > receive queue are set in units of application-level data bytes. They > are not. perhaps it is out of date, but the netstat manpage on my linux system has this to say about the Q's: Recv-Q The count of bytes not copied by the user program connected to this socket. Send-Q The count of bytes not acknowledged by the remote host. Which certainly sounds like application-level bytes to me. rick jones -- a wide gulf separates "what if" from "if only" these opinions are mine, all mine; HP might not want them anyway... :) feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...
From: David Schwartz on 5 Dec 2009 03:23
On Dec 4, 12:48 pm, Rick Jones <rick.jon...(a)hp.com> wrote: > perhaps it is out of date, but the netstat manpage on my linux system > has this to say about the Q's: > > Recv-Q > The count of bytes not copied by the user program connected > to this socket. > > Send-Q > The count of bytes not acknowledged by the remote host. > > Which certainly sounds like application-level bytes to me. I was talking about how the queue sizes are set, not how they are measured. This description is TCP-specific, but the SO_SNDBUF/ SO_RCVBUF socket options are protocol-neutral. Think about UDP. Where do you think the source IP and port are stored if not in the receive queue? DS |