Raise initial congestion window size / speedup slow start?
in [Kernel]
|
Prev: [PATCH] wm8727: add a missing return in wm8727_platform_probe
Next: [PATCH] block: Add secure discard
From: Ed W on 14 Jul 2010 07:00 Hi, my network connection looks like 500Kbits with a round trip latency of perhaps 1s+ (it's a satellite link). From what I can see the linux initial congestion window is signficantly limiting me here, with slow start taking many many seconds to open up the window wide enough to get the data flowing? For protocols like http this is really hurting with all the short lived connections never really getting up to speed. (throw in some random packet loss and things really screech to a halt) Reading around there appear to be several previous attempts to modify the kernel to start with a slightly wider initial congestion window, say 10 packets. (Seems even google did some work on this and agreed that a small of initial cwd to 10 ish, would help even many non satellite users?) However, all the work I can find is quite old and doesn't seem to give me much of a leg up in terms of experimenting with such changes on a modern kernel? Does someone have some pointers on where to look to modify initial congestion window please? Thanks Ed W -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Bill Davidsen on 14 Jul 2010 11:30 Ed W wrote: > Hi, my network connection looks like 500Kbits with a round trip latency > of perhaps 1s+ (it's a satellite link). > > From what I can see the linux initial congestion window is signficantly > limiting me here, with slow start taking many many seconds to open up > the window wide enough to get the data flowing? For protocols like http > this is really hurting with all the short lived connections never really > getting up to speed. (throw in some random packet loss and things > really screech to a halt) > > Reading around there appear to be several previous attempts to modify > the kernel to start with a slightly wider initial congestion window, say > 10 packets. (Seems even google did some work on this and agreed that a > small of initial cwd to 10 ish, would help even many non satellite > users?) However, all the work I can find is quite old and doesn't seem > to give me much of a leg up in terms of experimenting with such changes > on a modern kernel? > > Does someone have some pointers on where to look to modify initial > congestion window please? > Are you sure that's the issue? The backlog is in incoming, is it not? Having dealt with moderately long delays push TB between timezones, have you set your window size up? Set /proc/sys/net/ipv4/tcp_adv_win_scale to 5 or 6 and see if that helps. You may have to go into /proc/sys/net/core and crank up the rmem_* settings, depending on your distribution. This allows the server to push a lot of data without an ack, which is what you want, the ack will be delayed by the long latency, so this helps. You can calculate how large to make the setting, but "make it bigger until tcpdump never shows the window size < 2k" is the best way, even a bit larger than that wont hurt, although it will take a bit of memory. -- Bill Davidsen <davidsen(a)tmr.com> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: David Miller on 14 Jul 2010 14:20 From: Bill Davidsen <davidsen(a)tmr.com> Date: Wed, 14 Jul 2010 11:21:15 -0400 > You may have to go into /proc/sys/net/core and crank up the > rmem_* settings, depending on your distribution. You should never, ever, have to touch the various networking sysctl values to get good performance in any normal setup. If you do, it's a bug, report it so we can fix it. I cringe every time someone says to do this, so please do me a favor and don't spread this further. :-) For one thing, TCP dynamically adjusts the socket buffer sizes based upon the behavior of traffic on the connection. And the TCP memory limit sysctls (not the core socket ones) are sized based upon available memory. They are there to protect you from situations such as having so much memory dedicated to socket buffers that there is none left to do other things effectively. It's a protective limit, rather than a setting meant to increase or improve performance. So like the others, leave these alone too. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Ed W on 14 Jul 2010 14:40 >> Does someone have some pointers on where to look to modify initial >> congestion window please? >> > Are you sure that's the issue? The backlog is in incoming, is it not? Well, I was simplifying a little bit, actually I have a bunch of protocols in use, http is one of them > Having dealt with moderately long delays push TB between timezones, > have you set your window size up? Set > /proc/sys/net/ipv4/tcp_adv_win_scale to 5 or 6 and see if that helps. > You may have to go into /proc/sys/net/core and crank up the rmem_* > settings, depending on your distribution. > > This allows the server to push a lot of data without an ack, which is > what you want, the ack will be delayed by the long latency, so this helps. I think I'm misunderstanding something fundamental here: - Surely the limited congestion window is what throttles me at connection initialisation time and this will not be affected by changing the params you mention above? For sure the sliding window will be relevant vs my bandwidth delay product once the tcp connection reaches steady state, but I'm mostly worried here about performance right at the creation of the connection? - Both you and Alan mention that the bulk of the traffic is "incoming" - this implies you think it's relevant? Obviously I'm missing something fundamental here because my understanding is that the congestion window shuts us down in both directions (at the start of the connection?) Thanks for the replies - I will take it over to netdev Ed W -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Ed W on 14 Jul 2010 14:50
On 14/07/2010 19:15, David Miller wrote: > From: Bill Davidsen<davidsen(a)tmr.com> > Date: Wed, 14 Jul 2010 11:21:15 -0400 > > >> You may have to go into /proc/sys/net/core and crank up the >> rmem_* settings, depending on your distribution. >> > You should never, ever, have to touch the various networking sysctl > values to get good performance in any normal setup. If you do, it's a > bug, report it so we can fix it. > Just checking the basics here because I don't think this is a bug so much as a, less common installation that differs from the "normal" case. - When we create a tcp connection we always start with tcp slow start - This sets the congestion window to effectively 4 packets? - This applies in both directions? - Remote sender responds to my hypothetical http request with the first 4 packets of data - We need to wait one RTT for the ack to come back and now we can send the next 8 packets, - Wait for the next ack and at 16 packets we are now moving at a sensible fraction of the bandwidth delay product? So just to be clear: - We don't seem to have any user-space tuning knobs to influence this right now? - In this age of short attention spans, a couple of extra seconds between clicking something and it responding is worth optimising (IMHO) - I think I need to take this to netdev, but anyone else with any ideas happy to hear them? Thanks Ed W -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |