cfq-iosched: IOPS mode for group scheduling and new group_idle tunable
in [Kernel]
|
Prev: [PATCH] PCI: MSI: Restore read_msi_msg_desc(); add get_cached_msi_msg_desc()
Next: [PATCH 2/2] xfs: shrinker should use a per-filesystem scan count
From: Heinz Diehl on 23 Jul 2010 10:10 On 23.07.2010, Vivek Goyal wrote: > This is V4 of the patchset which implements a new tunable group_idle and also > implements IOPS mode for group fairness. Following are changes since V3. [....] Just for information: this patchset, applied to 2.6.35-rc6, gives about 20-25% increase in speed/throughput on my desktop system (Phenom 2.5GHz Quadcore, 3 disks) with the tunables set according to what you've used/reported here (the setup with slice_idle set to 0), and it's measurable with fs_mark, too. After 2 hours of hard testing, the machine remains stable and responsive. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Vivek Goyal on 23 Jul 2010 10:20 On Fri, Jul 23, 2010 at 04:03:43PM +0200, Heinz Diehl wrote: > On 23.07.2010, Vivek Goyal wrote: > > > This is V4 of the patchset which implements a new tunable group_idle and also > > implements IOPS mode for group fairness. Following are changes since V3. > [....] > > Just for information: this patchset, applied to 2.6.35-rc6, gives about > 20-25% increase in speed/throughput on my desktop system > (Phenom 2.5GHz Quadcore, 3 disks) with the tunables set according > to what you've used/reported here (the setup with slice_idle set to 0), > and it's measurable with fs_mark, too. > > After 2 hours of hard testing, the machine remains stable and responsive. Thanks for some testing Heinz. I am assuming you are not using cgroups and blkio controller. In that case, you are seeing improvements probably due to first patch where we don't idle on service tree if slice_idle=0. Hence we cut down on overall idling and can see throughput incrase. What kind of configuration these 3 disks are on your system? Some Hardare RAID or software RAID ? Thanks Vivek -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Heinz Diehl on 23 Jul 2010 11:00 On 23.07.2010, Vivek Goyal wrote: > Thanks for some testing Heinz. I am assuming you are not using cgroups > and blkio controller. Not at all. > In that case, you are seeing improvements probably due to first patch > where we don't idle on service tree if slice_idle=0. Hence we cut down on > overall idling and can see throughput incrase. Hmm, in any case it's not getting worse by setting slice_idle to 8. My main motivation to test your patches was that I thought the other way 'round, and was just curious on how this patchset will affect machines which are NOT a high end server/storage system :-) > What kind of configuration these 3 disks are on your system? Some Hardare > RAID or software RAID ? Just 3 SATA disks plugged into the onboard controller, no RAID or whatsoever. I used fs_mark for testing: "fs_mark -S 1 -D 10000 -N 100000 -d /home/htd/fsmark/test -s 65536 -t 1 -w 4096 -F" These are the results with plain cfq (2.6.35-rc6) and the settings which gave the best speed/throughput on my machine: low_latency = 0 slice_idle = 4 quantum = 32 Setting slice_idle to 0 didn't improve anything, I tried this before. FSUse% Count Size Files/sec App Overhead 27 1000 65536 360.3 34133 27 2000 65536 384.4 34657 27 3000 65536 401.1 32994 27 4000 65536 394.3 33781 27 5000 65536 406.8 32569 27 6000 65536 401.9 34001 27 7000 65536 374.5 33192 27 8000 65536 398.3 32839 27 9000 65536 405.2 34110 27 10000 65536 398.9 33887 27 11000 65536 402.3 34111 27 12000 65536 398.1 33652 27 13000 65536 412.9 32443 27 14000 65536 408.1 32197 And this is after applying your patchset, with your settings (and slice_idle = 0): FSUse% Count Size Files/sec App Overhead 27 1000 65536 600.7 29579 27 2000 65536 568.4 30650 27 3000 65536 522.0 29171 27 4000 65536 534.1 29751 27 5000 65536 550.7 30168 27 6000 65536 521.7 30158 27 7000 65536 493.3 29211 27 8000 65536 495.3 30183 27 9000 65536 587.8 29881 27 10000 65536 469.9 29602 27 11000 65536 482.7 29557 27 12000 65536 486.6 30700 27 13000 65536 516.1 30243 There's some 2-3% further improvement on my system with these settings, which after som fiddling turned out to give most performance here (don't need the group settings, of course): group_idle = 0 group_isolation = 0 low_latency = 1 quantum = 8 slice_idle = 8 Thanks, Heinz. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Vivek Goyal on 23 Jul 2010 14:40 On Fri, Jul 23, 2010 at 04:56:31PM +0200, Heinz Diehl wrote: > On 23.07.2010, Vivek Goyal wrote: > > > Thanks for some testing Heinz. I am assuming you are not using cgroups > > and blkio controller. > > Not at all. > > > In that case, you are seeing improvements probably due to first patch > > where we don't idle on service tree if slice_idle=0. Hence we cut down on > > overall idling and can see throughput incrase. > > Hmm, in any case it's not getting worse by setting slice_idle to 8. > > My main motivation to test your patches was that I thought > the other way 'round, and was just curious on how this patchset > will affect machines which are NOT a high end server/storage system :-) > > > What kind of configuration these 3 disks are on your system? Some Hardare > > RAID or software RAID ? > > Just 3 SATA disks plugged into the onboard controller, no RAID or whatsoever. > > I used fs_mark for testing: > "fs_mark -S 1 -D 10000 -N 100000 -d /home/htd/fsmark/test -s 65536 -t 1 -w 4096 -F" > > These are the results with plain cfq (2.6.35-rc6) and the settings which > gave the best speed/throughput on my machine: > > low_latency = 0 > slice_idle = 4 > quantum = 32 > > Setting slice_idle to 0 didn't improve anything, I tried this before. > > FSUse% Count Size Files/sec App Overhead > 27 1000 65536 360.3 34133 > 27 2000 65536 384.4 34657 > 27 3000 65536 401.1 32994 > 27 4000 65536 394.3 33781 > 27 5000 65536 406.8 32569 > 27 6000 65536 401.9 34001 > 27 7000 65536 374.5 33192 > 27 8000 65536 398.3 32839 > 27 9000 65536 405.2 34110 > 27 10000 65536 398.9 33887 > 27 11000 65536 402.3 34111 > 27 12000 65536 398.1 33652 > 27 13000 65536 412.9 32443 > 27 14000 65536 408.1 32197 > > > And this is after applying your patchset, with your settings > (and slice_idle = 0): > > FSUse% Count Size Files/sec App Overhead > 27 1000 65536 600.7 29579 > 27 2000 65536 568.4 30650 > 27 3000 65536 522.0 29171 > 27 4000 65536 534.1 29751 > 27 5000 65536 550.7 30168 > 27 6000 65536 521.7 30158 > 27 7000 65536 493.3 29211 > 27 8000 65536 495.3 30183 > 27 9000 65536 587.8 29881 > 27 10000 65536 469.9 29602 > 27 11000 65536 482.7 29557 > 27 12000 65536 486.6 30700 > 27 13000 65536 516.1 30243 > I think that above improvement is due to first patch and changes in cfq_should_idle(). cfq_should_idle() used to return 1 even if slice_idle=0 and that created bottlenecks at some places like in select_queue() we will not expire a queue till request from that queue completed. This stopped a new queue from dispatching requests etc... Anyway, for fs_mark problem, can you give following patch a try. https://patchwork.kernel.org/patch/113061/ Above patch should improve your fs_mark numbers even without setting slice_idle=0. Thanks Vivek -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Heinz Diehl on 24 Jul 2010 04:10
On 23.07.2010, Vivek Goyal wrote: > Anyway, for fs_mark problem, can you give following patch a try. > https://patchwork.kernel.org/patch/113061/ Ported it to 2.6.35-rc6, and these are my results using the same fs_mark call as before: slice_idle = 0 FSUse% Count Size Files/sec App Overhead 28 1000 65536 241.6 39574 28 2000 65536 231.1 39939 28 3000 65536 230.4 39722 28 4000 65536 243.2 39646 28 5000 65536 227.0 39892 28 6000 65536 224.1 39555 28 7000 65536 228.2 39761 28 8000 65536 235.3 39766 28 9000 65536 237.3 40518 28 10000 65536 225.7 39861 28 11000 65536 227.2 39441 slice_idle = 8 FSUse% Count Size Files/sec App Overhead 28 1000 65536 502.2 30545 28 2000 65536 407.6 29406 28 3000 65536 381.8 30152 28 4000 65536 438.1 30038 28 5000 65536 447.5 30477 28 6000 65536 422.0 29610 28 7000 65536 383.1 30327 28 8000 65536 415.3 30102 28 9000 65536 397.6 31013 28 10000 65536 401.4 29201 28 11000 65536 408.8 29720 28 12000 65536 391.2 29157 Huh...there's quite a difference! It's definitely the slice_idle settings which affect the results here. Besides, this patch gives noticeably bad desktop interactivity on my system. Don't know if this is related, but I'm not quite shure if XFS (which I use exclusively) uses the jbd/jbd2 journaling layer at all. Thanks, Heinz. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |