Poor interactive performance with I/O loads with fsync()ing
in [Kernel]
|
Prev: [PATCH 1/1] NET: netpoll, fix potential NULL ptr dereference
Next: Failure with the 2.6.34-rc1 kernel
From: Ben Gamari on 9 Apr 2010 11:00 On Mon, 29 Mar 2010 00:08:58 +0200, Andi Kleen <andi(a)firstfloor.org> wrote: > Ben Gamari <bgamari.foss(a)gmail.com> writes: > ext4/XFS/JFS/btrfs should be better in this regard > I am using btrfs, so yes, I was expecting things to be better. Unfortunately, the improvement seems to be non-existent under high IO/fsync load. - Ben -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Ben Gamari on 9 Apr 2010 11:30 On Wed, 17 Mar 2010 10:37:04 +0100, Ingo Molnar <mingo(a)elte.hu> wrote: > * Nick Piggin <npiggin(a)suse.de> wrote: > > > Hi, > > > > On Tue, Mar 16, 2010 at 08:31:12AM -0700, Ben Gamari wrote: > > > Hey all, > > > > > > Recently I started using the Xapian-based notmuch mail client for everyday > > > use. One of the things I was quite surprised by after the switch was the > > > incredible hit in interactive performance that is observed during database > > > updates. Things are particularly bad during runs of 'notmuch new,' which scans > > > the file system looking for new messages and adds them to the database. > > > Specifically, the worst of the performance hit appears to occur when the > > > database is being updated. > > > > > > During these periods, even small chunks of I/O can become minute-long ordeals. > > > It is common for latencytop to show 30 second long latencies for page faults > > > and writing pages. Interactive performance is absolutely abysmal, with other > > > unrelated processes feeling horrible latencies, causing media players, > > > editors, and even terminals to grind to a halt. > > > > > > Despite the system being clearly I/O bound, iostat shows pitiful disk > > > throughput (700kByte/second read, 300 kByte/second write). Certainly this poor > > > performance can, at least to some degree, be attributable to the fact that > > > Xapian uses fdatasync() to ensure data consistency. That being said, it seems > > > like Xapian's page usage causes horrible thrashing, hence the performance hit > > > on unrelated processes. > > > > Where are the unrelated processes waiting? Can you get a sample of several > > backtraces? (/proc/<pid>/stack should do it) > > A call-graph profile will show the precise reason for IO latencies, and their > relatively likelihood. > Are these backtraces at all useful? I've received no feedback thusfar, so I can only that either, a) there is insufficient data to draw any conclusions and there is no interest in pursuing this further, or b) nobody has looked at the backtraces As I've said in the past, I am very interested in seeing this problem looked at and would love to contribute whatever I can to that effort. However, without knowing what information is necessary, I can be of only very limited use in my own debugging efforts. Thanks, - Ben -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 11 Apr 2010 11:10 On 04/09/2010 05:56 PM, Ben Gamari wrote: > On Mon, 29 Mar 2010 00:08:58 +0200, Andi Kleen<andi(a)firstfloor.org> wrote: > >> Ben Gamari<bgamari.foss(a)gmail.com> writes: >> ext4/XFS/JFS/btrfs should be better in this regard >> >> > I am using btrfs, so yes, I was expecting things to be better. Unfortunately, > the improvement seems to be non-existent under high IO/fsync load. > > btrfs is known to perform poorly under fsync. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Ben Gamari on 11 Apr 2010 12:40 On Sun, 11 Apr 2010 18:03:00 +0300, Avi Kivity <avi(a)redhat.com> wrote: > On 04/09/2010 05:56 PM, Ben Gamari wrote: > > On Mon, 29 Mar 2010 00:08:58 +0200, Andi Kleen<andi(a)firstfloor.org> wrote: > > > >> Ben Gamari<bgamari.foss(a)gmail.com> writes: > >> ext4/XFS/JFS/btrfs should be better in this regard > >> > >> > > I am using btrfs, so yes, I was expecting things to be better. Unfortunately, > > the improvement seems to be non-existent under high IO/fsync load. > > > > btrfs is known to perform poorly under fsync. > Has the reason for this been identified? Judging from the nature of metadata loads, it would seem that it should be substantially easier to implement fsync() efficiently. - Ben -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andi Kleen on 11 Apr 2010 13:30
> Has the reason for this been identified? Judging from the nature of metadata > loads, it would seem that it should be substantially easier to implement > fsync() efficiently. By design a copy on write tree fs would need to flush a whole tree hierarchy on a sync. btrfs avoids this by using a special log for fsync, but that causes more overhead if you have that log on the same disk. So IO subsystem will do more work. It's a bit like JBD data journaling. However it should not have the stalls inherent in ext3's journaling. -Andi -- ak(a)linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |