Prev: SysRq: do not pass tty argument around
Next: 2.6.35 Regression: Ages spent discarding blocks that weren't used!
From: Christoph Hellwig on 4 Aug 2010 05:00
On Wed, Aug 04, 2010 at 09:35:46AM +0200, Dominik Brodowski wrote:
> How can I best track down the cause of the performance problem,
> a) without rebooting too often, and
> b) without breaking up the setup specified above (production system)?

So did you just upgrade the system from an earlier kernel that did not
show these problems? Or did no one notice them before?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Christoph Hellwig on 4 Aug 2010 05:30
On Wed, Aug 04, 2010 at 11:13:17AM +0200, Dominik Brodowski wrote:
> > show these problems?
>
> No, 2.6.31 to 2.6.34 show similar behaviour.

Ok, so it's been around for a while. Can you test the write speed of
each individual device layer by doing a large read from it, using:

dd if=<device> of=/dev/null bs=8k iflag=direct

where device starts with the /dev/sda* device, and goes up to the MD
device, the dm-crypt device and the LV. And yes, it's safe to read
from the device while it's otherwise mounted/used.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Christoph Hellwig on 4 Aug 2010 07:20
On Wed, Aug 04, 2010 at 12:25:26PM +0200, Dominik Brodowski wrote:
> Hey,
>
> many thanks for your feedback. It seems the crypto step is the culprit:
>
> Reading 1.1 GB with dd, iflag=direct, bs=8k:
>
> /dev/sd* 35.3 MB/s ( 90 %)
> /dev/md* 39.1 MB/s (100 %)
> /dev/mapper/md*_crypt 3.9 MB/s ( 10 %)
> /dev/mapper/vg1-* 3.9 MB/s ( 10 %)
>
> The "good" news: it also happens on my notebook, even though it has a
> different setup (no raid, disk -> lv/vg -> crypt). On my notebook, I'm
> more than happy to test out different kernel versions, patches etc.
>
> /dev/sd* 17.7 MB/s (100 %)
> /dev/mapper/vg1-* 16.2 MB/s ( 92 %)
> /dev/mapper/*_crypt 3.1 MB/s ( 18 %)

The good news is that you have it tracked down, the bad news is that
I know very little about dm-crypt. Maybe the issue is the single
threaded decryption in dm-crypt? Can you check how much CPU time
the dm crypt kernel thread uses?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Mikael Abrahamsson on 4 Aug 2010 08:00
On Wed, 4 Aug 2010, Christoph Hellwig wrote:

> The good news is that you have it tracked down, the bad news is that I
> know very little about dm-crypt. Maybe the issue is the single threaded
> decryption in dm-crypt? Can you check how much CPU time the dm crypt
> kernel thread uses?

I'm not sure it's that. I have a Core i5 with AES-NI and that didn't
significantly increase my overall performance, as it's not there the
bottleneck is (at least in my system).

I earlier sent out an email wondering if someone could shed some light on
how scheduling, block caching and read-ahead works together when one does
disks->md->crypto->lvm->fs, becase that's a lot of layers and potentially
a lot of unneeded buffering, readahead and scheduling magic?

--
Mikael Abrahamsson email: swmike(a)swm.pp.se
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Mike Snitzer on 4 Aug 2010 09:00
On Wed, Aug 04 2010 at 7:53am -0400,
Mikael Abrahamsson <swmike(a)swm.pp.se> wrote:

> On Wed, 4 Aug 2010, Christoph Hellwig wrote:
>
> >The good news is that you have it tracked down, the bad news is
> >that I know very little about dm-crypt. Maybe the issue is the
> >single threaded decryption in dm-crypt? Can you check how much
> >CPU time the dm crypt kernel thread uses?
>
> I'm not sure it's that. I have a Core i5 with AES-NI and that didn't
> significantly increase my overall performance, as it's not there the
> bottleneck is (at least in my system).

You could try applying both of these patches that are pending review for
hopeful inclussion in 2.6.36:

https://patchwork.kernel.org/patch/103404/
https://patchwork.kernel.org/patch/112657/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
 | 
Pages: 1
Prev: SysRq: do not pass tty argument around
Next: 2.6.35 Regression: Ages spent discarding blocks that weren't used!