Prev: 2.6.35 Regression: Ages spent discarding blocks that weren't used!
Next: x86, cpu: RDC doesn't have CPUID, which is what c_ident is
From: Dominik Brodowski on 4 Aug 2010 05:20
Christoph,

On Wed, Aug 04, 2010 at 04:50:39AM -0400, Christoph Hellwig wrote:
> On Wed, Aug 04, 2010 at 09:35:46AM +0200, Dominik Brodowski wrote:
> > How can I best track down the cause of the performance problem,
> > a) without rebooting too often, and
> > b) without breaking up the setup specified above (production system)?
>
> So did you just upgrade the system from an earlier kernel that did not
> show these problems?

No, 2.6.31 to 2.6.34 show similar behaviour.

> Or did no one notice them before?

Well, there are some reports relating to XFS on MD or RAID, though I couldn't
find a resolution to the issues reported, e.g.

- http://kerneltrap.org/mailarchive/linux-raid/2009/10/12/6490333
- http://lkml.indiana.edu/hypermail/linux/kernel/1006.1/00099.html

However, I think we can rule out barriers, as XFS is mounted "nobarrier"
here.

Best,
Dominik
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Dominik Brodowski on 4 Aug 2010 06:30
Hey,

many thanks for your feedback. It seems the crypto step is the culprit:

Reading 1.1 GB with dd, iflag=direct, bs=8k:

/dev/sd* 35.3 MB/s ( 90 %)
/dev/md* 39.1 MB/s (100 %)
/dev/mapper/md*_crypt 3.9 MB/s ( 10 %)
/dev/mapper/vg1-* 3.9 MB/s ( 10 %)

The "good" news: it also happens on my notebook, even though it has a
different setup (no raid, disk -> lv/vg -> crypt). On my notebook, I'm
more than happy to test out different kernel versions, patches etc.

/dev/sd* 17.7 MB/s (100 %)
/dev/mapper/vg1-* 16.2 MB/s ( 92 %)
/dev/mapper/*_crypt 3.1 MB/s ( 18 %)

On a different system, a friend of mine reported (with 2.6.33):

/dev/sd* 51.9 MB/s (100 %)
dm-crypt 32.9 MB/s ( 64 %)

This shows that the speed drop when using dmcrypt is not always a factor of
5 to 10... Btw, it occurs both with aes-lrw-benbi and aes-cbc-essiv:sha256 ,
and (on my notebook) the CPU is mostly idling or waiting.

Best,
Dominik

PS: Bonnie output:

Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.03d ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ABCABCABCABCABC 16G 60186 4 24796 4 53386 5 281.1 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 3176 16 +++++ +++ 4641 28 5501 20 +++++ +++ 2286 9
ABCABCABCABCABCABC,16G,,,60186,4,24796,4,,,53386,5,281.1,1,16,3176,16,+++++,+++,4641,28,5501,20,+++++,+++,2286,9

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Dominik Brodowski on 4 Aug 2010 07:30
On Wed, Aug 04, 2010 at 07:18:03AM -0400, Christoph Hellwig wrote:
> On Wed, Aug 04, 2010 at 12:25:26PM +0200, Dominik Brodowski wrote:
> > Hey,
> >
> > many thanks for your feedback. It seems the crypto step is the culprit:
> >
> > Reading 1.1 GB with dd, iflag=direct, bs=8k:
> >
> > /dev/sd* 35.3 MB/s ( 90 %)
> > /dev/md* 39.1 MB/s (100 %)
> > /dev/mapper/md*_crypt 3.9 MB/s ( 10 %)
> > /dev/mapper/vg1-* 3.9 MB/s ( 10 %)
> >
> > The "good" news: it also happens on my notebook, even though it has a
> > different setup (no raid, disk -> lv/vg -> crypt). On my notebook, I'm
> > more than happy to test out different kernel versions, patches etc.
> >
> > /dev/sd* 17.7 MB/s (100 %)
> > /dev/mapper/vg1-* 16.2 MB/s ( 92 %)
> > /dev/mapper/*_crypt 3.1 MB/s ( 18 %)
>
> The good news is that you have it tracked down, the bad news is that
> I know very little about dm-crypt. Maybe the issue is the single
> threaded decryption in dm-crypt? Can you check how much CPU time
> the dm crypt kernel thread uses?

2 CPUs overall:
Cpu(s): 1.0%us, 5.7%sy, 0.0%ni, 44.8%id, 47.0%wa, 0.0%hi, 1.5%si, 0.0%st

Thanks & best,
Dominik
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
 | 
Pages: 1
Prev: 2.6.35 Regression: Ages spent discarding blocks that weren't used!
Next: x86, cpu: RDC doesn't have CPUID, which is what c_ident is