VFS scalability git tree
in [Kernel]
|
Prev: [PATCH rfc] firewire: cdev: improve FW_CDEV_IOC_ALLOCATE
Next: tmio_mmc: Make ack_mmc_irqs() write-only
From: Christoph Hellwig on 23 Jul 2010 07:20 I might sound like a broken record, but if you want to make forward progress with this split it into smaller series. What would be useful for example would be one series each to split the global inode_lock and dcache_lock, without introducing all the fancy new locking primitives, per-bucket locks and lru schemes for a start. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Dave Chinner on 23 Jul 2010 07:20 On Fri, Jul 23, 2010 at 05:01:00AM +1000, Nick Piggin wrote: > I'm pleased to announce I have a git tree up of my vfs scalability work. > > git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin.git > http://git.kernel.org/?p=linux/kernel/git/npiggin/linux-npiggin.git > > Branch vfs-scale-working I've got a couple of patches needed to build XFS - they shrinker merge left some bad fragments - I'll post them in a minute. This email is for the longest ever lockdep warning I've seen that occurred on boot. Cheers, Dave. [ 6.368707] ====================================================== [ 6.369773] [ INFO: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected ] [ 6.370379] 2.6.35-rc5-dgc+ #58 [ 6.370882] ------------------------------------------------------ [ 6.371475] pmcd/2124 [HC0[0]:SC0[1]:HE1:SE0] is trying to acquire: [ 6.372062] (&sb->s_type->i_lock_key#6){+.+...}, at: [<ffffffff81736f8c>] socket_get_id+0x3c/0x60 [ 6.372268] [ 6.372268] and this task is already holding: [ 6.372268] (&(&hashinfo->ehash_locks[i])->rlock){+.-...}, at: [<ffffffff81791750>] established_get_first+0x60/0x120 [ 6.372268] which would create a new lock dependency: [ 6.372268] (&(&hashinfo->ehash_locks[i])->rlock){+.-...} -> (&sb->s_type->i_lock_key#6){+.+...} [ 6.372268] [ 6.372268] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 6.372268] (&(&hashinfo->ehash_locks[i])->rlock){+.-...} [ 6.372268] ... which became SOFTIRQ-irq-safe at: [ 6.372268] [<ffffffff810b3b26>] __lock_acquire+0x576/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff8177a1ba>] __inet_hash_nolisten+0xfa/0x180 [ 6.372268] [<ffffffff8179392a>] tcp_v4_syn_recv_sock+0x1aa/0x2d0 [ 6.372268] [<ffffffff81795502>] tcp_check_req+0x202/0x440 [ 6.372268] [<ffffffff817948c4>] tcp_v4_do_rcv+0x304/0x4f0 [ 6.372268] [<ffffffff81795134>] tcp_v4_rcv+0x684/0x7e0 [ 6.372268] [<ffffffff81771512>] ip_local_deliver+0xe2/0x1c0 [ 6.372268] [<ffffffff81771af7>] ip_rcv+0x397/0x760 [ 6.372268] [<ffffffff8174d067>] __netif_receive_skb+0x277/0x330 [ 6.372268] [<ffffffff8174d1f4>] process_backlog+0xd4/0x1e0 [ 6.372268] [<ffffffff8174dc38>] net_rx_action+0x188/0x2b0 [ 6.372268] [<ffffffff81084cc2>] __do_softirq+0xd2/0x260 [ 6.372268] [<ffffffff81035edc>] call_softirq+0x1c/0x50 [ 6.372268] [<ffffffff8108551b>] local_bh_enable_ip+0xeb/0xf0 [ 6.372268] [<ffffffff8182c544>] _raw_spin_unlock_bh+0x34/0x40 [ 6.372268] [<ffffffff8173c59e>] release_sock+0x14e/0x1a0 [ 6.372268] [<ffffffff817a3975>] inet_stream_connect+0x75/0x320 [ 6.372268] [<ffffffff81737917>] sys_connect+0xa7/0xc0 [ 6.372268] [<ffffffff81034ff2>] system_call_fastpath+0x16/0x1b [ 6.372268] [ 6.372268] to a SOFTIRQ-irq-unsafe lock: [ 6.372268] (&sb->s_type->i_lock_key#6){+.+...} [ 6.372268] ... which became SOFTIRQ-irq-unsafe at: [ 6.372268] ... [<ffffffff810b3b73>] __lock_acquire+0x5c3/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff8116af72>] new_inode+0x52/0xd0 [ 6.372268] [<ffffffff81174a40>] get_sb_pseudo+0xb0/0x180 [ 6.372268] [<ffffffff81735a41>] sockfs_get_sb+0x21/0x30 [ 6.372268] [<ffffffff81152dba>] vfs_kern_mount+0x8a/0x1e0 [ 6.372268] [<ffffffff81152f29>] kern_mount_data+0x19/0x20 [ 6.372268] [<ffffffff81e1c075>] sock_init+0x4e/0x59 [ 6.372268] [<ffffffff810001dc>] do_one_initcall+0x3c/0x1a0 [ 6.372268] [<ffffffff81de5767>] kernel_init+0x17a/0x204 [ 6.372268] [<ffffffff81035de4>] kernel_thread_helper+0x4/0x10 [ 6.372268] [ 6.372268] other info that might help us debug this: [ 6.372268] [ 6.372268] 3 locks held by pmcd/2124: [ 6.372268] #0: (&p->lock){+.+.+.}, at: [<ffffffff81171dae>] seq_read+0x3e/0x430 [ 6.372268] #1: (&(&hashinfo->ehash_locks[i])->rlock){+.-...}, at: [<ffffffff81791750>] established_get_first+0x60/0x120 [ 6.372268] #2: (clock-AF_INET){++....}, at: [<ffffffff8173b6ae>] sock_i_ino+0x2e/0x70 [ 6.372268] [ 6.372268] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 6.372268] -> (&(&hashinfo->ehash_locks[i])->rlock){+.-...} ops: 3 { [ 6.372268] HARDIRQ-ON-W at: [ 6.372268] [<ffffffff810b3b47>] __lock_acquire+0x597/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff8177a1ba>] __inet_hash_nolisten+0xfa/0x180 [ 6.372268] [<ffffffff8177ab6a>] __inet_hash_connect+0x33a/0x3d0 [ 6.372268] [<ffffffff8177ac4f>] inet_hash_connect+0x4f/0x60 [ 6.372268] [<ffffffff81792522>] tcp_v4_connect+0x272/0x4f0 [ 6.372268] [<ffffffff817a3b8e>] inet_stream_connect+0x28e/0x320 [ 6.372268] [<ffffffff81737917>] sys_connect+0xa7/0xc0 [ 6.372268] [<ffffffff81034ff2>] system_call_fastpath+0x16/0x1b [ 6.372268] IN-SOFTIRQ-W at: [ 6.372268] [<ffffffff810b3b26>] __lock_acquire+0x576/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff8177a1ba>] __inet_hash_nolisten+0xfa/0x180 [ 6.372268] [<ffffffff8179392a>] tcp_v4_syn_recv_sock+0x1aa/0x2d0 [ 6.372268] [<ffffffff81795502>] tcp_check_req+0x202/0x440 [ 6.372268] [<ffffffff817948c4>] tcp_v4_do_rcv+0x304/0x4f0 [ 6.372268] [<ffffffff81795134>] tcp_v4_rcv+0x684/0x7e0 [ 6.372268] [<ffffffff81771512>] ip_local_deliver+0xe2/0x1c0 [ 6.372268] [<ffffffff81771af7>] ip_rcv+0x397/0x760 [ 6.372268] [<ffffffff8174d067>] __netif_receive_skb+0x277/0x330 [ 6.372268] [<ffffffff8174d1f4>] process_backlog+0xd4/0x1e0 [ 6.372268] [<ffffffff8174dc38>] net_rx_action+0x188/0x2b0 [ 6.372268] [<ffffffff81084cc2>] __do_softirq+0xd2/0x260 [ 6.372268] [<ffffffff81035edc>] call_softirq+0x1c/0x50 [ 6.372268] [<ffffffff8108551b>] local_bh_enable_ip+0xeb/0xf0 [ 6.372268] [<ffffffff8182c544>] _raw_spin_unlock_bh+0x34/0x40 [ 6.372268] [<ffffffff8173c59e>] release_sock+0x14e/0x1a0 [ 6.372268] [<ffffffff817a3975>] inet_stream_connect+0x75/0x320 [ 6.372268] [<ffffffff81737917>] sys_connect+0xa7/0xc0 [ 6.372268] [<ffffffff81034ff2>] system_call_fastpath+0x16/0x1b [ 6.372268] INITIAL USE at: [ 6.372268] [<ffffffff810b37e2>] __lock_acquire+0x232/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff8177a1ba>] __inet_hash_nolisten+0xfa/0x180 [ 6.372268] [<ffffffff8177ab6a>] __inet_hash_connect+0x33a/0x3d0 [ 6.372268] [<ffffffff8177ac4f>] inet_hash_connect+0x4f/0x60 [ 6.372268] [<ffffffff81792522>] tcp_v4_connect+0x272/0x4f0 [ 6.372268] [<ffffffff817a3b8e>] inet_stream_connect+0x28e/0x320 [ 6.372268] [<ffffffff81737917>] sys_connect+0xa7/0xc0 [ 6.372268] [<ffffffff81034ff2>] system_call_fastpath+0x16/0x1b [ 6.372268] } [ 6.372268] ... key at: [<ffffffff8285ddf8>] __key.47027+0x0/0x8 [ 6.372268] ... acquired at: [ 6.372268] [<ffffffff810b2940>] check_irq_usage+0x60/0xf0 [ 6.372268] [<ffffffff810b41ff>] __lock_acquire+0xc4f/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff81736f8c>] socket_get_id+0x3c/0x60 [ 6.372268] [<ffffffff8173b6c3>] sock_i_ino+0x43/0x70 [ 6.372268] [<ffffffff81790fc9>] tcp4_seq_show+0x1a9/0x520 [ 6.372268] [<ffffffff81172005>] seq_read+0x295/0x430 [ 6.372268] [<ffffffff811ad9f4>] proc_reg_read+0x84/0xc0 [ 6.372268] [<ffffffff81150165>] vfs_read+0xb5/0x170 [ 6.372268] [<ffffffff81150274>] sys_read+0x54/0x90 [ 6.372268] [<ffffffff81034ff2>] system_call_fastpath+0x16/0x1b [ 6.372268] [ 6.372268] [ 6.372268] the dependencies between the lock to be acquired and SOFTIRQ-irq-unsafe lock: [ 6.372268] -> (&sb->s_type->i_lock_key#6){+.+...} ops: 1185 { [ 6.372268] HARDIRQ-ON-W at: [ 6.372268] [<ffffffff810b3b47>] __lock_acquire+0x597/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff8116af72>] new_inode+0x52/0xd0 [ 6.372268] [<ffffffff81174a40>] get_sb_pseudo+0xb0/0x180 [ 6.372268] [<ffffffff81735a41>] sockfs_get_sb+0x21/0x30 [ 6.372268] [<ffffffff81152dba>] vfs_kern_mount+0x8a/0x1e0 [ 6.372268] [<ffffffff81152f29>] kern_mount_data+0x19/0x20 [ 6.372268] [<ffffffff81e1c075>] sock_init+0x4e/0x59 [ 6.372268] [<ffffffff810001dc>] do_one_initcall+0x3c/0x1a0 [ 6.372268] [<ffffffff81de5767>] kernel_init+0x17a/0x204 [ 6.372268] [<ffffffff81035de4>] kernel_thread_helper+0x4/0x10 [ 6.372268] SOFTIRQ-ON-W at: [ 6.372268] [<ffffffff810b3b73>] __lock_acquire+0x5c3/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff8116af72>] new_inode+0x52/0xd0 [ 6.372268] [<ffffffff81174a40>] get_sb_pseudo+0xb0/0x180 [ 6.372268] [<ffffffff81735a41>] sockfs_get_sb+0x21/0x30 [ 6.372268] [<ffffffff81152dba>] vfs_kern_mount+0x8a/0x1e0 [ 6.372268] [<ffffffff81152f29>] kern_mount_data+0x19/0x20 [ 6.372268] [<ffffffff81e1c075>] sock_init+0x4e/0x59 [ 6.372268] [<ffffffff810001dc>] do_one_initcall+0x3c/0x1a0 [ 6.372268] [<ffffffff81de5767>] kernel_init+0x17a/0x204 [ 6.372268] [<ffffffff81035de4>] kernel_thread_helper+0x4/0x10 [ 6.372268] INITIAL USE at: [ 6.372268] [<ffffffff810b37e2>] __lock_acquire+0x232/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff8116af72>] new_inode+0x52/0xd0 [ 6.372268] [<ffffffff81174a40>] get_sb_pseudo+0xb0/0x180 [ 6.372268] [<f [<ffffffff81152dba>] vfs_kern_mount+0x8a/0x1e0 [ 6.372268] [<ffffffff81152f29>] kern_mount_data+0x19/0x20 [ 6.372268] [<ffffffff81e1c075>] sock_init+0x4e/0x59 [ 6.372268] [<ffffffff810001dc>] do_one_initcall+0x3c/0x1a0 [ 6.372268] [<ffffffff81de5767>] kernel_init+0x17a/0x204 [ 6.372268] [<ffffffff81035de4>] kernel_thread_helper+0x4/0x10 [ 6.372268] } [ 6.372268] ... key at: [<ffffffff81bd5bd8>] sock_fs_type+0x58/0x80 [ 6.372268] ... acquired at: [ 6.372268] [<ffffffff810b2940>] check_irq_usage+0x60/0xf0 [ 6.372268] [<ffffffff810b41ff>] __lock_acquire+0xc4f/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff81736f8c>] socket_get_id+0x3c/0x60 [ 6.372268] [<ffffffff8173b6c3>] sock_i_ino+0x43/0x70 [ 6.372268] [<ffffffff81790fc9>] tcp4_seq_show+0x1a9/0x520 [ 6.372268] [<ffffffff81172005>] seq_read+0x295/0x430 [ 6.372268] [<ffffffff811ad9f4>] proc_reg_read+0x84/0xc0 [ 6.372268] [<ffffffff81150165>] vfs_read+0xb5/0x170 [ 6.372268] [<ffffffff81150274>] sys_read+0x54/0x90 [ 6.372268] [<ffffffff81034ff2>] system_call_fastpath+0x16/0x1b [ 6.372268] [ 6.372268] [ 6.372268] stack backtrace: [ 6.372268] Pid: 2124, comm: pmcd Not tainted 2.6.35-rc5-dgc+ #58 [ 6.372268] Call Trace: [ 6.372268] [<ffffffff810b28d9>] check_usage+0x499/0x4a0 [ 6.372268] [<ffffffff810b24c6>] ? check_usage+0x86/0x4a0 [ 6.372268] [<ffffffff810af729>] ? __bfs+0x129/0x260 [ 6.372268] [<ffffffff810b2940>] check_irq_usage+0x60/0xf0 [ 6.372268] [<ffffffff810b41ff>] __lock_acquire+0xc4f/0x1450 [ 6.372268] [<ffffffff810b4aa6>] lock_acquire+0xa6/0x160 [ 6.372268] [<ffffffff81736f8c>] ? socket_get_id+0x3c/0x60 [ 6.372268] [<ffffffff8182bb26>] _raw_spin_lock+0x36/0x70 [ 6.372268] [<ffffffff81736f8c>] ? socket_get_id+0x3c/0x60 [ 6.372268] [<ffffffff81736f8c>] socket_get_id+0x3c/0x60 [ 6.372268] [<ffffffff8173b6c3>] sock_i_ino+0x43/0x70 [ 6.372268] [<ffffffff81790fc9>] tcp4_seq_show+0x1a9/0x520 [ 6.372268] [<ffffffff81791750>] ? established_get_first+0x60/0x120 [ 6.372268] [<ffffffff8182beb7>] ? _raw_spin_lock_bh+0x67/0x70 [ 6.372268] [<ffffffff81172005>] seq_read+0x295/0x430 [ 6.372268] [<ffffffff81171d70>] ? seq_read+0x0/0x430 [ 6.372268] [<ffffffff811ad9f4>] proc_reg_read+0x84/0xc0 [ 6.372268] [<ffffffff81150165>] vfs_read+0xb5/0x170 [ 6.372268] [<ffffffff81150274>] sys_read+0x54/0x90 [ 6.372268] [<ffffffff81034ff2>] system_call_fastpath+0x16/0x1b -- Dave Chinner david(a)fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Dave Chinner on 23 Jul 2010 10:00 On Fri, Jul 23, 2010 at 05:01:00AM +1000, Nick Piggin wrote: > I'm pleased to announce I have a git tree up of my vfs scalability work. > > git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin.git > http://git.kernel.org/?p=linux/kernel/git/npiggin/linux-npiggin.git > > Branch vfs-scale-working Bug's I've noticed so far: - Using XFS, the existing vfs inode count statistic does not decrease as inodes are free. - the existing vfs dentry count remains at zero - the existing vfs free inode count remains at zero $ pminfo -f vfs.inodes vfs.dentry vfs.inodes.count value 7472612 vfs.inodes.free value 0 vfs.dentry.count value 0 vfs.dentry.free value 0 Performance Summary: With lockdep and CONFIG_XFS_DEBUG enabled, a 16 thread parallel sequential create/unlink workload on an 8p/4GB RAM VM with a virtio block device sitting on a short-stroked 12x2TB SAS array w/ 512MB BBWC in RAID0 via dm and using the noop elevator in the guest VM: $ sudo mkfs.xfs -f -l size=128m -d agcount=16 /dev/vdb meta-data=/dev/vdb isize=256 agcount=16, agsize=1638400 blks = sectsz=512 attr=2 data = bsize=4096 blocks=26214400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal log bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 $ sudo mount -o delaylog,logbsize=262144,nobarrier /dev/vdb /mnt/scratch $ sudo chmod 777 /mnt/scratch $ cd ~/src/fs_mark-3.3/ $ ./fs_mark -S0 -n 500000 -s 0 -d /mnt/scratch/0 -d /mnt/scratch/1 -d /mnt/scratch/3 -d /mnt/scratch/2 -d /mnt/scratch/4 -d /mnt/scratch/5 -d /mnt/scratch/6 -d /mnt/scratch/7 -d /mnt/scratch/8 -d /mnt/scratch/9 -d /mnt/scratch/10 -d /mnt/scratch/11 -d /mnt/scratch/12 -d /mnt/scratch/13 -d /mnt/scratch/14 -d /mnt/scratch/15 files/s 2.6.34-rc4 12550 2.6.35-rc5+scale 12285 So the same within the error margins of the benchmark. Screenshot of monitoring graphs - you can see the effect of the broken stats: http://userweb.kernel.org/~dgc/shrinker-2.6.36/fs_mark-2.6.35-rc4-16x500-xfs.png http://userweb.kernel.org/~dgc/shrinker-2.6.36/fs_mark-2.6.35-rc5-npiggin-scale-lockdep-16x500-xfs.png With a production build (i.e. no lockdep, no xfs debug), I'll run the same fs_mark parallel create/unlink workload to show scalability as I ran here: http://oss.sgi.com/archives/xfs/2010-05/msg00329.html The numbers can't be directly compared, but the test and the setup is the same. The XFS numbers below are with delayed logging enabled. ext4 is using default mkfs and mount parameters except for barrier=0. All numbers are averages of three runs. fs_mark rate (thousands of files/second) 2.6.35-rc5 2.6.35-rc5-scale threads xfs ext4 xfs ext4 1 20 39 20 39 2 35 55 35 57 4 60 41 57 42 8 79 9 75 9 ext4 is getting IO bound at more than 2 threads, so apart from pointing out that XFS is 8-9x faster than ext4 at 8 thread, I'm going to ignore ext4 for the purposes of testing scalability here. For XFS w/ delayed logging, 2.6.35-rc5 is only getting to about 600% CPU and with Nick's patches it's about 650% (10% higher) for slightly lower throughput. So at this class of machine for this workload, the changes result in a slight reduction in scalability. I looked at dbench on XFS as well, but didn't see any significant change in the numbers at up to 200 load threads, so not much to talk about there. Sometime over the weekend I'll build a 16p VM and see what I get from that... Cheers, Dave. -- Dave Chinner david(a)fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Nick Piggin on 23 Jul 2010 11:40 On Fri, Jul 23, 2010 at 05:01:00AM +1000, Nick Piggin wrote: > I'm pleased to announce I have a git tree up of my vfs scalability work. > > git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin.git > http://git.kernel.org/?p=linux/kernel/git/npiggin/linux-npiggin.git > Summary of a few numbers I've run. google's socket teardown workload > runs 3-4x faster on my 2 socket Opteron. Single thread git diff runs 20% > on same machine. 32 node Altix runs dbench on ramfs 150x faster (100MB/s > up to 15GB/s). Following post just contains some preliminary benchmark numbers on a POWER7. Boring if you're not interested in this stuff. IBM and Mikey kindly allowed me to do some test runs on a big POWER7 system today. Very is the only word I'm authorized to describe how big is big. We tested the vfs-scale-working and master branches from my git tree as of today. I'll stick with relative numbers to be safe. All tests were run on ramfs. First and very important is single threaded performance of basic code. POWER7 is obviously vastly different from a Barcelona or Nehalem. and store-free path walk uses a lot of seqlocks, which are cheap on x86, a little more epensive on others. Test case time difference, vanilla to vfs-scale (negative is better) stat() -10.8% +/- 0.3% close(open()) 4.3% +/- 0.3% unlink(creat()) 36.8% +/- 0.3% stat is significantly faster which is really good. open/close is a bit slower which we didn't get time to analyse. There are one or two seqlock checks which might be avoided, which could make up the difference. It's not horrible, but I hope to get POWER7 open/close more competitive (on x86 open/close is even a bit faster). Note this is a worst case for rcu-path-walk: lookup of "./file", because it has to take refcount on the final element. With more elements, rcu walk should gain the advantage. creat/unlink is showing the big RCU penalty. However I have penciled out a working design with Linus of how to do SLAB_DESTROY_BY_RCU. However it makes the store-free path walking and some inode RCU list walking a little bit trickier, so I prefer not to dump too much on at once. There is something that can be done if regressions show up. I don't anticipate many regressions outside microbenchmarks, and this is about the absolute worst case. On to parallel tests. Firstly, the google socket workload. Running with "NR_THREADS" children, vfs-scale patches do this: root(a)p7ih06:~/google# time ./google --files_per_cpu 10000 > /dev/null real 0m4.976s user 8m38.925s sys 6m45.236s root(a)p7ih06:~/google# time ./google --files_per_cpu 20000 > /dev/null real 0m7.816s user 11m21.034s sys 14m38.258s root(a)p7ih06:~/google# time ./google --files_per_cpu 40000 > /dev/null real 0m11.358s user 11m37.955s sys 28m44.911s Reducing to NR_THREADS/4 children allows vanilla to complete: root(a)p7ih06:~/google# time ./google --files_per_cpu 10000 real 1m23.118s user 3m31.820s sys 81m10.405s I was actually surprised it did that well. Dbench was an interesting one. We didn't manage to stretch the box's legs, unfortunately! dbench with 1 proc gave about 500MB/s, 64 procs gave 21GB/s, 128 and throughput dropped dramatically. Turns out that weird things start happening with rename seqlock versus d_lookup, and d_move contention (dbench does a sprinkle of renaming). That can be improved I think, but noth worth bothering with for the time being. It's not really worth testing vanilla at high dbench parallelism. Parallel git diff workload looked OK. It seemed to be scaling fine in the vfs, but it hit a bottlneck in powerpc's tlb invalidation, so numbers may not be so interesting. Lastly, some parallel syscall microbenchmarks: procs vanilla vfs-scale open-close, seperate-cwd 1 384557.70 355923.82 op/s/proc NR_CORES 86.63 164054.64 op/s/proc NR_THREADS 18.68 (ouch!) open-close, same-cwd 1 381074.32 339161.25 NR_CORES 104.16 107653.05 creat-unlink, seperate-cwd 1 145891.05 104301.06 NR_CORES 29.81 10061.66 creat-unlink, same-cwd 1 129681.27 104301.06 NR_CORES 12.68 181.24 So we can see the single thread performance regressions here, but the vanilla case really chokes at high CPU counts. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Nick Piggin on 23 Jul 2010 11:50 On Fri, Jul 23, 2010 at 07:17:46AM -0400, Christoph Hellwig wrote: > I might sound like a broken record, but if you want to make forward > progress with this split it into smaller series. No I appreciate the advice. I put this tree up for people to fetch without posting patches all the time. I think it is important to test and to see the big picture when reviewing the patches, but you are right about how to actually submit patches on the ML. > What would be useful for example would be one series each to split > the global inode_lock and dcache_lock, without introducing all the > fancy new locking primitives, per-bucket locks and lru schemes for > a start. I've kept the series fairly well structured like that. Basically it is in these parts: 1. files lock 2. vfsmount lock 3. mnt refcount 4a. put several new global spinlocks around different parts of dcache 4b. remove dcache_lock after the above protect everything 4c. start doing fine grained locking of hash, inode alias, lru, etc etc 5a, 5b, 5c. same for inodes 6. some further optimisations and cleanups 7. store-free path walking This kind of sequence. I will again try to submit a first couple of things to Al soon. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
|
Next
|
Last
Pages: 1 2 3 4 Prev: [PATCH rfc] firewire: cdev: improve FW_CDEV_IOC_ALLOCATE Next: tmio_mmc: Make ack_mmc_irqs() write-only |