cp slows machine
in [Slackware]
|
From: Mike Jones on 16 Dec 2009 17:51 Responding to Jerry Peters: > Grant <g_r_a_n_t_(a)bugsplatter.id.au> wrote: >> On Tue, 15 Dec 2009 15:20:31 -0600, Aaron W. Hsu <arcfide(a)sacrideo.us> >> wrote: >> >>>Mike Jones <Not(a)Arizona.Bay> writes: >>> >>>>Next up, how to I force applications to RAM instead of disk? ;) >>> >>>You need to find out how they are using the disk first. For that, you >>>need to actually do the top based analysis on your programs. >> >> Anything read from disk is buffered in memory automagically, so where's >> the gain? >> >> Grant. > > But isn't that part of his problem? A large copy evicts useful stuff > from memory to buffer something that will be read exactly once. > > Jerry That sounds about right, though I wouldn't trust me on this. ;\ -- *=( http://www.thedailymash.co.uk/ *=( For all your UK news needs.
From: Aaron W. Hsu on 16 Dec 2009 19:42 Grant <g_r_a_n_t_(a)bugsplatter.id.au> writes: >On Tue, 15 Dec 2009 15:20:31 -0600, Aaron W. Hsu <arcfide(a)sacrideo.us> wrote: >>Mike Jones <Not(a)Arizona.Bay> writes: >> >>>Next up, how to I force applications to RAM instead of disk? ;) >> >>You need to find out how they are using the disk first. For that, you >>need to actually do the top based analysis on your programs. >Anything read from disk is buffered in memory automagically, so >where's the gain? Theoretically this is supposed to work well, but in practice, I have found that some applications are really bad about this. For some reason, they just can't seem to get their I/O right. Aaron W. Hsu -- A professor is one who talks in someone else's sleep.
From: Jerry Peters on 17 Dec 2009 16:04 Grant <g_r_a_n_t_(a)bugsplatter.id.au> wrote: > On Wed, 16 Dec 2009 21:42:53 +0000 (UTC), Jerry Peters <jerry(a)example.invalid> wrote: > >>Grant <g_r_a_n_t_(a)bugsplatter.id.au> wrote: >>> On Tue, 15 Dec 2009 15:20:31 -0600, Aaron W. Hsu <arcfide(a)sacrideo.us> wrote: >>> >>>>Mike Jones <Not(a)Arizona.Bay> writes: >>>> >>>>>Next up, how to I force applications to RAM instead of disk? ;) >>>> >>>>You need to find out how they are using the disk first. For that, you >>>>need to actually do the top based analysis on your programs. >>> >>> Anything read from disk is buffered in memory automagically, so >>> where's the gain? >>> >>> Grant. >> >>But isn't that part of his problem? A large copy evicts useful stuff >>from memory to buffer something that will be read exactly once. > > Hmm, yes. I thought they fixed that? Been a long time since I cared > enough to explore the VM (virtual memory) settings -- so that might > still be the case. But how? How would the kernel know you're copying a file, unless cp or whatever program you're using tells it by posix_fadvise? > > So the question may be better put as 'how do I limit buffer space?'. > > Grant. I had a similiar problem with dump to a slow device like DVD-RAM. It would get so bad that the machine would appear to hang for long periods of time. I solved it by calling fdatasync & posix_fadvise dontneed every 16 MB of output data written. Dumping a filesystem to DVD-RAM is still slow but at least the computer remains usable. There's been some recent work on per BDI flush behavior which might make my hack to dump unnecessary, but I haven't tested it. Jerry
From: Grant on 17 Dec 2009 17:14
On Thu, 17 Dec 2009 21:04:39 +0000 (UTC), Jerry Peters <jerry(a)example.invalid> wrote: >Grant <g_r_a_n_t_(a)bugsplatter.id.au> wrote: >> On Wed, 16 Dec 2009 21:42:53 +0000 (UTC), Jerry Peters <jerry(a)example.invalid> wrote: >> >>>Grant <g_r_a_n_t_(a)bugsplatter.id.au> wrote: >>>> On Tue, 15 Dec 2009 15:20:31 -0600, Aaron W. Hsu <arcfide(a)sacrideo.us> wrote: >>>> >>>>>Mike Jones <Not(a)Arizona.Bay> writes: >>>>> >>>>>>Next up, how to I force applications to RAM instead of disk? ;) >>>>> >>>>>You need to find out how they are using the disk first. For that, you >>>>>need to actually do the top based analysis on your programs. >>>> >>>> Anything read from disk is buffered in memory automagically, so >>>> where's the gain? >>>> >>>> Grant. >>> >>>But isn't that part of his problem? A large copy evicts useful stuff >>>from memory to buffer something that will be read exactly once. >> >> Hmm, yes. I thought they fixed that? Been a long time since I cared >> enough to explore the VM (virtual memory) settings -- so that might >> still be the case. > >But how? How would the kernel know you're copying a file, unless cp or >whatever program you're using tells it by posix_fadvise? Well, the Linux tunables live here: grant(a)pooh64:~$ cat /etc/slackware-version Slackware 13.0.0.0.0 grant(a)pooh64:~$ ls /proc/sys/vm/ block_dump legacy_va_layout overcommit_ratio dirty_background_bytes lowmem_reserve_ratio page-cluster dirty_background_ratio max_map_count panic_on_oom dirty_bytes min_free_kbytes percpu_pagelist_fraction dirty_expire_centisecs mmap_min_addr scan_unevictable_pages dirty_ratio nr_pdflush_threads stat_interval dirty_writeback_centisecs oom_dump_tasks swappiness drop_caches oom_kill_allocating_task vfs_cache_pressure laptop_mode overcommit_memory If you downlaod and install: http://www.kernel.org/pub/linux/docs/man-pages/man-pages-3.23.tar.bz2 it describes only a few of the settings -- I didn't look further as I'm not fussed by system performance at the moment, I tend to kick off large copy jobs and come back way later. Grant. -- http://bugsplatter.id.au |