Prev: CONFIDENTIAL
Next: asm-generic: add NEED_SG_DMA_LENGTH to define sg_dma_len()
From: Taras Glek on 5 Apr 2010 19:10
Hello,
I am working on improving Mozilla startup times. It turns out that page
faults(caused by lack of cooperation between user/kernelspace) are the
main cause of slow startup. I need some insights from someone who
understands linux vm behavior.

Current Situation:
The dynamic linker mmap()s executable and data sections of our
executable but it doesn't call madvise().
By default page faults trigger 131072byte reads. To make matters worse,
the compile-time linker + gcc lay out code in a manner that does not
correspond to how the resulting executable will be executed(ie the
layout is basically random). This means that during startup 15-40mb
binaries are read in basically random fashion. Even if one orders the
binary optimally, throughput is still suboptimal due to the puny readahead.

IO Hints:
Fortunately when one specifies madvise(WILLNEED) pagefaults trigger 2mb
reads and a binary that tends to take 110 page faults(ie program stops
execution and waits for disk) can be reduced down to 6. This has the
potential to double application startup of large apps without any clear
downsides. Suse ships their glibc with a dynamic linker patch to
fadvise() dynamic libraries(not sure why they switched from doing
madvise before).

I filed a glibc bug about this at
http://sourceware.org/bugzilla/show_bug.cgi?id=11431 . Uli commented
with his concern about wasting memory resources. What is the impact of
madvise(WILLNEED) or the fadvise equivalent on systems under memory
pressure? Does the kernel simply start ignoring these hints?

Also, once an application is started is it reasonable to keep it
madvise(WILLNEED)ed or should the madvise flags be reset?

Perhaps the kernel could monitor the page-in patterns to increase the
readahead sizes? This may already happen, I've noticed that a handful of
pagefaults trigger > 131072bytes of IO, perhaps this just needs tweaking.

Thanks,
Taras Glek

PS. For more details on this issue see my blog at
https://blog.mozilla.com/tglek/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Dave Chinner on 5 Apr 2010 19:20
On Mon, Apr 05, 2010 at 03:43:02PM -0700, Taras Glek wrote:
> Hello,
> I am working on improving Mozilla startup times. It turns out that
> page faults(caused by lack of cooperation between user/kernelspace)
> are the main cause of slow startup. I need some insights from
> someone who understands linux vm behavior.
>
> Current Situation:
> The dynamic linker mmap()s executable and data sections of our
> executable but it doesn't call madvise().
> By default page faults trigger 131072byte reads. To make matters

Try tuning /sys/block/<dev>/queue/read_ahead_kb and see if that
makes any difference - that's the default maximum readahead for the
given block device and defaults to 128k.

There has been some recent work to increase the default readahead
size, so if changing the default improves performance then perhaps
a fix for your problem is already in the works?

Cheers,

Dave.
--
Dave Chinner
david(a)fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Roland Dreier on 5 Apr 2010 20:00
Almost certainly teaching my grandmother to suck eggs, but are you aware
of the work Michael Meeks has done on improving openoffice.org startup time?
--
Roland Dreier <rolandd(a)cisco.com> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Johannes Weiner on 6 Apr 2010 06:00
On Mon, Apr 05, 2010 at 03:43:02PM -0700, Taras Glek wrote:
> Hello,
> I am working on improving Mozilla startup times. It turns out that page
> faults(caused by lack of cooperation between user/kernelspace) are the
> main cause of slow startup. I need some insights from someone who
> understands linux vm behavior.
>
> Current Situation:
> The dynamic linker mmap()s executable and data sections of our
> executable but it doesn't call madvise().
> By default page faults trigger 131072byte reads. To make matters worse,
> the compile-time linker + gcc lay out code in a manner that does not
> correspond to how the resulting executable will be executed(ie the
> layout is basically random). This means that during startup 15-40mb
> binaries are read in basically random fashion. Even if one orders the
> binary optimally, throughput is still suboptimal due to the puny readahead.
>
> IO Hints:
> Fortunately when one specifies madvise(WILLNEED) pagefaults trigger 2mb
> reads and a binary that tends to take 110 page faults(ie program stops
> execution and waits for disk) can be reduced down to 6. This has the
> potential to double application startup of large apps without any clear
> downsides. Suse ships their glibc with a dynamic linker patch to
> fadvise() dynamic libraries(not sure why they switched from doing
> madvise before).
>
> I filed a glibc bug about this at
> http://sourceware.org/bugzilla/show_bug.cgi?id=11431 . Uli commented
> with his concern about wasting memory resources. What is the impact of
> madvise(WILLNEED) or the fadvise equivalent on systems under memory
> pressure? Does the kernel simply start ignoring these hints?

It will throttle based on memory pressure. In idle situations it will
eat your file cache, however, to satisfy the request.

Now, the file cache should be much bigger than the amount of unneeded
pages you prefault with the hint over the whole library, so I guess the
benefit of prefaulting the right pages outweighs the downside of evicting
some cache for unused library pages.

Still, it's a workaround for deficits in the demand-paging/readahead
heuristics and thus a bit ugly, I feel. Maybe Wu can help.

> Also, once an application is started is it reasonable to keep it
> madvise(WILLNEED)ed or should the madvise flags be reset?

It's a one-time operation that starts immediate readahead, no permanent
changes are done.

> Perhaps the kernel could monitor the page-in patterns to increase the
> readahead sizes? This may already happen, I've noticed that a handful of
> pagefaults trigger > 131072bytes of IO, perhaps this just needs tweaking.

CCd the man :-)

> Thanks,
> Taras Glek
>
> PS. For more details on this issue see my blog at
> https://blog.mozilla.com/tglek/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo(a)vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Taras Glek on 6 Apr 2010 18:00
On 04/06/2010 02:51 AM, Johannes Weiner wrote:
> On Mon, Apr 05, 2010 at 03:43:02PM -0700, Taras Glek wrote:
>
>> Hello,
>> I am working on improving Mozilla startup times. It turns out that page
>> faults(caused by lack of cooperation between user/kernelspace) are the
>> main cause of slow startup. I need some insights from someone who
>> understands linux vm behavior.
>>
>> Current Situation:
>> The dynamic linker mmap()s executable and data sections of our
>> executable but it doesn't call madvise().
>> By default page faults trigger 131072byte reads. To make matters worse,
>> the compile-time linker + gcc lay out code in a manner that does not
>> correspond to how the resulting executable will be executed(ie the
>> layout is basically random). This means that during startup 15-40mb
>> binaries are read in basically random fashion. Even if one orders the
>> binary optimally, throughput is still suboptimal due to the puny readahead.
>>
>> IO Hints:
>> Fortunately when one specifies madvise(WILLNEED) pagefaults trigger 2mb
>> reads and a binary that tends to take 110 page faults(ie program stops
>> execution and waits for disk) can be reduced down to 6. This has the
>> potential to double application startup of large apps without any clear
>> downsides. Suse ships their glibc with a dynamic linker patch to
>> fadvise() dynamic libraries(not sure why they switched from doing
>> madvise before).
>>
>> I filed a glibc bug about this at
>> http://sourceware.org/bugzilla/show_bug.cgi?id=11431 . Uli commented
>> with his concern about wasting memory resources. What is the impact of
>> madvise(WILLNEED) or the fadvise equivalent on systems under memory
>> pressure? Does the kernel simply start ignoring these hints?
>>
> It will throttle based on memory pressure. In idle situations it will
> eat your file cache, however, to satisfy the request.
>
Define idle situations. Do you mean that madv(willneed) will aggresively
readahead, but only while cpu(or disk?) is idle?
I am trying to optimize application startup which means that the cpu is
busy while not blocked on io.
> Now, the file cache should be much bigger than the amount of unneeded
> pages you prefault with the hint over the whole library, so I guess the
> benefit of prefaulting the right pages outweighs the downside of evicting
> some cache for unused library pages.
>
> Still, it's a workaround for deficits in the demand-paging/readahead
> heuristics and thus a bit ugly, I feel. Maybe Wu can help.
>
>
Can't wait to hear the juicy details.
>> Also, once an application is started is it reasonable to keep it
>> madvise(WILLNEED)ed or should the madvise flags be reset?
>>
> It's a one-time operation that starts immediate readahead, no permanent
> changes are done.
>
I may be measuring this wrong, but in my experience the only change
madvise(willneed) does in increase the length parameter to
__do_page_cache_readahead(). My script is at
http://hg.mozilla.org/users/tglek_mozilla.com/startup/file/6453ad2a7906/kernelio.stp
..


Taras
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
 |  Next  |  Last
Pages: 1 2 3 4 5 6 7
Prev: CONFIDENTIAL
Next: asm-generic: add NEED_SG_DMA_LENGTH to define sg_dma_len()