patch: preload dictionary new version
in [PgSql]
|
Prev: ALTER TABLE SET STATISTICS requires AccessExclusiveLock
Next: ALTER TABLE SET STATISTICS requiresAccessExclusiveLock
From: Tom Lane on 12 Jul 2010 00:02 Itagaki Takahiro <itagaki.takahiro(a)gmail.com> writes: > 2010/7/8 Tom Lane <tgl(a)sss.pgh.pa.us>: >> For example, the dictionary-load code could automatically execute >> the precompile step if it observed that the precompiled copy of the >> dictionary was missing or had an older file timestamp than the source. > There might be a problem in automatic precompiler -- Where should we > save the result? OS users of postgres servers don't have write-permission > to $PGSHARE in normal cases. Instead, we can store the precompiled > result to $PGDATA/pg_dict_cache or so. Yeah. Actually we'd *have* to do something like that because $PGSHARE should contain only architecture-independent files, while the precompiled files would presumably have dependencies on endianness etc. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Pavel Stehule on 12 Jul 2010 03:30 2010/7/12 Tom Lane <tgl(a)sss.pgh.pa.us>: > Itagaki Takahiro <itagaki.takahiro(a)gmail.com> writes: >> 2010/7/8 Tom Lane <tgl(a)sss.pgh.pa.us>: >>> For example, the dictionary-load code could automatically execute >>> the precompile step if it observed that the precompiled copy of the >>> dictionary was missing or had an older file timestamp than the source. I am not sure, but it can be recompiled when tseach code is actualised (minor update) too. > >> There might be a problem in automatic precompiler -- Where should we >> save the result? OS users of postgres servers don't have write-permission >> to $PGSHARE in normal cases. Instead, we can store the precompiled >> result to $PGDATA/pg_dict_cache or so. > > Yeah. Actually we'd *have* to do something like that because $PGSHARE > should contain only architecture-independent files, while the > precompiled files would presumably have dependencies on endianness etc. > It is file and can be broken - so we have to check its consistency. There have to some evidency of dictionaries in cache - how will be identified a precompiled file? We cannot use a dictionary name, because it is a combination of dictionary and stop words. Have to have to thinking about filenames length here? Will be beter some generated name and a new system table? Regards Pavel Stehule > regards, tom lane > -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Pavel Stehule on 14 Jul 2010 09:12 Hello this patch is significantly reduced original patch. It doesn't propose a simple allocator - just eliminate a high memory usage for ispell dictionary. without this patch the ispell dictionary takes 55MB for tsearch2 context and 27MB in temp context. With this patch it takes "only" 25MB tsearch2 context and 19MB in temp context - its for Czech dictionary and UTF8 encoding. The patch is litlle bit ugly - it was reason, why I proposed a simple allocator, but it reduce a memory usage on 53% - the startup is better from 620 to 560 ms ~ 10% faster. little bit strange is repeated time - it goes down from 18ms to 5ms. Regards Pavel Stehule
From: Itagaki Takahiro on 20 Jul 2010 02:42 2010/7/14 Pavel Stehule <pavel.stehule(a)gmail.com>: > this patch is significantly reduced original patch. It doesn't propose > a simple allocator - just eliminate a high memory usage for ispell > dictionary. I don't think introducing new methods is a good idea. If you want a simple allocator, MemoryContextMethods layer seems better for me. The original purpose of the patch -- preloading dictionaries -- is also need to be redesigned with precompiler approach. I'll move the proposal to "Returned with Feedback" status. -- Itagaki Takahiro -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Pavel Stehule on 20 Jul 2010 07:39
Hello 2010/7/20 Itagaki Takahiro <itagaki.takahiro(a)gmail.com>: > 2010/7/14 Pavel Stehule <pavel.stehule(a)gmail.com>: >> this patch is significantly reduced original patch. It doesn't propose >> a simple allocator - just eliminate a high memory usage for ispell >> dictionary. > > I don't think introducing new methods is a good idea. If you want a > simple allocator, MemoryContextMethods layer seems better for me. > can you explain it, please? > The original purpose of the patch -- preloading dictionaries -- > is also need to be redesigned with precompiler approach. > I'll move the proposal to "Returned with Feedback" status. > ok. thank you very much Pavel Stehule > -- > Itagaki Takahiro > -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |