dynamically allocating chunks from shared memory
in [PgSql]
|
Prev: managing git disk space usage
Next: [HACKERS] antisocial things you can do in git (but not CVS)
From: Alvaro Herrera on 26 Jul 2010 10:31 Excerpts from Robert Haas's message of lun jul 26 08:52:46 -0400 2010: > Here's another idea. Instead of making imessages use an SLRU, how > about having it steal pages from shared_buffers? This would require > segmenting messages into small enough chunks that they'd fit, but the > nice part is that it would avoid the need to have a completely > separate shared memory arena. Ideally, we'd make the infrastructure > general enough that things like SLRU could use it also; and get rid of > or reduce in size some of the special-purpose chunks we're now > allocating. What's the problem you see with "another shared memory arena"? Right now we allocate a single large arena, and the lot of shared_buffers, SLRU pools, locking objects, etc are all allocated from there. If we want another 2 MB for "dynamic shmem", we'd just allocate 2 MB more in that large arena and give those to this new code. -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Robert Haas on 26 Jul 2010 13:16 On Mon, Jul 26, 2010 at 12:51 PM, Markus Wanner <markus(a)bluegap.ch> wrote: >> Dynamically allocating out of a 2MB >> segment gives up most of that flexibility. > > Absolutely, that's why I'd like to see other modules that use the dynamic > allocator. The more the better. Right, I agree. The problem is that I don't think they can. The elephant in the room is shared_buffers, which I believe to be typically BY FAR the largest consumer of shared memory. It would be absolutely fantastic if we had a shared_buffers implementation that could free up unused buffers when they're not needed, or add more when required. But there are several reasons why I don't believe that will ever happen. One, much of the code that uses shared_buffers relies on shared_buffers being located at a fixed memory address on a contiguous chunk, and it's hard to see how we could change that assumption without sacrificing performance. Two, the overall size of the shared memory arena is largely dependent on the size of shared_buffers, so unless you also have the ability to resize the arena on the fly (which is well-nigh to impossible with our current architecture, and maybe with any architecture), resizing shared_buffers doesn't actually add that much flexibility. Three, the need for shared buffers is elastic rather than absolute: stealing a few shared buffers for a defined purpose (like sending imessages) is perfectly reasonable, but it's rarely going to be a good idea for the buffer manager to proactively free up memory just in case some other part of the system might need some. If you have a system that normally has 4GB of shared buffers and some other module borrows 100MB and then returns it, the system will just cache less data while that memory is in use and then start right back up caching more again once it's returned. That's very nice, and it's hard to see how else to achieve that result. Of course, there are other parts of the system (a whole bunch of them) that used shared memory also, and perhaps some of those could be modified to use the dynamic allocator as well. But they're getting by without it now, so maybe they don't really need it. The SLRU stuff, I think, works more or less like shared buffers (so you have the same set of issues) and I think most of the other users are allocating small, fixed-size chunks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Markus Wanner on 26 Jul 2010 13:50 Hi, On 07/26/2010 07:16 PM, Robert Haas wrote: > Of course, there are other parts of the system (a whole bunch of them) > that used shared memory also, and perhaps some of those could be > modified to use the dynamic allocator as well. But they're getting by > without it now, so maybe they don't really need it. The SLRU stuff, I > think, works more or less like shared buffers (so you have the same > set of issues) and I think most of the other users are allocating > small, fixed-size chunks. Yeah, I see your point(s). Note however, that a thread based design doesn't have this problem *at all*. Memory generally is shared (between threads) and you can dynamically allocate more or less (until Linux' OOM killer hits you.. yet another story). The OS reuses memory you don't currently need even for other applications. Users as well as developers know the threaded model (arguably, much better than the process based one). So that's what we get compared to. And what developers (including me) are used to. I think we are getting by with fixed allocations at the moment, because we did a lot to get by with it. By working around these limitations. However, that's just my thinking. Thank you for your inputs. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Robert Haas on 26 Jul 2010 14:56 On Mon, Jul 26, 2010 at 1:50 PM, Markus Wanner <markus(a)bluegap.ch> wrote: > Note however, that a thread based design doesn't have this problem *at all*. > Memory generally is shared (between threads) and you can dynamically > allocate more or less (until Linux' OOM killer hits you.. yet another > story). The OS reuses memory you don't currently need even for other > applications. > > Users as well as developers know the threaded model (arguably, much better > than the process based one). So that's what we get compared to. And what > developers (including me) are used to. I'm sort of used to the process model, myself, but I may be in the minority. > I think we are getting by with fixed allocations at the moment, because we > did a lot to get by with it. By working around these limitations. > > However, that's just my thinking. Thank you for your inputs. I completely agree with you that fixed allocations suck. We're just disagreeing (hopefully, in a friendly and collegial fashion) about what to do about it. I actually think that memory management is one of the weakest elements of our current architecture, though I think for somewhat different reasons than what you're thinking about. Besides the fact that we have various smaller pools of dynamically shared memory (e.g. a separate ring of buffers for each SLRU), I'm also unhappy about some of the things we do with backend-private memory, work_mem being the biggest culprit by far, because it's very difficult for the DBA to set the knobs in a way that uses all of the memory he wants to allocate to the database efficiently no overruns and none left over. The case where you can count on the database and all of your temporary files, etc. to fit in RAM is really an exceptional case: in general, you need to assume that there will be more demand for memory than there will be memory available, and as much as possible you want the system (rather than the user) to decide how it should optimally be allocated. The query planner and executor actually do have most of what is needed to execute queries using more or less memory, but they lack the global intelligence needed for intelligent decision-making. Letting the OS buffer cache rather than the PG buffer cache handle most of the system's memory helps, but it's not a complete solution. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Markus Wanner on 26 Jul 2010 12:34
Hi, On 07/26/2010 04:31 PM, Alvaro Herrera wrote: > Excerpts from Robert Haas's message of lun jul 26 08:52:46 -0400 2010: >> Here's another idea. Instead of making imessages use an SLRU, how >> about having it steal pages from shared_buffers? This would require >> segmenting messages into small enough chunks that they'd fit, but the >> nice part is that it would avoid the need to have a completely >> separate shared memory arena. Ideally, we'd make the infrastructure >> general enough that things like SLRU could use it also; and get rid of >> or reduce in size some of the special-purpose chunks we're now >> allocating. To me that sounds like solving the same kind of problem for every module separately and somewhat differently. I tend to like general solutions (often too much, but that's another story), and to me it still seems a completely dynamic memory allocator solves that generically (and way more elegant than 'stealing pages' sounds). > Right > now we allocate a single large arena, and the lot of shared_buffers, > SLRU pools, locking objects, etc are all allocated from there. Uh.. they all allocate from different, statically sized pool, don't they? > If we > want another 2 MB for "dynamic shmem", we'd just allocate 2 MB more in > that large arena and give those to this new code. That's how it could work if we used a dynamic allocator. But currently, if I understand correctly, once the shared_buffers pool is full, it cannot steal memory from the SLRU pools. Or am I mistaken? Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |