Batch malloc
in [General Programming]
|
Prev: hacking
Next: How get percentage of use of cpu
From: Jon Harrop on 17 Jan 2010 11:33 Anyone know of a memory allocator that allows blocks to be allocated in batches? -- Dr Jon D Harrop, Flying Frog Consultancy Ltd. http://www.ffconsultancy.com/?u
From: Richard Harter on 17 Jan 2010 10:59 On Sun, 17 Jan 2010 16:33:22 +0000, Jon Harrop <jon(a)ffconsultancy.com> wrote: > >Anyone know of a memory allocator that allows blocks to be allocated in >batches? Could you be more specific? Allocating blocks in batches could mean any of several different things. What is it that you want to do that can't be done with a simple wrapper? Richard Harter, cri(a)tiac.net http://home.tiac.net/~cri, http://www.varinoma.com Infinity is one of those things that keep philosophers busy when they could be more profitably spending their time weeding their garden.
From: BGB / cr88192 on 17 Jan 2010 17:33 "Richard Harter" <cri(a)tiac.net> wrote in message news:4b533367.369413093(a)text.giganews.com... > On Sun, 17 Jan 2010 16:33:22 +0000, Jon Harrop > <jon(a)ffconsultancy.com> wrote: > >> >>Anyone know of a memory allocator that allows blocks to be allocated in >>batches? > > Could you be more specific? Allocating blocks in batches could > mean any of several different things. What is it that you want > to do that can't be done with a simple wrapper? > agreed, this is vague... however, to infer a partial answer: one can allocated a chunk of memory (say via malloc, mmap, or VirtualAlloc), and divide it up into smaller objects (in any number of possible ways). one example is a "slab", where the memory is treated as an array of fixed-size objects which are often linked into linked lists (a used list or a free list, ...). another is to have a rover, and simply slide along the chunk carving out blocks for objects (works very well for certain memory usage patterns, but is not well suitible for "general purpose" use). another option is to use bitmaps, and allocate objects in terms of groups of adjacent fixed-size "cells" (this is one of my major allocator strategies, usually used for smaller objects). (this is because a lot of my code tends towards smaller objects made of many disjoint pieces). (often I speed up searches for free space by using a rover, since free space is far more likely in regions which have not been touched recently than in places which have just recently been searched). ....
From: Richard Harter on 17 Jan 2010 23:08 On Sun, 17 Jan 2010 15:33:03 -0700, "BGB / cr88192" <cr88192(a)hotmail.com> wrote: > >"Richard Harter" <cri(a)tiac.net> wrote in message >news:4b533367.369413093(a)text.giganews.com... >> On Sun, 17 Jan 2010 16:33:22 +0000, Jon Harrop >> <jon(a)ffconsultancy.com> wrote: >> >>> >>>Anyone know of a memory allocator that allows blocks to be allocated in >>>batches? >> >> Could you be more specific? Allocating blocks in batches could >> mean any of several different things. What is it that you want >> to do that can't be done with a simple wrapper? >> > >agreed, this is vague... > > >however, to infer a partial answer: >one can allocated a chunk of memory (say via malloc, mmap, or VirtualAlloc), >and divide it up into smaller objects (in any number of possible ways). > >one example is a "slab", where the memory is treated as an array of >fixed-size objects which are often linked into linked lists (a used list or >a free list, ...). > >another is to have a rover, and simply slide along the chunk carving out >blocks for objects (works very well for certain memory usage patterns, but >is not well suitible for "general purpose" use). > >another option is to use bitmaps, and allocate objects in terms of groups of >adjacent fixed-size "cells" (this is one of my major allocator strategies, >usually used for smaller objects). (this is because a lot of my code tends >towards smaller objects made of many disjoint pieces). (often I speed up >searches for free space by using a rover, since free space is far more >likely in regions which have not been touched recently than in places which >have just recently been searched). The puzzling thing is that Jon is almost certainly familiar with all of these techiques. That implies that he has something else in mind. Richard Harter, cri(a)tiac.net http://home.tiac.net/~cri, http://www.varinoma.com Infinity is one of those things that keep philosophers busy when they could be more profitably spending their time weeding their garden.
From: BGB / cr88192 on 18 Jan 2010 01:19
"Richard Harter" <cri(a)tiac.net> wrote in message news:4b53de37.413140546(a)text.giganews.com... > On Sun, 17 Jan 2010 15:33:03 -0700, "BGB / cr88192" > <cr88192(a)hotmail.com> wrote: > >> >>"Richard Harter" <cri(a)tiac.net> wrote in message >>news:4b533367.369413093(a)text.giganews.com... >>> On Sun, 17 Jan 2010 16:33:22 +0000, Jon Harrop >>> <jon(a)ffconsultancy.com> wrote: >>> >>>> >>>>Anyone know of a memory allocator that allows blocks to be allocated in >>>>batches? >>> >>> Could you be more specific? Allocating blocks in batches could >>> mean any of several different things. What is it that you want >>> to do that can't be done with a simple wrapper? >>> >> >>agreed, this is vague... >> >> >>however, to infer a partial answer: >>one can allocated a chunk of memory (say via malloc, mmap, or >>VirtualAlloc), >>and divide it up into smaller objects (in any number of possible ways). >> >>one example is a "slab", where the memory is treated as an array of >>fixed-size objects which are often linked into linked lists (a used list >>or >>a free list, ...). >> >>another is to have a rover, and simply slide along the chunk carving out >>blocks for objects (works very well for certain memory usage patterns, but >>is not well suitible for "general purpose" use). >> >>another option is to use bitmaps, and allocate objects in terms of groups >>of >>adjacent fixed-size "cells" (this is one of my major allocator strategies, >>usually used for smaller objects). (this is because a lot of my code tends >>towards smaller objects made of many disjoint pieces). (often I speed up >>searches for free space by using a rover, since free space is far more >>likely in regions which have not been touched recently than in places >>which >>have just recently been searched). > > The puzzling thing is that Jon is almost certainly familiar with > all of these techiques. That implies that he has something else > in mind. > > yeah, dunno... then again, I guess there are lots of developers who have been doing programming maybe for many years without much grasp of what is actually going on in memory ("well I use new and an object appears, and I use delete and it goes away"...). although, I guess it would be a bit strange if one went into VM implementation without at some point figuring out how memory managers work... so, I don't know... meanwhile, I am off trying to find a way to get my C compile times maybe 10 or 100x faster by making headers optionally optional... (IOW: optionally switching to a very different scoping model...). how to address this in the preprocessor (without making a horrid mess) remains a bit uncertain though... (maybe either a pragma or a "magic" define to enable macro-importing...). |