Heavy malloc()/free() load and threads
in [Unix Programming]
|
From: Urs Thuermann on 11 Mar 2010 05:54 I have a typical producer/consumer problem (with varying-sized elements) which I currently have solved using a ring buffer in the classical way. Because of the varying size, no prior safe way to know the needed buffer size and to reduce copying I want to change that to linked list of items passed from producer to consumer. The producer would allocate memory for an item and append it to the list, the consumer would dequeue from the beginning of the list and free the memory for that item afterwards. The average rate will be roughly 40 to 50 allocations/deallocations in a strict LIFO order and there will be 30000 to 60000 items on the list. My questions are: 1. Will typical implementations of malloc()/free() in libc handle this load well? Or should I implement my own memory management? I currently use glibc on Debian but would also like to know about other libc implementations. BTW, the hardware is an Intel x86 CPU at about 2GHz. 2. Are malloc() and free() thread-safe (according to POSIX and/or in typical libc implementations) or do I have to use a mutex? urs
From: Casper H.S. Dik on 11 Mar 2010 06:47 Urs Thuermann <urs(a)isnogud.escape.de> writes: >The average rate will be roughly 40 to 50 allocations/deallocations in >a strict LIFO order and there will be 30000 to 60000 items on the list. Per what? >My questions are: >1. Will typical implementations of malloc()/free() in libc handle this > load well? Or should I implement my own memory management? I > currently use glibc on Debian but would also like to know about > other libc implementations. BTW, the hardware is an Intel x86 CPU > at about 2GHz. Never use your own memory management except in specific cases. Some OSes have multiple implementations of malloc(); some are specifically made to work better with threads. Solaris, e.g., has libumem (works like the Solaris kernel memory allocator), and libmtalloc. The default allocator not the most efficient when running with many threads but they all work. >2. Are malloc() and free() thread-safe (according to POSIX and/or in > typical libc implementations) or do I have to use a mutex? Yes, required by the standard. Casper -- Expressed in this posting are my opinions. They are in no way related to opinions held by my employer, Sun Microsystems. Statements on Sun products included here are not gospel and may be fiction rather than truth.
From: Nicolas George on 11 Mar 2010 09:01 Urs Thuermann wrote in message <ygfzl2fxh5k.fsf(a)janus.isnogud.escape.de>: > 1. Will typical implementations of malloc()/free() in libc handle this > load well? Or should I implement my own memory management? Code it, debug it, and test it. If you have performances problems, profile. If the profiling shows that memory allocation is the bottleneck, then you can think about changing it.
From: Rainer Weikusat on 11 Mar 2010 09:11 Urs Thuermann <urs(a)isnogud.escape.de> writes: > I have a typical producer/consumer problem (with varying-sized > elements) which I currently have solved using a ring buffer in the > classical way. Because of the varying size, no prior safe way to know > the needed buffer size and to reduce copying I want to change that to > linked list of items passed from producer to consumer. > > The producer would allocate memory for an item and append it to the > list, the consumer would dequeue from the beginning of the list and > free the memory for that item afterwards. > > The average rate will be roughly 40 to 50 allocations/deallocations in > a strict LIFO order and there will be 30000 to 60000 items on the list. > > My questions are: > > 1. Will typical implementations of malloc()/free() in libc handle this > load well? There is nobody on this world who can answer this question. You have to try and you will know if it didn't work after it failed.
From: Rainer Weikusat on 11 Mar 2010 09:45
Casper H.S. Dik <Casper.Dik(a)Sun.COM> writes: > Urs Thuermann <urs(a)isnogud.escape.de> writes: > >>The average rate will be roughly 40 to 50 allocations/deallocations in >>a strict LIFO order and there will be 30000 to 60000 items on the list. > > Per what? > >>My questions are: > >>1. Will typical implementations of malloc()/free() in libc handle this >> load well? Or should I implement my own memory management? I >> currently use glibc on Debian but would also like to know about >> other libc implementations. BTW, the hardware is an Intel x86 CPU >> at about 2GHz. > > Never use your own memory management except in specific cases. Never use malloc except in specific cases, namely, as a Sun document I read years ago aptly stated, "when product timeliness is more important than product quality". The only real-world benefit of 'malloc' is that it enables someone using it to avoid writing 50 - 300 LOC, based on the complexity of the allocation requirements. The last 'general purpose' slab-like allocator I implemented has 136 LOC, which makes it about 4% of total amount of code of the program using it. Memory allocation based solely on object sizes isn't a technical proposition, it is a political agenda, namely, an eternal research problem (to sink money into), at least until now. |