Heavy malloc()/free() load and threads
in [Unix Programming]
|
From: Ian Collins on 20 Mar 2010 04:27 On 03/20/10 08:59 PM, Jonathan de Boyne Pollard wrote: Please stop cross-posting to so many groups and worse still, snipping attributions. -- Ian Collins
From: Richard Bos on 20 Mar 2010 15:05 Ian Collins <ian-news(a)hotmail.com> wrote: > On 03/20/10 08:59 PM, Jonathan de Boyne Pollard wrote: > > Please stop cross-posting to so many groups and worse still, snipping > attributions. Not to mention posting bloody HTML, in the raw. If you _must_ post HTML, use the proper headers so that my server can dump it in the bit-bucket in non-binary groups, as it should. Richard
From: Rainer Weikusat on 21 Mar 2010 15:10 Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups(a)NTLWorld.COM> writes: > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> In addition to the useless crosspost, the HTML is also unkind. I was actually seriously trying to edit this into a readable text in order to address your somewhat 'weird' statements about malloc implementations not written by yourself but quickly grew tired of the effort. Feel free to repost this in human-readable form and with a selection of 'newsgroups' that lets you look less like a troll.
From: Scott Lurndal on 21 Mar 2010 21:33 Ian Collins <ian-news(a)hotmail.com> writes: >On 03/20/10 08:59 PM, Jonathan de Boyne Pollard wrote: > >Please stop cross-posting to so many groups and worse still, snipping >attributions. > And even worse still, posting in HTML.
From: balson on 23 Mar 2010 10:05
sfuerst wrote: > On Mar 11, 3:54 am, Urs Thuermann<u...(a)isnogud.escape.de> wrote: >> I have a typical producer/consumer problem (with varying-sized >> elements) which I currently have solved using a ring buffer in the >> classical way. Because of the varying size, no prior safe way to know >> the needed buffer size and to reduce copying I want to change that to >> linked list of items passed from producer to consumer. >> >> The producer would allocate memory for an item and append it to the >> list, the consumer would dequeue from the beginning of the list and >> free the memory for that item afterwards. >> >> The average rate will be roughly 40 to 50 allocations/deallocations in >> a strict LIFO order and there will be 30000 to 60000 items on the list. >> >> My questions are: >> >> 1. Will typical implementations of malloc()/free() in libc handle this >> load well? Or should I implement my own memory management? I >> currently use glibc on Debian but would also like to know about >> other libc implementations. BTW, the hardware is an Intel x86 CPU >> at about 2GHz. > > The easiest response is to say "Benchmark it and see". However, I > think I can guess what you will find. > > The glibc allocator is very fast. However, it has a problem with the > producer-consumer pattern. It uses "arenas" with separate locking. > When allocating, it tries to lock the last used arena, and if that > fails, uses a different one. When freeing, it needs to lock the arena > of that chunk of memory. The net result is that threads can end up > owning arenas with vastly different sizes, which can look like a > memory leak. This behaviour is described at > http://google-perftools.googlecode.com/svn/trunk/doc/tcmalloc.html > > So google wrote their own allocator (tcmalloc) to fix this problem. > You may want to give it a try as it performs fairly well. The only > problem with it is that it doesn't return memory to the operating > system. This may pose a problem for long-running server tasks. > > Another allocator you might want to try is Hoard. That one is fairly > old, and faster than tcmalloc for small allocation sizes. For large > sizes though, it is catastrophically slower, so you may want to > benchmark it to see if it helps you or not. > > Then there is the allocator I wrote... which is faster than all three > of the above. It unfortunately, isn't open source. I have a nice > page showing the results of googles "t-test1" memory allocator > benchmark for various numbers of threads and average allocation sizes > for the above allocators. It should help you decide which one may be > the best for you: http://locklessinc.com/benchmarks.shtml > > A final option is to write your own allocator. This is probably only > worth doing if the size-range of things on your queue is small. Then > you may use a simple linked list of free nodes, which should be fast > enough for you. If you have a wide size-range, then you should know > that writing a general purpose allocator isn't a 200 line job. To get > something better than the above allocators will probably require a few > thousand lines or more. > >> 2. Are malloc() and free() thread-safe (according to POSIX and/or in >> typical libc implementations) or do I have to use a mutex? >> >> urs > > They are in glibc. In older libcs, not necessarily. I'd also be > careful of some of the lesser-known allocators. Most of the ones I've > tested seemed to be fairly buggy if you have multiple threads > executing allocations and frees simultaneously. i.e. if something > crashes with the t-test1 benchmark, then it definitely isn't worth > using. This is the main reason that I don't have more allocators on > the benchmark page. Most simply didn't work properly when tested > hard. > > The only other one of mention is jemalloc, which is the allocator that > firefox uses. That allocator is designed for minimal memory usage > rather than allocation speed. The result is that it is quite slow. I > thought that it would be unfair to compare it to the speed-demon > allocators which didn't care quite so much for memory usage. > > Steven All of these allocators are probably good for experimenting and lab use, etc. When it come time for getting serious about speed and scalability, this is the link you'll need: http://www.cherrystonesoftware.com Jim |