Prev: ssh_exchange_identification: Connection closed by remote host
Next: Migrating to ZFS
From: Guilbert STABILO on 22 Oct 2009 06:43
On Oct 22, 11:24 am, Chris Ridd <chrisr...(a)mac.com> wrote:
> On 2009-10-22 09:44:12 +0100, Guilbert STABILO
> <guilbert.stab...(a)yahoo.fr> said:
>
> > On Oct 22, 10:38 am, Ian Collins <ian-n...(a)hotmail.com> wrote:
>
> >> You could provide your own allocator which passed "normal" sized
> >> allocations to standard malloc and uses a mapped file for huge ones.
>
> > Yes but I have to intercept the malloc which occurs in the commercial
> > library I am linking with and I am not sure it is possible (I only
> > have the binary libraries).
> > My memory issue occurs in those libraries.
>
> Set LD_PRELOAD=<path to malloc library> in the environment before
> running your program. The umem_malloc man page describes a number of
> the available malloc libraries. I note there's a libmapmalloc.so.1...
> --
> Chris

I just tried doing this :

setenv LD_PRELOAD /usr/lib/libmtmalloc.so

.... but I now get the following error when I run my application or any
shell command (like ps) which uses malloc:

ld.so.1: ps: fatal: /usr/lib/libmapmalloc.so.1: wrong ELF class:
ELFCLASS32

So I suppose, I have to relink when possible with libmapmalloc.so.1
before being able to load it.
From: Guilbert STABILO on 22 Oct 2009 07:00
On Oct 22, 11:24 am, Chris Ridd <chrisr...(a)mac.com> wrote:

> Set LD_PRELOAD=<path to malloc library> in the environment before
> running your program. The umem_malloc man page describes a number of
> the available malloc libraries. I note there's a libmapmalloc.so.1...

Setting the LD_PRELOAD without relinking did not work but I finally
found the way of doing it without changing nor my code neither the
design (the malloc interface stays the same).

Here is my first compile/link line which produced the memory growing
problem:

$ g++ testmlk.cpp -o testmlk

And here is the result:

** Run 1 **

Init: 23101 testmlk 2192
Allocating 2000 MB: 23101 testmlk 1955320
Unallocating 2000 MB: 23101 testmlk 1955320

** Run 2 **

Init: 23113 testmlk 2192
Allocating 2000 MB: 23113 testmlk 1955320
Unallocating 2000 MB: 23113 testmlk 1955320

** Run 3 **

Init: 23125 testmlk 2192
Allocating 2000 MB: 23125 testmlk 1955320
Unallocating 2000 MB: 23125 testmlk 1955320

** Run 4 **

Init: 23137 testmlk 2192
Allocating 2000 MB: 23137 testmlk 1955320
Unallocating 2000 MB: 23137 testmlk 1955320

** Run 5 **

Init: 23149 testmlk 2192
Allocating 2000 MB: 23149 testmlk 1955320
Unallocating 2000 MB: 23149 testmlk 1955320

** Run 6 **

Init: 23161 testmlk 2192
Allocating 2000 MB: terminate called after throwing an instance of
'std::bad_alloc'
what(): St9bad_alloc


If I now link my code with the "mapmalloc" library:

$ g++ testmlk.cpp -o testmlk -L/usr/lib -lmapmalloc

It works exactly as expected (memory is given back to the system after
each deallocation) as shown below:

** Run 1 **

Init: 9904 testmlk 2196
Allocating 2000 MB: 9904 testmlk 1955324
Unallocating 2000 MB: 9904 testmlk 2196

** Run 2 **

Init: 9916 testmlk 2196
Allocating 2000 MB: 9916 testmlk 1955324
Unallocating 2000 MB: 9916 testmlk 2196

** Run 3 **

Init: 9932 testmlk 2196
Allocating 2000 MB: 9932 testmlk 1955324
Unallocating 2000 MB: 9932 testmlk 2196

** Run 4 **

Init: 9948 testmlk 2196
Allocating 2000 MB: 9948 testmlk 1955324
Unallocating 2000 MB: 9948 testmlk 2196

** Run 5 **

Init: 9964 testmlk 2196
Allocating 2000 MB: 9964 testmlk 1955324
Unallocating 2000 MB: 9964 testmlk 2196

** Run 6 **

Init: 9982 testmlk 2196
Allocating 2000 MB: 9982 testmlk 1955324
Unallocating 2000 MB: 9982 testmlk 2196

That's perfect !

From: Guilbert STABILO on 22 Oct 2009 07:02
On Oct 22, 12:43 pm, Guilbert STABILO <guilbert.stab...(a)yahoo.fr>
wrote:

> setenv LD_PRELOAD /usr/lib/libmtmalloc.so

I made a type mistake, I wanted to write:

setenv LD_PRELOAD /usr/lib/libmapmalloc.so.1


From: Casper H.S. Dik on 22 Oct 2009 08:22
Guilbert STABILO <guilbert.stabilo(a)yahoo.fr> writes:

>I just tried doing this :

>setenv LD_PRELOAD /usr/lib/libmtmalloc.so

Use:

setenv LD_PRELOAD libmtmalloc.so

or use:


setenv LD_PRELOAD_32 /usr/lib/libmtmalloc.so
setenv LD_PRELOAD_64 /usr/lib/64/libmtmalloc.so

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
From: Paul Floyd on 22 Oct 2009 15:09
On 21 Oct 2009 20:27:02 GMT, Guilbert STABILO <guilbert.stabilo(a)yahoo.fr> wrote:

[snip - memory problems]

>============================= testmlk.cpp ===========================
> #include <iostream>
> using namespace std;
> #include <unistd.h>
>
> void testmlk()
> {
> pid_t pid = getpid();
> char szPID[10];
> sprintf(szPID, "%u", pid);
> string sPsCmd = "ps -ef -opid -ofname -ovsz | grep " + string
> (szPID);
>
> char *pszTest = NULL;
> cerr << "\nInit: ";
> system(sPsCmd.data());
>
> cerr << "Allocating 2000 MB: ";
> pszTest = new char [2000000000];
> system(sPsCmd.data());
>
> cerr << "Unallocating 2000 MB: ";
>
> if(NULL != pszTest)
> {
> delete [] pszTest;
> pszTest = NULL;
> }
>
> else
> {
> cerr << "Allocation error !" << endl;
> }

This code is wrong. As you can see from your log, when new fails, a
std::bad_alloc exception is thrown. new does not return a 0 pointer.

You need to either use the nothrow version of new (and I wouldn't
recommend it), or catch the exception.

cerr << "Allocating 2000 MB: ";
try
{
pszTest = new char [2000000000];
}
catch (std::bad_alloc &e)
{
cerr << "Allocation error !" << endl;
// clean up and try to exit gracefully
}
// catch other exceptions if needed
system(sPsCmd.data());

cerr << "Unallocating 2000 MB: ";

delete [] pszTest;
pszTest = NULL;

>
> system(sPsCmd.data());
> }
>
> int main(int iArgc, char **ppArgv)
> {
> testmlk();
> cerr << endl;
> sleep(200);
> cerr << "Exiting ..." << endl;
> return(0);
> }
>
>============================= testmlk.cpp ===========================
>
>
>============================= testmlk.log ===========================
> ** Run 1 **
>
> Init: 23101 testmlk 2192
> Allocating 2000 MB: 23101 testmlk 1955320
> Unallocating 2000 MB: 23101 testmlk 1955320
>
> ** Run 2 **
>
> Init: 23113 testmlk 2192
> Allocating 2000 MB: 23113 testmlk 1955320
> Unallocating 2000 MB: 23113 testmlk 1955320
>
> ** Run 3 **
>
> Init: 23125 testmlk 2192
> Allocating 2000 MB: 23125 testmlk 1955320
> Unallocating 2000 MB: 23125 testmlk 1955320
>
> ** Run 4 **
>
> Init: 23137 testmlk 2192
> Allocating 2000 MB: 23137 testmlk 1955320
> Unallocating 2000 MB: 23137 testmlk 1955320
>
> ** Run 5 **
>
> Init: 23149 testmlk 2192
> Allocating 2000 MB: 23149 testmlk 1955320
> Unallocating 2000 MB: 23149 testmlk 1955320
>
> ** Run 6 **
>
> Init: 23161 testmlk 2192
> Allocating 2000 MB: terminate called after throwing an instance of
> 'std::bad_alloc'
> what(): St9bad_alloc
>============================= testmlk.log ===========================

On Solaris, you need as much VM as you allocate. So if you want to
allocate 20G, then you need 20G of VM. It looks like your machine has 4G
of RAM. That means that you're going to need at least 16G of swap.

Other OSes do this differently. On Windows, swap can grow dynamically
(with potentially poor performance while the swap is growing). On AIX
and Linux, the OS can overcommit. That is, new will lie to you that it
has allocated memory, whilst crossing fingers that you don't need all of
the memory. Memory doesn't really get allocated until it is accessed. If
you access more memory, then the jolly 'Out of Memory' killer will kill
an application with a signal 9. It'll try to guess which application is
guzzling memory, which might be yours, and it might be an innocent
bystander. In any case, the killed application gets no opportunity to
right matters. It is summarily killed.

As you see, no free lunches.
Solaris - not the fastest, but sure.
Windows - OK until you start growing swap.
Linux/AIX - fast, scarily unsafe.

A bientot
Paul
--
Paul Floyd http://paulf.free.fr
First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: ssh_exchange_identification: Connection closed by remote host
Next: Migrating to ZFS