when does the system send SIGKLILL
in [Unix Programming]
|
Prev: Suggestions needed
Next: A tool that suggests optimized logic for a piece of code/module/function
From: cerr on 13 Jan 2010 13:32 On Jan 13, 10:28 am, cerr <ron.egg...(a)gmail.com> wrote: > On Jan 12, 5:18 pm, la...(a)ludens.elte.hu (Ersek, Laszlo) wrote: > > > > > > > In article <cb714bed-32c7-4ed6-a3c7-b6f8a02c9...(a)j24g2000yqa.googlegroups.com>, cerr <ron.egg...(a)gmail.com> writes: > > > > This GDB was configured as "i586-linux-uclibc". > > > [root(a)DEVNEMS logrecord]# ldd prs > > > libpthread.so.0 =3D> /lib/libpthread.so.0 (0xb7f58000) > > > libssl.so.0.9.7 =3D> /usr/lib/libssl.so.0.9.7 (0xb7f31000) > > > librt.so.0 =3D> /lib/librt.so.0 (0xb7f2f000) > > > libstdc++.so.6 =3D> /lib/libstdc++.so.6 (0xb7ebc000) > > > libm.so.0 =3D> /lib/libm.so.0 (0xb7eae000) > > > libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0xb7ea6000) > > > libc.so.0 =3D> /lib/libc.so.0 (0xb7e5a000) > > > libcrypto.so.0.9.7 =3D> /usr/lib/libcrypto.so.0.9.7 (0xb7d8b000) > > > libdl.so.0 =3D> /lib/libdl.so.0 (0xb7d88000) > > > ld-uClibc.so.0 =3D> /lib/ld-uClibc.so.0 (0xb7f6d000) > > > > I'm not quite certain what this would tell us tho... :( > > > At least it allows for some wild speculation :) > > aha, hehe :) > > > > > > > First, memwatch: > > >http://www.linkdata.se/memwatch > > > I'd risk after a very superficial look at memwatch that it does some > > nifty signals hacking. If your program is multi-threaded, that's not > > very easy. The memwatch USING itself says: > > > Is this stuff thread-safe? > > > I doubt it. As of version 2.66, there is rudimentary support > > for threads, if you happen to be using Win32 or if you have > > pthreads. Define WIN32 or MW_PTHREADS to signify this fact. > > > This will cause a global mutex to be created, and memwatch > > will lock it when accessing the global memory chain, but it's > > still far from certified threadsafe. > > I'm using multiple threads, yes and i am using pthread on Linux. So > the global mutexes would be able to lock-up these things but then > again, it certainly is suspecious that I added another thread with > lots of dynamic memory allocating when it starts receiving SIGKILLs.. > > > > > Second, you use uclibc. I have no idea whether memwatch was developed > > for / tested with uclibc. I'd say try building the app with memwatch > > disabled. I don't know if uclibc ships its own pthreads implementation, > > but if so, it may use some signals internally. (At least before NPTL, > > glibc used LinuxThreads which utilized some realtime (queued) signals. > > Or so I remember.) > > That would be the next step - to build it without memwatch, yes! As I > wrote these replys from the bottom up I would luke to see if my > breakpoints kick-in succesfully first - because if i receive SIGKILLs > without the breakpoints kicking-in, it comes from something else we > can say safely, right? Uh, and yes, I just did receive a SIGKILL without stopping at a breakpoint: Program terminated with signal SIGKILL, Killed. The program no longer exists. (gdb) So I guess it's time to omit memwatch.... [snip]
From: cerr on 13 Jan 2010 13:51 On Jan 13, 10:32 am, cerr <ron.egg...(a)gmail.com> wrote: > On Jan 13, 10:28 am, cerr <ron.egg...(a)gmail.com> wrote: > > > > > > > On Jan 12, 5:18 pm, la...(a)ludens.elte.hu (Ersek, Laszlo) wrote: > > > > In article <cb714bed-32c7-4ed6-a3c7-b6f8a02c9...(a)j24g2000yqa.googlegroups.com>, cerr <ron.egg...(a)gmail.com> writes: > > > > > This GDB was configured as "i586-linux-uclibc". > > > > [root(a)DEVNEMS logrecord]# ldd prs > > > > libpthread.so.0 =3D> /lib/libpthread.so.0 (0xb7f58000) > > > > libssl.so.0.9.7 =3D> /usr/lib/libssl.so.0.9.7 (0xb7f31000) > > > > librt.so.0 =3D> /lib/librt.so.0 (0xb7f2f000) > > > > libstdc++.so.6 =3D> /lib/libstdc++.so.6 (0xb7ebc000) > > > > libm.so.0 =3D> /lib/libm.so.0 (0xb7eae000) > > > > libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0xb7ea6000) > > > > libc.so.0 =3D> /lib/libc.so.0 (0xb7e5a000) > > > > libcrypto.so.0.9.7 =3D> /usr/lib/libcrypto.so.0.9..7 (0xb7d8b000) > > > > libdl.so.0 =3D> /lib/libdl.so.0 (0xb7d88000) > > > > ld-uClibc.so.0 =3D> /lib/ld-uClibc.so.0 (0xb7f6d000) > > > > > I'm not quite certain what this would tell us tho... :( > > > > At least it allows for some wild speculation :) > > > aha, hehe :) > > > > First, memwatch: > > > >http://www.linkdata.se/memwatch > > > > I'd risk after a very superficial look at memwatch that it does some > > > nifty signals hacking. If your program is multi-threaded, that's not > > > very easy. The memwatch USING itself says: > > > > Is this stuff thread-safe? > > > > I doubt it. As of version 2.66, there is rudimentary support > > > for threads, if you happen to be using Win32 or if you have > > > pthreads. Define WIN32 or MW_PTHREADS to signify this fact. > > > > This will cause a global mutex to be created, and memwatch > > > will lock it when accessing the global memory chain, but it's > > > still far from certified threadsafe. > > > I'm using multiple threads, yes and i am using pthread on Linux. So > > the global mutexes would be able to lock-up these things but then > > again, it certainly is suspecious that I added another thread with > > lots of dynamic memory allocating when it starts receiving SIGKILLs.. > > > > Second, you use uclibc. I have no idea whether memwatch was developed > > > for / tested with uclibc. I'd say try building the app with memwatch > > > disabled. I don't know if uclibc ships its own pthreads implementation, > > > but if so, it may use some signals internally. (At least before NPTL, > > > glibc used LinuxThreads which utilized some realtime (queued) signals.. > > > Or so I remember.) > > > That would be the next step - to build it without memwatch, yes! As I > > wrote these replys from the bottom up I would luke to see if my > > breakpoints kick-in succesfully first - because if i receive SIGKILLs > > without the breakpoints kicking-in, it comes from something else we > > can say safely, right? > > Uh, and yes, I just did receive a SIGKILL without stopping at a > breakpoint: > > Program terminated with signal SIGKILL, Killed. > The program no longer exists. > (gdb) > > So I guess it's time to omit memwatch.... > [snip] Uh, I just realize memwatch isn't even being used in my application... :o ... it's just there cause someone before me apparently meant to use it but it's not used at all i just commented out the include and it compiles flawlessly..... hence SIGKILL is coming from somewhere else for sure! We're back in the speculating game... :o
From: cerr on 13 Jan 2010 16:43 On Jan 11, 4:00 pm, la...(a)ludens.elte.hu (Ersek, Laszlo) wrote: > In article <b9bed0c3-6994-44d9-aef1-029488ad8...(a)q4g2000yqm.googlegroups.com>, cerr <ron.egg...(a)gmail.com> writes: > > > I'm running Linux and i've been looking at my syslog file (all > > messages going into /var/log/messages) and i dont' see any messages > > that aren't coming from my own application....so a memory leak would > > be a reason you're saying, huh? > > Hmm....I can't see this being a problem, i think i free() everything > > that's been allocated with malloc/realloc.... > > Your process could still get killed by the OOM killer before you reach > free(). Try to turn off overcommit (and then reboot, perhaps): After a reboot, /proc/sys/vm/overcommit_memory shows 0 again... :( So that doesn't work with a reboot... I hope the kernel checking the content of this file "on the fly" > http://groups.google.com/group/comp.lang.c/msg/b6b1aff01a34738d?dmode... > > If you start seeing SIGSEGVs instead of SIGKILLs, or the application > starts complaining about malloc() returning a null pointer, then the > problem is memory exhaustion. > > Since you may be editing /etc/sysctl.conf anyway, add this too: > > vm.mmap_min_addr = 4096 > > http://wiki.debian.org/mmap_min_addr > > Cheers, > lacos
From: Ersek, Laszlo on 13 Jan 2010 17:07 In article <f9fb5c71-269c-4470-b597-51ca33b16e49(a)p8g2000yqb.googlegroups.com>, cerr <ron.eggler(a)gmail.com> writes: > On Jan 12, 5:18=A0pm, la...(a)ludens.elte.hu (Ersek, Laszlo) wrote: >> Third, before you start the application in gdb, set breakpoints at >> pthread_kill(), kill(), and raise(). (You may need system library debug >> symbols for this.) Whenever you stop in one of them, get a backtrace. >> Some system library (eg. the pthreads implementation) might detect such >> a mess that it has no choice but to kill the process. > > Well, i connected to the remote target, hit continue and ctrl-c-ed > back to the (gdb) prpmt in order to add breakpoints like: > (gdb) break pthread_kill() > Function "pthread_kill()" not defined. > Make breakpoint pending on future shared library load? (y or [n]) > (gdb) break kill() > Function "kill()" not defined. > Make breakpoint pending on future shared library load? (y or [n]) > (gdb) break raise() > Function "raise()" not defined. > Make breakpoint pending on future shared library load? (y or [n]) > (gdb) continue > Continuing. > > I have very little experience with gdb and hope i did this > correctly...? I doubt it. You should omit the () after the function names: (gdb) break pthread_kill Then you should answer y when asked, so that gdb prints Breakpoint 1 (pthread_kill) pending. You may want to install (and use) system libraries with debugging symbols as well. For example, on debian: $ apt-cache show libc6-dbg Package: libc6-dbg [...] Description: GNU C Library: Libraries with debugging symbols Contains unstripped shared libraries. This package is provided primarily to provide a backtrace with names in a debugger, which makes it somewhat easier to interpret core dumps. The libraries are installed in /usr/lib/debug and can be used by placing that directory in LD_LIBRARY_PATH. [...] Cheers, lacos
From: Ersek, Laszlo on 13 Jan 2010 17:12
In article <73b9a15c-9734-4b27-92e0-1ff4d3626695(a)f5g2000yqh.googlegroups.com>, cerr <ron.eggler(a)gmail.com> writes: > After a reboot, /proc/sys/vm/overcommit_memory shows 0 again... :( Of course, that's the default. You should modify one of your init scripts to set up this runtime kernel parameter before starting any long-lived processes. On Debian, this is done by modifying /etc/sysctl.conf. Cheers, lacos |